Stream Governance を使用してデータメッシュを構築する方法 | ウェビナーに参加

Kafka Summit Europe 2021

View sessions and slides from Kafka Summit Europe 2021

Keynotes

Making Kafka Cloud Native

  • Jay Kreps, Confluent

A talk discussing the rise of Apache Kafka and data in motion plus the impact of cloud native data systems. This talk will cover how Kafka needs to evolve to keep up with the future of cloud, what this means for distributed systems engineers, and what work is being done to truly make Kafka Cloud...

BMW Group and Confluent - A Fireside Chat

  • Jay Kreps, Confluent
  • Felix Böhm, BMW Group

Today’s data is in motion. Enterprises need to see their business and their data in near real time; they need to respond in milliseconds not hours; and they need to integrate, aggregate, curate, and disseminate data within and across production environments.

How to Build the Data Mesh Foundation: A Principled Approach

  • Zhamak Dehghani, ThoughtWorks

Organizations have been chasing the dream of data democratization, unlocking and accessing data at scale to serve their customers and business, for over a half a century from early days of data warehousing. They have been trying to reach this dream through multiple generations of architectures...

Wrap-Up Keynote

  • Tim Berglund, StarTree

Breakouts

On Track with Apache Kafka: Building a Streaming ETL solution with Rail Data

  • Robin Moffatt, Confluent

Do you want to know what streaming ETL actually looks like in practice? Or what you can REALLY do with Apache Kafka once you get going—using config & SQL alone? This project integrates live data from the UK rail network via ActiveMQ and data from other sources to build a fully-functioning platform.

Apache Kafka and the Data Mesh

  • Ben Stopford, Confluent
  • Michael Noll, Confluent

Data mesh is a relatively recent term that describes a set of principles that good modern data systems uphold. A kind of “microservices” for the data-centric world. While the data mesh is not technology-specific as a pattern, the building of systems that adopt and implement data mesh principles...

A Kafkaesque Raft Protocol

  • Jason Gustafson, Confluent

KIP-500 set the vision for Zookeeper-free Kafka. However, even without Zookeeper, the need for consensus never went away. In this talk, we will discuss one of the core community’s initiatives, a native Raft-like protocol used to ensure different brokers can agree on critical pieces of metadata...

Advanced Change Data Streaming Patterns in Distributed Systems

  • Gunnar Morling, Decodable
  • Hans-Peter Grahsl, Red Hat

Microservices are one of the big trends in software engineering of the last few years. In this session we'll discuss and showcase how open-source change data capture (CDC) with Debezium can help developers with typical challenges they often face when working on microservices.

Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka

  • Xiaoman Dong, Stripe
  • Joey Pereira, Stripe

At Stripe, we operate a general ledger modeled as double-entry bookkeeping for all financial transactions. Warehousing such data is challenging due to its high volume and high cardinality of unique accounts. Apache Pinot works well in synergy with Kafka to provide an excellent solution.

Beyond the Brokers

  • Emma Humber, Confluent
  • Andrew Borley, IBM

While Kafka has guarantees around the number of server failures a cluster can tolerate, to avoid service interruptions, or even data loss, it is prudent to have infrastructure in place for when an environment becomes unavailable during a planned or unplanned outage.

Bayer Document Stream Processing

  • Astrid Rheinländer, Computational Scientist
  • Christoph Böhm, Data Engineer

Bayer selected Apache Kafka as the primary layer for a variety of document streams flowing through several text processing and enrichment steps. Every day, Bayer analyzes numerous documents including clinical trials, patents, reports, news, literature, etc.

Building a fully Kafka-based product as a Data Scientist

  • Patrick Neff, BAADER

A data analytics project for a food processing factory revealed that business problems could be solved and processes improved by implementing streaming applications.

Building a Modern, Scalable Cyber Intelligence Platform with Apache Kafka

  • Jac Noel, Intel

As cyber threats continuously grow in sophistication and frequency, companies need to quickly acclimate to effectively detect, respond, and protect their environments. We’ll discuss the details described in the IT@Intel white paper that was published in Nov 2020 with same title.

Building Event Streaming Microservices with Spring Boot and Apache Kafka

  • Jan Svoboda, Confluent

Developing cloud native microservices introduced us to many new challenges. One of the most difficult is to build reliable microservices integrations and their data exchange patterns. In this session I will share my 10 years of experience with building microservices...

Can Apache Kafka Replace a Database? – The 2021 Update

  • Kai Waehner, Confluent

Can and should Apache Kafka replace a database? How long can and should I store data in Kafka? How can I query and process data in Kafka? This session explains the idea behind databases and different features like storage, queries, transactions, and processing to evaluate when Kafka is a good ...

Capacity Planning Your Kafka Cluster

  • Jason Bell, Digitalis

There's little talk about capacity planning Kafka clusters, it's very much learn as you go, every cluster is different. In this talk Kafka DevOps Engineer Jason Bell takes you through the things that will help you, from broker capacity, thinking about topics and how the other Confluent components...

Cloud-Based Event Stream Processing Architectures and Patterns with Apache Kafka

  • Israel Ekpo, Microsoft

The Apache Kafka ecosystem is very rich with components and pieces that make for designing and implementing secure, efficient, fault-tolerant and scalable event stream processing (ESP) systems.

Cost-effective GraphQL Queries against Kafka Topics at scale

  • Torben Meyer, bakdata GmbH

In our projects, we often have to query the content of Kafka topics. To that end, we expose REST-APIs based on Kafka Streams’ interactive queries. However, this approach has some shortcomings.

Data governance and discoverability at AO.com

  • Jon Vines, Confluent
  • Christoph Schubert, Confluent

One challenge of widespread adoption of any technology within an organization is balancing organic growth and maintaining standards and best practice. At AO.com - one of the UK's largest online electrical retailers...

Delivering Cloud-Native Data Pipelines with Kafka Connect on Kubernetes

  • Viktor Gamov, Confluent

Getting data between systems, particularly at scale, is a common challenge faced by data engineers. We will give a short intro to Kafka Connect and container technologies before proceeding to a deep dive into practical applications.

Delivering: from Kafka to WebSockets

  • Adam Warski, SoftwareMill

Here's the challenge: we've got a Kafka topic, where services publish messages to be delivered to browser-based clients through web sockets. Sounds simple? It might, but we're faced with an increasing number of messages, as well as a growing count of web socket clients.

Demystifying Event-Driven Architectures with Apache Kafka

  • Bogdan Sucaciu, Deloitte Digital

Event-Driven Architectures (EDA ) are perceived as mythical objects that instantly transform your systems into ""real-time"" ones! BUT, come to think of it, aren't they already ""real-time""? I mean, adding an item to the cart is pretty much instant in ( most ) webshops.

Designing a Service Mesh with Kafka and Sagas

  • David Navalho , Marionete
  • Ricardo Miranda, Closer.pt

Kafka has become more than a simple message bus: with a full stack of tooling and new concepts, it’s easy to start deploying complex service meshes, communicating through Kafka, enabling decoupled microservices, stable performance, high scalability and reusability.

Designing Payloads for Event-Driven Systems

  • Lorna Mitchell, Aiven

Event-driven systems come in different shapes and sizes, and the rules for payload construction are: there are no rules (but there are guidelines). Flexible payloads are both the best and worst thing about event streaming - you never quite know what to expect from each system's payloads.

Development of dynamic pricing for tours using real-time data feeds

  • Mourad Benabdelkerim, FREE NOW

FREE NOW business is growing rapidly as a ride-hailing industry in general which creates a fair amount of technical challenges related to real-time data aggregation and processing.

Enabling Data Scientists to easily create and own Kafka Consumers

  • Stefan Krawczyk, Stitch Fix

In this talk I’ll go over how we built an opinionated Kafka client to easily enable Data Scientists to deploy and own production Kafka consumers, by focusing on writing python functions rather than fighting pitfalls with Kafka.

Encrypting Kafka messages at rest to secure applications

  • Robert Barnes, HashiCorp

Whilst Kafka has the ability to encrypt data in transit, it does not have the functionality out of the box to encrypt data at rest. This places the responsibility of encryption of data placed on message queues on developers. Implementing cryptography correctly in our applications is challenging ...

Event-driven Applications with Kafka, Micronaut, and AWS Lambda

  • Dave Klein, Tabular

Micronaut is an application framework that provides dependency injection, developer productivity features, and excellent support for Apache Kafka. In this session, we'll explore the ways that Apache Kafka and Micronaut work together to enable us to build fast, efficient, event-driven applications.

Everything you ever needed to know about Kafka on Kubernetes but were afraid to ask

  • Jakub Scholz, Red Hat

Kubernetes became the de-facto standard for running cloud-native applications. And many users turn to it also to run stateful applications such as Apache Kafka. You can use different tools to deploy Kafka on Kubernetes - write your own YAML files, use Helm Charts, or ...

Failing to Cross the Streams – Lessons Learned the Hard Way

  • Philip Schmitt, TNG Technology Consulting

This is a talk about debugging Stream–Table joins – based on my first-hand experience of stumbling into various pitfalls.

Getting started with AsyncAPI: how to describe your Kafka cluster

  • Salma Saeed, IBM
  • Dale Lane, IBM

AsyncAPI is an open source initiative that makes working with event-driven architectures as easy as working with REST APIs.

Getting up to speed with MirrorMaker 2 - Kafka Summit Europe 2021

  • Mickael Maison, Red Hat
  • Ryanne Dolan , LinkedIn

MirrorMaker 2, released recently as part of Kafka 2.4.0, allows you to mirror multiple clusters and create many replication topologies. Learn all about this awesome new tool and how to reliably and easily mirror clusters.

How ksqlDB works

  • Michael Drogalis, Confluent

ksqlDB, the event streaming database, is becoming one of the most popular ways to work with Apache Kafka®. Every day, there are many questions about the project, but here’s a question with an answer that we are always trying to improve: How does ksqlDB work?

How to mutate your immutable log

  • Andrey Falko , Salesforce

Is your data pipeline under development and you simply want to iterate quickly? Immutability is one of the key and desirable features of Kafka. However, when mistakes happen and you are paged at night you sometimes wish there was an “easy button” to change the log.

How to over-engineer things and have fun?

  • Oto Brglez, OPALAB

A journey into the building of a modern distributed real-time Wi-Fi spying system with the sole intention to have fun and play around.

How we eased out security journey with OAuth (Goodbye Kerberos!)

  • Paul Makkar , Saxo Bank
  • Rahul Gulati, Saxo Bank

As we modernize and scale, the demands of hybrid cloud, multiple domains, polyglot computing and Data Mesh require us to also modernize our approach to security.

How Zillow Unlocked Kafka to 50 Teams in 8 months

  • Shahar Cizer Kobrinsky, Zillow

As an AWS shop, Zillow engineering teams have been using various messaging and streaming services for years. As Zillow 2.0 piled through, new requirements and pain points made us rethink our streaming stack.

Introducing Confluent labs Parallel Consumer client

  • Anthony Stubbes, Confluent

Consuming messages in parallel is what Apache Kafka® is all about, so you may well wonder, why would we want anything else? It turns out that, in practice, there are a number of situations where Kafka’s partition-level parallelism gets in the way of optimal design.

IoT Data Streaming - Why MQTT and Kafka are a match made in heaven

  • Dominik Obermaier, HiveMQ
  • Margaretha Erber, Hive MQ

A fast and efficient integration of end device data into data processing systems is becoming increasingly important in the Internet of Things. Factors such as secure and reliable data transmission, real-time data processing and the analysis of huge amounts of data afterwards play a major role.

Kafka error handling patterns and best practices

  • Hemant Desale, Goldman Sachs
  • Aruna Kalagnanam, Goldman Sachs

Transaction Banking from Goldman Sachs is a high volume, latency sensitive digital banking platform offering. We have chosen an event driven architecture to build highly decoupled and independent microservices in a cloud native manner and are designed to meet the objectives of Security...

Lessons from building a stream-first metadata platform

  • Shirshanka Das, Acryl Data

We'll learn:

  • How different data discovery / catalog tools are architected and the tradeoffs in each kind of architecture
  • How streaming architectures can benefit metadata
  • How event-driven metadata architectures can supercharge your data productivity and governance workflows at your company

Look how easy it is to go from events to blazing-fast analytics

  • Neha Pawar , StarTree

Apache Kafka is the de facto standard for real-time event streaming, but what do you do if you want to perform user-facing, ad-hoc, real-time analytics too? That's a hard problem.

Managing multiple event types in a single topic with Schema Registry

  • Bill Bejeck, Confluent

With Apache Kafka, it's typical to place different events in their own topic. But different event types can be related. Consider customer interactions with an online retailer. The customer searches through the site and clicks on various items before deciding on a final purchase.

Mind the App: How to Monitor Your Kafka Streams Applications

  • Bruno Cadonna, Confluent

You cannot operate what you cannot measure. In this talk, I am going to present the built-in metrics framework of Kafka Streams that supports monitoring Kafka Streams applications.

Mistakes - I’ve made a few. Blunders in event-driven architecture

  • Simon Aubury, Simple Machines

Building systems around an event-driven architecture is a powerful pattern for creating awesome data intensive applications. Apache Kafka simplifies scalability and provides an event-driven backbone for service architectures.

A Modern C++ Kafka API

  • Kenneth Jia, Morgan Stanley

We wanted to embed a Kafka producer/consumer in C++ and decided to use ""librdkafka"", a robust C/C++ library that is open source, well-maintained, and widely used.

Kafka’s New Control Plane: The Quorum Controller

  • Colin McCabe, Confluent

Data such as the location of partitions and the configuration of topics are stored outside of Kafka itself, in a separate ZooKeeper cluster. In 2019, we outlined a plan to break this dependency and bring metadata management into Kafka itself through a dynamic service that runs inside the Kafka...

Monitoring and Resiliency Testing our Apache Kafka Clusters at Goldman Sachs

  • Sheikh Araf, Goldman Sachs
  • Ameya Panse, Goldman Sachs

In our payments platform at Goldman Sachs Transaction Banking, Apache Kafka plays a critical role as the messaging bus in our micro-services architecture. Being a part of the financial service industry we need to ensure high-availability of our platform and quick response time during failures.

Navigating the obdervability storm with Kafka

  • Jose Manuel Cristobal, Adidas

When all your stores are closed, e-commerce becomes your bigger store and the most challenging.

Not Your Mother's Kafka - Deep Dive into Confluent Cloud Infrastructure

  • Gwen Shapira, Confluent

Confluent Cloud runs a modified version of Apache Kafka - redesigned to be cloud-native and deliver a serverless user experience.

Reusing Kafka Data Structure Between Projects

  • Laura Schornack, Northern Trust
  • Maureen Penzenik, Northern Trust

Cut the time of application delivery by reusing Kafka data structure between projects! Expecting boundaries and data definitions to remain consistent between source and consuming projects can be a constant source of surprise - a Kafka spiderweb.

Server Sent Events using Reactive Kafka and Spring Web flux

  • Gagan Solur Venkatesh, Egen Solutions Inc

SSE can be used in apps like live stock updates, that use one way data communications and also helps to replace long polling by maintaining a single connection and keeping a continuous event stream going through it.

Stream Data Deduplication Powered by Kafka Streams

  • Philipp Schirmer, Bakdata

Representations of data, e.g., describing news, persons or places, differ. Therefore, we need to identify duplicates, for example, if we want to stream deduplicated news from different sources into a sentiment classifier.

Stream Processing with Kafka and KSQL in Jupiter

  • Namit Mahuvakar, Jupiter

Describing the convenience of building an event driven application using stream processing and leveraging the power of KSQL. Events model our lives and actions be it machine or human generated.

Streaming with Structure

  • Salma Saeed, IBM
  • Kate Stanley, Red Hat

Using a publish-subscribe messaging system like Apache Kafka is a great way to minimise coupling between your applications. The stream history that Kafka provides allows consumers to come and go, without the producers ever being aware.

Temporal-Joins in Kafka Streams and ksqlDB

  • Matthias J. Sax, Confluent

Joins in Kafka Streams and ksqlDB are a killer-feature for data processing and basic join semantics are well understood. However, in a streaming world records are associated with timestamps that impact the semantics of joins: welcome to the fabulous world of temporal join semantics.

Twitter’s Apache Kafka Adoption Journey

  • Ming Liu, Twitter

Until recently, the Messaging team at Twitter had been running an in-house build Pub/Sub system, namely EventBus (built on top of Apache DistributedLog and Apache Bookkeeper, and similar in architecture to Apache Pulsar) to cater to our pubsub needs.

Understanding Kafka Produce and Fetch api calls for high throughput applications

  • Mik Kocikowski, Cloudflare

The data team at Cloudflare uses Kafka to process tens of petabytes a day. All this data is moved using the 2 foundational Kafka api calls: Produce (api key 0) and Fetch (api key 1).

Using Kafka with Rust: from Zero to "One"

  • Abhishek Gupta, Microsoft

Rust is a fast, memory-efficient language with a rich type system and a set of tools to make you productive.

What is it like to bury "dinosaurs" (legacy systems) using Kafka in a traditional investment house?

  • Yue Wang, Newton Investment Management

Newton Investment Management as a traditional investment management house used to have its IT systems communicating with each other by connecting to a centralised database.

What is the State of my Kafka Streams Application? Unleashing Metrics

  • Neil Buesing, Kinetic Edge

Just as the Apache Kafka Brokers provide JMX metrics to monitor your cluster's health, Kafka Streams provides a rich set of metrics for monitoring your application's health and performance.

Developing a custom Kafka connector? Make it shine!

  • Igor Buzatović, Porsche Digital Croatia d.o.o.

Some people see their cars just as a means to get them from point A to point B without breaking down halfway, but most of us want it also to be comfortable, performant, easy to drive, and of course - to look good. We can think of Kafka Connect connectors in a similar way.

Why Kafka Works the Way It Does (And Not Some Other Way)

  • Tim Berglund, StarTree

Studying the ""how"" of Kafka makes you better at using Kafka, but studying its ""whys"" makes you better at so much more.

Writing Blazing Fast, and Production-Ready Kafka Streams apps in less than 30 min using Azkarra

  • Florian Hussonnois, StreamThoughts

If you have already worked on various Kafka Streams applications before, then you have probably found yourself in the situation of rewriting the same piece of code again and again.

Lightning

5 lessons learned for successful migration to Confluent cloud

  • Natan Silinitsky, wix.com

Confluent Cloud makes Devops engineers lives a lot more easier. Yet moving 1500 microservices, 10K topics and 100K partitions to a multi-cluster Confluent cloud can be a challenge.

Asynchronous Transaction Processing With Kafka as a Single Source of Truth - Mobile ticketing system

  • Dalibor Blazevic

Our backend system (ERP, CRM, Billing) is completely cloud, asynchronous Microservices and Kafka based. No databases at all.

Automatic provisioning Confluent Cloud Kafka cluster

  • Paweł Dudziński, Sauce Labs

I'd like to talk about how we manage our Confluent Cloud Kafka clusters.

Building a Codeless Log Pipeline w/ Confluent Sink Connector

  • Pollyanna Valente, Loggi
  • Daniel Sousa, Loggi

Kubernetes became the de-facto standard for running cloud-native applications. And many users turn to it also to run stateful applications such as Apache Kafka.

Building Stateful applications on Streaming Platforms

  • Premjit Mishra, Dell Boomi

Can and should Apache Kafka replace a database? How long can and should I store data in Kafka? How can I query and process data in Kafka? These are common questions that come up more and more.

Confluent Cloud Networking

  • Rajan Sundaram, Confluent

Introduction to networking options available in Confluent Cloud Self Serve provisioning of confluent Kafka clusters.

EDA Governance Model: a multicloud approach based on GitOps

  • Marco Laucelli, Galeo Tech
  • Alejandro Alija, Galeo Tech

The first question that arises when you start a new EDA project is how to govern the system? An entire ecosystem of applications, backends, events, and APIs must co-exist under the same architecture.

Enforcing Schemas with Kafka Connect

  • David Navalho, Marionete
  • Anatol Lutski, Saxo Bank

Applying some measure of governance over how schemas are managed helps ensure good quality data, as well as better lineage tracking and governance.

Error Resilient Design: Building Scalable & Fault-Tolerant Microservices with Kafka

  • Asaf Halili, Oribi

Handling failures is important, but it’s a must have when a product handles sensitive data. It also becomes exponentially harder in the world of microservices, since a failure can happen in any of the services and even in their dependencies.

From legacy systems to microservices and back

  • Andera Gioia, Quantyca

Legacy systems are the kings of our IT architecture. They rule the evolution of the technology ecosystem that hosts them thanks to the control they have acquired over time on core data and key business processes.

Kickstart your Kafka with Faker Data

  • Francesco Tisiot, Aiven

We all love to play with the shiny toys, but an event stream with no events is a sorry sight. In this session you’ll see how to create your own streaming dataset for Apache Kafka using Python and the Faker library.

Near Real time Streaming and Data Processing of Ceilometer data using Kafka

  • Rahul Gite, University of Maryland Baltimore County

The planetary boundary layer (PBL) is the lowest part of the atmosphere, ranging anywhere between 100 and 2000 m above the surface of the ground. The planetary boundary layer height (PBLH) plays a vital role in the environment-related study of air pollutants.

Analytics in Interactive Gaming - The Kafka Way

  • Naresh Yegireddi, PlayStation

Being a pioneer in the interactive gaming industry, SONY PlayStation has played a vital role in implementing technological advancements thus help bringing global video gaming community together.

Schema Management: the elephant in the Cloud room

  • Antón Rodríguez, New Relic

Schema management is a key component in every big Event Streaming platform. The Schema Registry solution has several advantages: better Data Quality, more performant, Data Evolvability, etc.

Single Message Transforms (SMT) in Kafka Connect

  • Betül Çetinkaya, Trendyol
  • Okan Yıldırım, Trendyol

Trendyol was established in 2010 to provide a seamless e-commerce experience to our customers and vendors.

The secret life of Kafka JVM Heap

  • Adithya Chandra, Confluent

In this session I will talk about our experience with the Java Garbage collector and Kafka in production.

Why you should have a Schema Registry

  • David Hettler, Celonis SE

Kafka moves blobs of data from one place to another. That's its job. Kafka doesn't care what the blob is or what it looks like. This can be a boon because it's simple and it allows for a multitude of use cases.

Writing Kafka applications without Kafka server access

  • Zoltan Balogh, IBM

Developers creating Apache Kafka client applications to be used with ""locked down"" (for example they're in a Cloud deployment) Kafka service endpoints have a unique problem: These developers do not have access to the Kafka server backends!

0-330km/h: Porsche's Data Streaming Journey

  • Sridhar Mamella, Porsche AG

The auto industry is midst a data revolution that is transforming how companies do business. Once a scarce resource, data has now become abundant and cheap.

Confluent Private Cloud

  • Rohit Bakhshi, Confluent

Cloud-native data infrastructure, such as Confluent and Kubernetes, combine well together to enable teams to use declarative spec based automation (GitOps) for deployment and management.

Sponsored

Accelerating Innovation with Apache Kafka, Heikki Nousiainen

  • Heikki Nousiainen, Aiven

Being a pioneer in the interactive gaming industry, SONY PlayStation has played a vital role in implementing technological advancements thus help bringing global video gaming community together.

Why SQL?

  • Kenny Gorman, MongoDB

While SQL is a simple declarative language, it can be used in very advanced ways when querying streams of data on Kafka

One Click Streaming Data Pipelines & Flows | Leveraging Kafka & Spark

  • Ido Friedman, Equalum

The Apache Kafka ecosystem is very rich with components and pieces that make for designing and implementing secure, efficient, fault-tolerant and scalable event stream processing (ESP) systems.

Hybrid Streaming Analytics for Apache Kafka Users

  • Firat Tekiner, Google

The data that organizations are required to analyze in order to make informed decisions is growing at an unprecedented rate.

Achieving Real-Time Analytics at Hermes

  • Zulf Qureshi, HVR
  • Stefan Roessier, Hermes

Hermes, Germany's largest post-independent logistics service provider for deliveries, had one main goal—make faster and smarter data-driven business decisions.

Should we manage events like APIs?

  • Kim Clark, IBM

APIs have become ubiquitous as a way of exposing the capabilities of the enterprise both internally and externally.

Integrating Confluent and Azure – Kickstart event streaming between Azure Cosmos DB and Confluent Cloud

  • Alicia Moniz, Confluent
  • Ramya Orunganti, Microsoft

Customers often face challenges while building and managing distributed systems like Apache Kafka as it is a complex and resource intensive process.

Keeping Analytics Data Fresh in a Streaming Architecture

  • John Neal, Qlik

We'll look in depth at how Qlik Replicate can be used to continuously stream changes from a source database into Apache Kafka. From there, we'll explore how a purpose-built consumer can be used to provide the bridge between Apache Kafka and an analytics application such as Qlik Sense.

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams - Kafka Summit Europe 2021

  • Jonathan Schabowsky, Solace

Ever wish you had a way to view and visualize graphically the relationships between schemas, topics and applications? In this talk we will show you how to do that and get more value from your Kafka Streaming infrastructure using an event portal.

Continuous Intelligence - Streaming Apps That Are Always In Sync

  • Simon Crosby, Swim

We will show how easy it is to build, deploy and run distributed, highly available event streaming applications that analyze data from hundreds of millions of sources - petabytes per day. The architecture is intuitively appealing and blazingly fast.

Cloud native Kafka

  • Sascha Holtbruegge, SVA
  • Margaretha Erber, Hive MQ

In this talk we want to deep dive on the different types of joins, with a focus of their temporal aspect. Furthermore, we relate the individual join operators to the overall "time engine" of the Kafka Streams query runtime and explain its relationship to operator semantics.

Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise

  • Claudio Acquaviva, Kong, Inc.

Learn how Kong Konnect Enterprise can complement Kafka Event Streaming, exposing it to new and external consumers while applying specific and critical policies to control its consumption, including API key, OAuth/OIDC and others for authentication, rate limiting, caching, log processing, etc.

Streaming data in the cloud with Confluent and MongoDB Atlas

  • Robert Waters, MongoDB

In this session you will learn how to setup and configure the Confluent Cloud with MongoDB Atlas.

SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architecture

  • Yassine Faihe, SingleStore

In this session, you will learn how Kafka and SingleStore enable modern, yet simple data architecture to analyze both fast paced incoming data as well as large historical datasets. In particular, you will understand why SingleStore is well suited process data streams coming from Kafka.

A Look into the Mirror: Patterns and Best Practices for MirrorMaker2

  • Cliff Gilmore, Confluent

In this session you will learn about patterns, best practices, and learnings compiled from running MirrorMaker2 in production at every scale.

Apicurio Registry: Event-driven APIs & Schema governance for Apache Kafka

  • Hugo Guerrero, Red Hat
  • Fabian Martinez, Red Hat

Apicurio Registry is an end-to-end solution to store API definitions and schemas for Kafka applications. The project includes serializers, deserializers, and additional tooling.

Comparing three data ingestion approaches where Apache Kafka integrates with a distributed graph database in real time

  • Benyue (Emma) Liu, TigerGraph

Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. In this session, we will present the high-level architecture in three different approaches and demo the data streaming process.

Stream processing IoT time series data with Kafka & InfluxDB

  • Al Sargent, InfluxData

Time series data is everywhere -- connected IoT devices, application monitoring & observability platforms, and more. Once this session is complete, you’ll be able to connect a Kafka queue to an InfluxDB instance as the beginning of your own time series data pipeline.

Lessons learned making Confluent Cloud

  • Addison Huddy, Confluent
  • Dan Rosanova, Confluent

Join a candid conversation between Confluent product managers, Dan Rosanova and Addison Huddy, as they reflect on the top 10 lessons learned building Confluent Cloud.

Introducing KSML: Kafka Streams for low code environments

  • Jeroen van Disseldorp, Axual
  • Ton van Bart, Axual

In this session we will present a new declarative approach to unlock Kafka Streams, called KSML. After this session you will be able to write streaming applications yourself, using only a few simple basic rules and Python snippets.

Better Kafka Performance Without Changing Any Code

  • Simon Ritter

We'll explore how making changes to the JVM design can eliminate the problems of garbage collection pauses and raise the throughput of applications. For cloud-based Kafka applications, this can deliver both lower latency and reduced infrastructure costs. All without changing a line of code!

Workshops

Applying ML on your Data in Motion with AWS and Confluent

  • Joseph Morais, Confluent
  • Kanchan Waikar, Amazon Web Services

Join experts from Confluent and AWS to learn how to build Apache Kafka®-based streaming applications backed by machine learning models. Adopting the recommendations will help you establish repeatable patterns for high performing event-based apps.

Scaling an Event-Driven Architecture with IBM and Confluent

  • Antony Amanse, IBM
  • Anton McConville, IBM

Scaling an Event-Driven Architecture with IBM and Confluent | Antony Amanse and Anton McConville, IBM

Azure Labs: Confluent on Azure Container Services & Real-time Search with Redis

  • Alicia Moniz, Confluent
  • Ramya Orunganti, Microsoft

Azure Labs: Confluent on Azure Container Services & Real-time Search with Redis | Alicia Moniz, Confluent and Ramya Orunganti, Microsoft