Kafka Summit San Francisco 2018

Panel - Microsoft, Slack, Confluent, University of Cambridge

We will cover a new solution and discuss the challenges in replicating database transactions, consisting of multiple ordered operations, to Kafka and through to the end consumer application. Be it a Z mainframe, DB2, Oracle or other database, there exists a need for Kafka consumer applications to understand the transactional context and operation ordering of the original source database’s transactions.

The solution will focus on how to layer the guarantees of a relational database with respect to ACID transaction properties over Kafka and provide them to an end consumer. Challenges to be discussed include:
-How to maintain Kafka’s performance while providing greatly improved semantic guarantees
-How to address the simultaneous need to maintain topic per database table representations in Kafka and yet provide the Kafka consumer the order of operations within a transaction that apply to any number of topics
-A demonstration of how a source database’s referential integrity-constrained tables can have their operational data replicated through Kafka and ultimately applied to another database with the same constraints
-Mechanisms for consumer applications to leverage this new functionality to process consumed transactions exactly once
-How to handle outages in the middle of replication and ensure semantic guarantees are maintained from the consumer’s perspective

The talk will include a demonstration of transactionally consistent, ordered, database replication leveraging a source database, IBM IIDR CDC Replication to Message Hub in the cloud, and resultant consumer output.

This session discusses end-to-end use cases such as connected cars, smart home or healthcare sensors, where you integrate Internet of Things (IoT) devices with enterprise IT using open source technologies and standards. MQTT is a lightweight messaging protocol for IoT. However, MQTT is not built for high scalability, longer storage or easy integration to legacy systems. Apache Kafka is a highly scalable distributed streaming platform, which ingests, stores, processes and forwards high volumes of data from thousands of IoT devices.

This session discusses the Apache Kafka open source ecosystem as a streaming platform to process IoT data. See a live demo of how MQTT brokers like Mosquitto or RabbitMQ integrate with Kafka, and how you can even integrate MQTT clients to Kafka without MQTT Broker. Learn how to analyze the IoT data either natively on Kafka with Kafka Streams/KSQL or on an external big data cluster like Spark, Flink or Elasticsearch leveraging Kafka Connect.

In this talk we will present how we in Schibsted have set up a new global streaming data platform using Kafka and Kafka Streams, replacing a homegrown solution based on Kinesis and micro batches in Amazon S3.

Schibsted is a global media and marketplaces company with over 200 million users across the world. With our new streaming platform, we aim to deliver better performance and enable new features, such as self-serve for our data consumers.

In this talk, we will present some of the ways this new platform enables collaboration across Schibsted, as well as some of the challenges we have faced. A number of collaborations with various teams in Schibsted are underway to build projects on top of this new platform. Examples include building tools for experimentation on our sites, visualization of user behavior and data quality checks. Our long-term goal is to provide a self-serve platform for real-time processing of data, enabling our data users to quickly create new data-driven applications. Data and analytics is a central part of Schibsted’s strategy, and we believe the streaming platform will play a significant role in building a global data-driven organization.

If you collect billions of data points every day and create billions more sending and tracking messages, then you know you need to get your infrastructure right. Our clients use Braze to engage their users over their lifecycle via push notifications, emails, in-app messages and more. Using our Currents product, clients can enable multiple configurable integrations to export this event data in real time to a variety of third-party systems, allowing them to tightly integrate with the rest of their operations and understand the impacts of their engagement strategy.

We use Kafka and the Kafka ecosystem to power this high volume real-time export. As you’d expect in a big data environment, we take data collected from a variety of sources—our SDKs, email partner APIs, our own systems—and produce it to Kafka, with topics for each type of event (about 30 types). Kafka Streams filters and transforms this data according to the configurations set by our clients. Clients can choose which types of events should be sent to which third-party systems. Kafka Connect helps to export the data to third-party systems in real time using custom developed connectors. We run a connector instance for each integration for each customer that consumes from the integration-specific topic. On top of it all, we built a service to manage the pipeline. The service provides configurations to the Streams application and also creates topics for new integrations and uses the Connect REST API to create and manage connectors.

In this talk, I will discuss:
-How we started our journey in designing this large-scale streaming architecture
-Why streaming technologies were necessary to solve our technology and business issues
-The lessons we learned along the way that can help you with your Kafka-based architecture

A company’s business processes typically span more than one microservice. In an e-commerce company, for example, a customer order might involve microservices for payments, inventory, shipping and more. Implementing long-running, asynchronous and complex collaboration of distributed microservices is challenging. How can we ensure visibility of cross-microservice flows and provide status and error monitoring? How do we guarantee that overall flows always complete, even if single services fail? Or, how do we at least recognize stuck flows so that we can fix them?

In this talk, I’ll demonstrate an approach based on real-life projects using the open source workflow engine zeebe.io to orchestrate microservices. Zeebe can connect to Kafka to coordinate workflows that span many microservices, providing end-to-end process visibility without violating the principles of loose coupling and service independence. Once an orchestration flow starts, Zeebe ensures that it is eventually carried out, retrying steps upon failure. In a Kafka architecture, Zeebe can easily produce events (or commands) and subscribe to events that will be correlated to workflows. Along the way, Zeebe facilitates monitoring and visibility into the progress and status of orchestration flows. Internally, Zeebe works as a distributed, event-driven and event-sourced system, making it not only very fast but horizontally scalable and fault tolerant—and able to handle the throughput required to operate alongside Kafka in a microservices architecture. Expect not only slides but also fun little live-hacking sessions and real-life stories.

Blizzard’s global data platform has become a driving force in both business and operational analytics. As more internal customers onboard with the system, there is increasing demand for custom applications to access this data in near real time. In order to avoid many independent teams with varying levels of Kafka expertise all accessing the firehose from our critical production Kafkas, we developed our own pub-sub system on top of Kafka to provide specific datasets to customers on their own cloud deployed Kafka clusters.

Internet of Things use cases are a perfect match for processing with a streaming platform such as Kafka and the Confluent Platform. Some of the questions to be answered are: How do we feed the data from our devices into Kafka? Do we directly send data to Kafka? Is Kafka accessible from outside the organization over the internet? What if we want to use a more specific IoT protocol such as MQTT or CoAP in between? How would we integrate it with Kafka? How can we enrich IoT streaming data with static data sitting in a traditional system?

This session will provide answers to these and other questions using a fictitious use case of a trucking company. Trucks are constantly sending data about position and driving habits, which can be used to derive real-time information and actions. A large part of the presentation will be a live demo. The demo will show the implementation of the pipeline incrementally: starting with sending the truck movement events directly to Kafka, then adding MQTT to the sensor data ingestion, followed by using Kafka Streams and KSQL to apply stream processing on the information received. The final pipeline will demonstrate the application of Kafka Connect with MQTT and JDBC source connectors for data ingestion and event stream enrichment, and Kafka Streams and KSQL for stream processing. The key takeaway is the live demonstration of a working end-to-end IoT streaming data ingestion pipeline using Kafka technologies.

With the rise of large-scale real-time computation, there is a growing need to link legacy MySQL systems with real-time platforms. Pinterest has a hundred billion pins stored in MySQL at the scale of a 100TB and most of this data is needed for building data-driven products for machine learning and data analytics.

This talk discusses how Pinterest designed and built a continuous database (DB) ingestion system for moving MySQL data into near-real-time computation pipelines with only 15 minutes of latency to support our dynamic personalized recommendations and search indices. Pinterest helps people discover and do things that they love. We have billions of core objects (pins/boards/users) stored in MySQL at the scale of 100TB. All this data needs to be ingested onto S3/Hadoop for machine learning and data analytics. As Pinterest is moving towards real-time computation, we are facing a stringent service-level agreement requirement such as making the MySQL data available on S3/Hadoop within 15 minutes, and serving the DB data incrementally in stream processing. We designed WaterMill: a continuous DB ingestion system to listen for MySQL binlog changes, publish the MySQL changelogs as an Apache Kafka® change stream and ingest and compact the stream into Parquet columnar tables in S3/Hadoop within 15 minutes.

We would like to share how we solved the problem of:
-Scalable data partitioning, efficient compaction algorithm
-Stories on schema migration, rewind and recovery
-PII (personally identifiable information) processing
-Columnar storage for efficient incremental query
-How the DB change stream powers other use cases such as cache invalidation in multi-datacenter
-How we deal with the issue of S3 eventual consistency and rate limiting; related technologies: Apache Kafka, stream processing, MySQL binlog processing, Amazon S3, Hadoop and Parquet columnar storage

Since its original introduction at Booking.com, Apache Kafka and overall concept of real-time data streaming have come a long way from being a complicated novelty to a common tool, used by a multitude of internal users ranging in their importance from the ad-hoc consumers to business-critical services powering up our property search engine.

Over the course of this talk we’ll dive deep into how a relatively small team of SREs is successfully managing a multi-cluster, multi-tenant setup of Kafka and its surrounding ecosystem capable of transporting millions of messages per day. We’ll discuss challenges they faced along their way while building this platform and take a close look not only at application but also at architectural-level decisions they made to overcome them. Surely, we will also review what kind of tooling and automation team is using to stay sane during the day and sleep well during the night.

For several years, LinkedIn has been using Kafka MirrorMaker as the mirroring solution for copying data between Kafka clusters across data centers. However, as LinkedIn data continued to grow, mirroring trillions of Kafka messages per day across data centers uncovered the scale limitations and operability challenges of Kafka MirrorMaker. To address such issues, we have developed a new mirroring solution, built on top our stream ingestion service, Brooklin. Brooklin MirrorMaker aims to provide improved performance and stability, while facilitating better management through finer control of data pipelines. Through flushless Kafka produce, dynamic management of data pipelines, per-partition error handling and flow control, we are able to increase throughput, better withstand consume and produce failures and reduce overall operating costs. As a result, we have eliminated the major pain points of Kafka MirrorMaker. In this talk, we will dive deeper into the challenges LinkedIn has faced with Kafka MirrorMaker, how we tackled them with Brooklin MirrorMaker and our plans for iterating further on this new mirroring solution.

Don’t you love signing in to your favorite application only to find out it’s undergoing scheduled maintenance? I know I don’t! Designing scalable systems is a challenge all software engineers face. Ultimately this means you will find yourself in a situation where you need to upgrade your software engines for better performance.

In this session, I will guide you through the experience of swapping out PagerDuty’s original prop engine consisting of a Cassandra queue-based system to that of a 747 which utilizes Kafka not only for ingestion of customer data but also internal heuristics and back pressure support. This includes the most critical techniques used by our engineering team which allowed us to validate that our proposed solution would in fact scale, all while continuing to consume millions of customers events and ensuring that our users get notified in a timely manner.

Audience members will leave with the necessary insight to have a smooth transition to a Kafka-based system all while not interrupting day-to-day operations for their customers.

Building cloud-based microservices can be a challenge when the system of record is a relational database residing on an on-premise mainframe. The challenge lies in the ability to efficiently and cost-effectively access the ever-increasing amount of data. Express Scripts is reimagining its data architecture to bring best-in-className user experience and provide the foundation of next-generation applications.

This talk will showcase how Kafka plays a key role within Express Scripts’ transformation from mainframe to a microservice-based ecosystem, ensuring data integrity between two worlds. It will discuss how change data capture (CDC) is leveraged to stream data changes to Kafka, allowing us to build a low-latency data sync pipeline. We will describe how we achieve transactional consistency by collapsing all events that belong together onto a single topic, yet have the ability to scale out to meet the real time SLAs and low-latency requirements through means of partitions. We will share our Kafka Streams configuration to handle the data transformation workload. We will discuss our overall Kafka cluster footprint, configuration and security measures.

Express Scripts Holding Company is an American Fortune 100 company. As of 2018, the company is the 25th largest in the U.S. as well as one of the largest pharmacy benefit management organizations in the U.S. Customers rely on 24/7 access to our services, and need the ability to interact with our systems in real time via various channels such as web and mobile. Sharing our mainframe t0 microservices migration journey, our experiences and lessons learned would be beneficial to other companies venturing on a similar path.

Debezium (noun | de·be·zi·um | /dɪ:ˈbɪ:ziːəm/): secret sauce for change data capture (CDC) streaming changes from your datastore that enables you to solve multiple challenges: synchronizing data between microservices, gradually extracting microservices from existing monoliths, maintaining different read models in CQRS-style architectures, updating caches and full-text indexes and feeding operational data to your analytics tools

Join this session to learn what CDC is about, how it can be implemented using Debezium, an open source CDC solution based on Apache Kafka and how it can be utilized for your microservices. Find out how Debezium captures all the changes from datastores such as MySQL, PostgreSQL and MongoDB, how to react to the change events in near real time and how Debezium is designed to not compromise on data correctness and completeness also if things go wrong. In a live demo we’ll show how to set up a change data stream out of your application’s database without any code changes needed. You’ll see how to sink the change events into other databases and how to push data changes to your clients using WebSockets.

In this talk we will share some stories and patterns from customers who have built streaming pipelines and event-driven systems using Confluent Cloud in combination with Google Cloud Platform-native analytics tools, such as BigQuery and Dataflow. We will discuss what Confluent Cloud enables for hybrid deployments and how and why to mix and match platform-native and platform-neutral tools.

Kafka Streams, Apache Kafka’s stream processing library, allows developers to build sophisticated stateful stream processing applications which you can deploy in an environment of your choice. Kafka Streams is not only scalable, but fully elastic allowing for dynamic scale-in and scale-out as the library handles state migration transparently in the background. By running Kafka Streams applications on Kubernetes, you will be able to use Kubernetes powerful control plane to standardize and simplify the application management—from deployment to dynamic scaling.

In this technical deep dive, we’ll explains the internals of dynamic scaling and state migration in Kafka Streams. We’ll then show, with a live demo, how a Kafka Streams application can run in a Docker container on Kubernetes and the dynamic scaling of an application running in Kubernetes.

Streaming SQL is the latest hot technology that takes a fundamentally familiar topic and adds modern techniques. SQL is ubiquitous and probably will continue to be for a long time, so if you start with that foundation and add modern streaming functionality, you get a technology that is ideal for querying real-time data streams. The widespread familiarity of SQL will help with adopting it in new environments and new use cases.

However, there are many aspects of streaming SQL that are challenging to grasp, including continuous queries, late arrival of events and different types of windowing. The trick is to not only leverage these new functionalities but also to present them to end users in a logical and meaningful way. We will explain various aspects of streaming SQL by showing visual representations of the different semantics provided.

In this talk, we will use KSQL to illustrate various examples. Attendees will get a better understanding on:
-Defining various aggregation window types and their behavior
-How to understand time scales and granularity
-The ability to replay results for testing and for creating new outputs

Big Fish Games is a leading producer and distributor of casual and mid-core games. They have distributed more than 2.5 billion games to customers in 150 countries, representing over 450 unique mobile games and over 3,500 unique PC games.

Apache Kafka is used in our pipeline to process data generated across game play. Recently, we introduced real-time analytics of game data using Kafka Streams integrated with Elasticsearch. This allows us to monitor the results of live operations (aka, live ops such as weekend events, limited time offers and user acquisition campaigns) and to make changes to these events after they have gone live. The result is a better game experience, better control of the game’s economy and overall better optimization of the live ops.

This presentation will include a detailed explanation of how we used Kafka Streams to transform raw data into Elasticsearch documents, and how the application was scaled to over a million daily active users. It will also touch on the limitations discovered with Kafka Connect integration with Elasticsearch and how to use Elasticsearch bulk processing with Kafka Streams. The presentation will also discuss how Dropwizard can provide a framework for monitoring and alerting a Kafka Stream application. Lastly, a demonstration of the real-time dashboards which are powered by Kafka Streams will be included.

Apache Kafka added a powerful stream processing library in mid-2016, Kafka Streams, which runs on top of Apache Kafka. The community has embraced Kafka Streams with many early adopters, and the adoption rate continues to grow. Large to mid-size organizations have come to rely on Kafka Streams in their production environments. Kafka Streams has many advanced features to make applications more robust.

The point of this presentation is to show users of Kafka Streams some of the latest and greatest features, as well as some that may be advanced, that can make streams applications more resilient. The target audience for this talk are those users already comfortable writing Kafka Streams applications and want to go from writing their first proof-of-concept applications to writing robust applications that can withstand the rigor that running in a production environment demands.

The talk will be a technical deep dive covering topics like:
-Best practices on configuring a Kafka Streams application
-How to meet production SLAs by minimizing failover and recovery times: configuring standby tasks and the pros and cons of having standby replicas for local state
-How to improve resiliency and 24×7 operability: the use of different configurable error handlers, callbacks and how they can be used to see what’s going on inside the application
-How to achieve efficient scalability: a thorough review of the relationship between the number of instances, threads and state stores and how they relate to each other

While this is a technical deep dive, the talk will also present sample code so that attendees can view the concepts discussed in practice. Attendees of this talk will walk away with a deeper understanding of how Kafka Streams works, and how to make their Kafka Streams applications more robust and efficient. There will be a mix of discussion.

All things change constantly! And dealing with constantly changing data at low latency is pretty hard. It doesn’t need to be that way. Apache Kafka, the de facto standard open source distributed stream processing system. Many of us know Kafka’s architectural and pub/sub API particulars. But that doesn’t mean we’re equipped to build the kind of real-time streaming data systems that the next generation of business requirements are going to demand. We need to get on board with streams!

Viktor Gamov will introduce Kafka Streams and KSQL—an important recent addition to the Confluent Open Source platform that lets us build sophisticated stream processing systems with little to no code at all! They will talk about how to deploy stream processing applications and look at the actual working code that will bring your thinking about streaming data systems from the ancient history of batch processing into the current era of streaming data!

P.S. No prior knowledge of Kafka Streams, KSQL or Ghostbusters needed!

At Bloomberg, we are building a streaming platform with Apache Kafka, Kafka Streams and Spark Streaming to handle high volume, real-time processing with rapid derivative market data. In this talk, we’ll share the experience of how we utilize Kafka Streams Processor API to build pipelines that are capable of handling millions of market movements per second with ultra-low latency, as well as performing complex analytics like outlier detection, source confidence evaluation (scoring), arbitrage detection and other financial-related processing.

We’ll cover:
-Our system architecture
-Best practices of using the Processor API and State Store API
-Dynamic gap session implementation
-Historical data re-processing practice in KStreams app
-Chaining multiple KStreams apps with Spark Streaming job

Monolithic architectures should become a thing of the past sooner or later—preferably, sooner, of course … However, as it usually is with shiny pictures of a perfect future outcome vs. sobering facts of the reality, moving from a monolith to microservices is sometimes easier said than done.

This talk will cover many lessons we learned during this process and how Kafka, change tracking and KSQL were successfully leveraged to break down a SQL server monolith while at the same time allowing SAP Concur to scale its backends to billions of daily transactions, enabling several new features and functionalities.

Learn how we:
-Leveraged Kafka Connect change tracking to propagate data changes out of SQL Server
-Used Kafka to provide a highly performant and horizontally scalable central nervous system for SAP Concur events
-Implemented KStreams/KSQL to perform real-time joins, aggregations, windowing and webhook integrations

You’ve built your streaming data platform. The early adopters are “all in” and have developed producers, consumers and stream processing apps for a number of use cases. A large percentage of the enterprise, however, has expressed interest but hasn’t made the leap. Why?

In 2014, Bayer Crop Science (formerly Monsanto) adopted a cloud first strategy and started a multi-year transition to the cloud. A Kafka-based cross-datacenter DataHub was created to facilitate this migration and to drive the shift to real-time stream processing. The DataHub has seen strong enterprise adoption and supports a myriad of use cases. Data is ingested from a wide variety of sources and the data can move effortlessly between an on premise datacenter, AWS and Google Cloud. The DataHub has evolved continuously over time to meet the current and anticipated needs of our internal customers. The “cost of admission” for the platform has been lowered dramatically over time via our DataHub Portal and technologies such as Kafka Connect, Kubernetes and Presto. Most operations are now self-service, onboarding of new data sources is relatively painless and stream processing via KSQL and other technologies is being incorporated into the core DataHub platform.

In this talk, Bob Lehmann will describe the origins and evolution of the Enterprise DataHub with an emphasis on steps that were taken to drive user adoption. Bob will also talk about integrations between the DataHub and other key data platforms at Bayer, lessons learned and the future direction for streaming data and stream processing at Bayer.

When a new project, Global Trip Record was launched at American Express GBT and we were looking for a robust, scalable and fault-tolerant middleware to handle all the orchestration and connectivity needs of the project.

The existing solution was monolithic, and we wanted to convert that to a microservices framework, but the biggest challenge was managing the increasing number of external applications that are connected to the platform. Any slow external application or partner system connected to the platform was slowing down the entire platform. There is always a need for partner systems to go offline or a need to resend the entire day’s data, especially with a system like our data lake where the data volumes are huge.

After evaluating multiple solutions, we settled on Apache Kafka, and started with a simple implementation of around 100,000 messages to just decouple one partner system and the core platform.

Today, we are running our microservices (Docker) running in OpenShift (Kubernetes) processing Kafka Streams, running real-time anomaly detection using Kafka Streams, powering our data lake through Kafka, feeding our distributed caching layer (Apache Ignite) and connecting all internal and external systems using Kafka. With a total of more than 10 million messages per day, i.e., 1.5TB of data with just a small three-node cluster, we are one happy platform for over a year now. With the kind of stability, flexibility and success in our project, a lot of other teams started and will soon be in production with Kafka Steams. The powerful combination of Kafka and OpenShift has proven to be an easily scalable model with great elasticity to the entire platform.

Comcast manages over 2 million miles of fiber and coax, and over 40 million in home devices. This “outside plant” is subject to adverse conditions from severe weather to power grid outages to construction-related disruptions. Maintaining the health of this large and important infrastructure requires a distributed, scalable, reliable and fast information system capable of real-time processing and rapid analysis and response. Using Apache Kafka and the Kafka Streams Processor API, Comcast built an innovative new system for monitoring, problem analysis, metrics reporting and action response for the outside plant.

In this talk, you’ll learn how topic partitions, state stores, key mapping, source and sink topics and processors from the Kafka Streams Processor API work together to build a powerful dynamic system. We will dive into the details about the inner workings of the state store—how it is backed by a Kafka “changelog” topic, how it is scaled horizontally by partition and how the instances are rebuilt on startup or on processor failure. We will discuss how these state stores essentially become like materialized views in a SQL database but are updated incrementally as data flows through the system, and how this allows the developers to maintain the data in the optimal structures for performing the processing. The best part is that the data is readily available when needed by the processors. You will see how a REST API using Kafka Streams “interactive queries” can be used to retrieve the data in the state stores. We will explore the deployment and monitoring mechanisms used to deliver this system as a set of independently deployed components.

Apache Kafka has an immense capability for building data-intensive applications. The ability to model and scale to thousands of parallel real-time streams bring with it incredible fundamentals upon which you can build scalable systems of almost any nature. We also have the brave new world of serverless that causes much confusion as to how to use Functions as a Service (FaaS) and streaming together.

This talk builds a horizontally scalable auction platform with real-time marketplace analytics to work at the scale of eBay. The design uses the concepts of ""turning the database inside out” to build a model of how to make a stream platform works synergistically with FaaS/AWS Lambda. First, we explore the founding principles of the log, producer/consumer topics, partitions and events. We then build up to Kafka Streams stateless and stateful stream processing and KSQL. These principles are mapped onto simple use cases in order to establish how to build higher order functionality. These use cases are combined to develop an architecture that provides the design semantics required for a real-time auction system and marketplace intelligence. The architecture is composed of a queryable data fabric using Kafka Streams state stores, a high-throughput worker-queue using exactly-once semantics (EoS) Kafka consumers, and a queue-worker hook to drive AWS Lambda functions.

The end result is a real-time system that scales elastically to service millions of auction events and provide live, marketplace analytics. The audience will learn how to compose large scale applications using all of the Apache Kafka stack as well as how we view “turning the database inside out” when used in conjunction with serverless architectures.

One of the most powerful capabilities of both KSQL and the Kafka Streams library is that they allow us to easily express multiple “types” of join over continuous streams of data, and then have those joins be executed in distributed fashion by a self-organizing group of machines—but how many of us really understand the intrinsic qualities of a RIGHT OUTER STREAM-STREAM JOIN, SPAN(5 MINUTES, 2 MINUTES)? And what happens when that data can arrive late or out of order? In this talk we will explore the available streaming join options, covering common uses and examples for each, how to decide between them and illustrations of what’s really happening under the covers in the face of real-world data.

In this talk, we are sharing the experience of building Pinterest’s real-time Ads Platform utilizing Kafka Streams. The real-time budgeting system is the most mission-critical component of the Ads Platform as it controls how each ad is delivered to maximize user, advertiser and Pinterest value. The system needs to handle over 50,000 queries per section (QPS) impressions, requires less than five seconds of end-to-end latency and recovers within five minutes during outages. It also needs to be scalable to handle the fast growth of Pinterest’s ads business.

The real-time budgeting system is composed of real-time stream-stream joiner, real-time spend aggregator and a spend predictor. At Pinterest’s scale, we need to overcome quite a few challenges to make each component work. For example, the stream-stream joiner needs to maintain terabyte size state while supporting fast recovery, and the real-time spend aggregator needs to publish to thousands of ads servers while supporting over one million read QPS. We choose Kafka Streams as it provides milliseconds latency guarantee, scalable event-based processing and easy-to-use APIs. In the process of building the system, we performed tons of tuning to RocksDB, Kafka Producer and Consumer, and pushed several open source contributions to Apache Kafka. We are also working on adding a remote checkpoint for Kafka Streams state to reduce the time of code start when adding more machines to the application. We believe that our experience can be beneficial to people who want to build real-time streaming solutions at large scale and deeply understand Kafka Streams.

Airports are complex networks consisting of an immense number of systems that are necessary to keep the daily stream of passengers in constant motion. Connecting these systems in order to make the big picture transparent to the people running the show, authorities and last but not least the passengers is no simple endeavor.

In this talk I will describe a fictional airport and its effort to restructure the IT infrastructure around Kafka Streams to serve the real-time data needs of a busy airport. I will start by giving a brief overview of Kafka Streams, KSQL and the opportunities they offer for real-time stream processing. Following that we will explore the the target architecture, which relies heavily on manifested views to serve up-to-date data, while also persisting to a traditional data lake for larger analytics workflows. Additionally we will take a look at the generic data transformation framework that was created to minimize integration effort of the data receiving systems. To illustrate these ideas I will describe some examples of possible integrations: joining flight data with radar and weather data to predict arrival time at the gate down to the second, constantly updated processing data from the luggage conveyor belts as well as results from prediction models for passenger flow, and many more.

My team at Zalando fell in love with KStreams and their programming model straight out of the gate. However, as a small team of developers, building out and supporting our infrastructure while still trying to deliver solutions for our business has not always resulted in a smooth journey.

Can a small team of a couple of developers run their own Kafka infrastructure confidently and still spend most of their time developing code?

In this talk, we will dive into some of the problems we experienced while running Kafka brokers and Kafka Streams applications, as well as the consultations we had with other teams around this matter. We will outline some of the pragmatic decisions we made regarding backups, monitoring and operations to minimize our time spent administering our Kafka brokers and various stream applications.

You’re doing disk IO wrong, let ZFS show you the way. ZFS on Linux is now stable. Say goodbye to JBOD, to directories in your reassignment plans, to unevenly used disks. Instead, have 8K Cloud IOPS for $25, SSD speed reads on spinning disks, in-kernel LZ4 compression and the smartest page cache on the planet. (Fear compactions no more!)

Learn how Jet’s Kafka clusters squeeze every drop of disk performance out of Azure, all completely transparent to Kafka.
-Striping cheap disks to maximize instance IOPS
-Block compression to reduce disk usage by ~80% (JSON data)
-Instance SSD as the secondary read cache (storing compressed data), eliminating >99% of disk reads and safe across host redeployments
-Upcoming features: Compressed blocks in memory, potentially quadrupling your page cache (RAM) for free

We’ll cover:
-Basic Principles
-Adapting ZFS for cloud instances (gotchas)
-Performance tuning for Kafka
-Benchmarks

​

Security and compliance are key concerns for many organizations today and it is very important that we can meet these requirements in our platform. This is also extremely critical for customers who are adopting Confluent cloud offerings, since moving the streaming platform to cloud exposes new security and governance

In this session, we will discuss how Confluent is providing control and visibility to address these concerns and enable secure streaming platforms. We will cover the main pillars of IT security in access control (authentication, authorization), data confidentiality (encryption) and auditing.

LINE is a messaging service with 160+ million active users. Last year I talked about how we operate our Kafka cluster that receives more than 160 billion messages daily, dealing with performance problems to meet our tight requirement. Since last year we have deployed three more new clusters each for different purposes, such as one in different datacenter, one for security sensitive usages and so on, still keeping the fundamental concept: one cluster for everyone to use. While letting many projects using few multi-tenancy clusters greatly saves our operational cost and enables us to concentrate our engineering resources for maximizing their reliability, hosting multiple topics of different kinds of workload led us through a lot of challenges, too.

In this talk I will introduce how we operate Kafka clusters shared among different services, solving troubles we met to maximize its reliability. Especially, one of the most critical issues we’ve solved—delayed consumer Fetch request causing a broker’s network threads to be blocked—should be very interesting because it could have worse overall performance of brokers in a very common situation, and we have managed to solve it leveraging advanced technique such as dynamic tracing and tricky patch to control in-kernel behavior from Java code.

The Kafka Connect framework makes it easy to move data into and out of Kafka, and you want to write a connector. Where do you start, and what are the most important things to know? This is an advanced talk that will cover important aspects of how the Connect framework works and best practices of designing, developing, testing and packaging connectors so th.at you and your users will be successful. We’ll review how the Connect framework is evolving, and how you can help develop and improve it

Security in Kafka is a cornerstone of true enterprise production-ready deployment: It enables companies to control access to the cluster and limit risks in data corruption and unwanted operations. Understanding how to use security in Kafka and exploiting its capabilities can be complex, especially as the documentation that is available is aimed at people with substantial existing knowledge on the matter.

This talk will be delivered in a “hero journey” fashion, tracing the experience of an engineer with basic understanding of Kafka who is tasked with securing a Kafka cluster. Along the way, I will illustrate the benefits and implications of various mechanisms and provide some real-world tips on how users can simplify security management.

Attendees of this talk will learn about aspects of security in Kafka, including:
-Encryption: What is SSL, what problems it solves and how Kafka leverages it. We’ll discuss encryption in flight vs. encryption at rest.
-Authentication: Without authentication, anyone would be able to write to any topic in a Kafka cluster, do anything and remain anonymous. We’ll explore the available authentication mechanisms and their suitability for different types of deployment, including mutual SSL authentication, SASL/GSSAPI, SASL/SCRAM and SASL/PLAIN.
-Authorization: How ACLs work in Kafka, ZooKeeper security (risks and mitigations) and how to manage ACLs at scale

At ZipRecruiter, we connect employers with jobseekers. To do that, we record and process millions of events every day using Apache Kafka. Kafka availability is critical for our business. If Kafka is down—we are losing money.

In this talk, we aim to explain the path we’ve gone through after suffering from a production incident that occurred during a rolling upgrade. This incident led us to perform chaos engineering to get our team more acquainted with Kafka’s internals and how to deal with incidents. We will share some best practices of chaos engineering Kafka and learn how to overcome failures when they appear. We will also share some of our strategies on how to perform a cluster update—rolling upgrade vs. a blue green vs. active-passive approach, including pros and cons for each method.

We will thoroughly explore three key monitoring concepts in the broker, that will leave you an expert in identifying problems with the least amount of pain:
-Under-replicated Partitions: The mother of all metrics
-Request Latencies: Why your users complain
-Thread pool utilization: How could 80% be a problem?

We will also discuss the necessity of availability monitoring and how to use it to get a true picture of what your users see, before they come beating down your door!

Kubernetes is fast becoming the platform of choice for running distributed, containerized applications in the cloud. It has great features for availability, scalability, monitoring, ease of deployment, a rich set of tools and an extremely fast-growing ecosystem that is making it ever more useful. However, running stateful applications such as Kafka on Kubernetes is not a common practice today. At Intuit, we took an experimentation and data-driven approach for evaluating Kafka on Kubernetes in AWS.

In this talk, we will provide details of our functional and non-functional requirements, the experimental configuration and the details of the evaluation. The evaluation process included functional tests for producing/consuming messages, network isolation tests, cross-region tests as well as performance and stress tests. We will focus on the problems we ran into and how we addressed them. This talk will demonstrate a Kubernetes cluster running Kafka along with the details of how each component is configured. Specifically, we will cover the Kafka and ZooKeeper StatefulSets, the ConfigMaps used for storing the server.properties used by all brokers, the service objects for enabling access to the brokers, securing the data and, last but not least, integration with Splunk and Wavefront for logging and monitoring respectively.

As Expedia, the world’s largest online travel agency, moved to a multi Virtual Private Cloud (VPC) strategy in AWS, we faced the challenge of making our systems accessible, or using other systems, across many VPCs. In most cases, a secure internet-facing endpoint or doing VPC Peering should do the work, right?

But what if the system isn’t a typical HTTP-based microservice? What if it’s a distributed, partitioned and binary protocol-based system, where anyone talks to everyone all the time? That’s exactly what we encountered when we tried to make our Kafka accessible to our clients. We solved this problem by leveraging Apache Kafka®‘s distributive nature, using AWS’ new VPC Endpoint technology and their recent Network Load Balancer, some Route53 records and a bit of creativity!

In this session, I’ll dive into:
-Our use case: Kafka accessible to other VPCs
-Why we didn’t go with internet-facing endpoint or use VPC Peering
-A brief description on how VPC endpoints work
-Our solution to the problem: That’s where the fun starts.

At Uber, trillions of messages are delivered through Apache Kafka every day, for streaming applications, general pub/sub and task queueing. In this talk, we will share our system designs and experience of how to guarantee at-least-once delivery at scale, how to tolerate data center failures and how to run the system efficiently.

Kafka has become a critical part of most large-scale distributed infrastructures. We provide it as a self-service tool where topics are first-className citizens. Due to its nature, Kafka is used as a connector between projects of different teams. This is one thing that makes it challenging to provide such a service for operations, multi-tenancy and security in large organizations.

In this talk we will present how we built infrastructure that enables teams to use Kafka while keeping things secure and flexible, and while keeping the operational overload low. We’ll talk about our data model, cluster discovery, our end-to-end asymmetric encryption framework and identity management.

At Shopify we manage multiple Apache Kafka clusters in multiple locations in Google’s cloud platform. We deploy our Kafka clusters as Kubernetes StatefulSets, and we use other K8s workloads to implement different tasks. Automating critical and repetitive operational tasks is one of our top priorities.

In this talk we’ll discuss how we leveraged Kubernetes Custom Resources and Controllers to automate some of the key cluster operational tasks, to detect clusters configuration changes and react to these changes with required actions. We will go through actual examples we implemented at Shopify, how we solved the problem of cluster discovery and how we automated topics creation across different clusters with zero human intervention and safety controls.

This presentation is a customer story about France-based regional airline HOP! and their need to make better use of data that was contained in various applications. They also needed this information to be available in real time. As one can imagine, airlines manage a wide variety of information such as weather, customer information, flight plans, sensor data from planes and much more.

In this presentation, Joe will discuss how HOP! was delivering their data before and the limitations associated with delivering this data. Joe will then talk about HOP!’s selection of Kafka and HVR as a solution to enabling data availability and real-time information for analysis and action.

In this session, attendees will learn:
-How Kafka was selected and chosen as a solution for HOP!’s complex challenges
-Architecture and capabilities implemented that enabled data feeding from multiple sources to Kafka
-Considerations and challenges with this approach
-Business results and future plans

A business-oriented view, illustrating both process models and in-flight task progress, is critical to understanding organizational health, efficiency and alignment to strategic goals. The intent of this talk is to illustrate the real-time relationship between Kafka-managed events (event driven) and business architecture via actionable models (real-time analytics).

Takeaways:
-Understand how business views technology in terms of capabilities aligned to strategy.
-Introduce process model and performance views into an event-oriented dashboard. This view illustrates the organization in terms of collaborating human and automated services.
-Illustrate how system architecture dovetails into business goals with an aligned business/IT architectures.

Cisco e-commerce platform is a custom-built mission-critical platform which accounts for $40+ billion of Cisco’s revenue annually. It’s a suite of 35 different applications and 300+ services that powers product configuration, pricing, quoting and order booking across all Cisco product lines including hardware, software, services and subscriptions. It’s a B2B platform used by the Cisco sales team, partners and direct customers, serving 140,000 unique users across the globe. In order to improve customer experience and business agility, Cisco decided to transition the platform to cloud-native technologies, MongoDB, Elasticsearch and Kafka.

In this session, we will share details around:
-Kafka architecture
-How we are experiencing significant resiliency advantages, zero-downtime deployment and improved performance
-How we’ve implemented Kafka to pass data to 20+ downstream applications, removing point-to-point integrations, batch jobs and standardizing the handshake
-How are we using Kafka for pushing data for machine learning and analytics use cases
-Best practices and lessons learned

At Tinder, we have been using Kafka for streaming and processing events, data science processes and many other integral jobs. Forming the core of the pipeline at Tinder, Kafka has been accepted as the pragmatic solution to match the ever increasing scale of users, events and backend jobs. We, at Tinder, are investing time and effort to optimize the usage of Kafka solving the problems we face in the dating apps context. Kafka forms the backbone for the plans of the company to sustain performance through envisioned scale as the company starts to grow in unexplored markets. Come, learn about the implementation of Kafka at Tinder and how Kafka has helped solve the use cases for dating apps. Engage in the success story behind the business case of Kafka at Tinder.

Using Kafka as a platform messaging bus is common, but bridging communication between real-time and asynchronous components can become complicated, especially when dealing with serverless environments. This has become increasingly common in modern banking where events need to be processed at near-real-time speed. Serverless environments are well-suited to address these needs, and Kafka remains an excellent solution for providing the reliable, resilient communication layer between serverless components and dedicated stream processing services.

In this talk, we will examine some of the strengths and weaknesses of using Kafka for real-time communication, some tips for efficient interactions with Kafka and AWS Lambda, and a number of useful patterns for maximizing the strengths of Kafka and serverless components.

The value of real-time data is growing as an increasing number of companies look to provide a comprehensive experience for their customers. Utilizing Kafka in key facets of your organization will yield greater customer satisfaction and promote a better understanding of user interactions. As data streaming is becoming more prevalent in a wide variety of industries, companies are seeking to modernize their tech stacks by employing the extensible, scalable infrastructure afforded by Kafka.

Over the past few months, we have successfully developed a containerized Kafka implementation at a major healthcare provider. In addition, we created producers to publish messages to the Kafka cluster and consumers to receive them on the other end. By capturing a plethora of data around customer activity, we created opportunities for the business to act upon real-time metrics in order to provide an improved customer experience.

In this talk, we will cover the user-related data sources we connected to Kafka, the reasons we chose them, and how the insights gained from each source can be leveraged in your business. You will walk out understanding how capturing a wide variety of customer activity data can create opportunities for the business to act on real-time metrics in order to provide an improved customer experience.

As a founder and CTO, one of the biggest surprises I encountered was how different my working relationship became with my former technical peers. A casual opinion about a technical matter, which used to be routine, became very problematic when I acquired status due to my title. And my schedule made it even harder for my team to work with me. I struggled to communicate with them, and they with me, about things that used to be fluid and easy. But in the process I also worked with some people who seemed to navigate and help ME navigate this new relationship much more easily.

In this session I’ll share my thoughts about effective ways to manage up, avoid micromanagement, advocate your expertise and ultimately grow your career while earning a place in your manager’s mind as an indispensable person who makes their job so much easier.

In this session we will be showing you how to apply design thinking to evaluate and use Kafka in a meaningful way within your organization. We’ll walk through five techniques we use within our own organization grounded in design thinking methodologies to help us tease out user problems and understand what to act on.

Use these best practices to work through question such as:
-How could event stream benefit my organization?
-Where does Kafka sit in my organization?
-What Kafka data is most useful to my consumers?
-How do I split responsibilities between your users and DevOps?
-How does Kafka tie into my existing tech stack?

We all know real-time data has a value. But how do you quantify that value in order to create a business case for becoming more data, or event driven?

The first half of this talk will explore the value of data across a variety of organizations, starting with the five most valuable companies in the world: Apple, Alphabet (Google), Microsoft, Amazon and Facebook (based on stock prices July 2017). We will go on to discuss other digital natives: Uber, Ebay, Netflix and LinkedIn, before exploring more traditional companies across retail, finance and automotive. Next, we’ll look at non-businesses such as governments and lobbyists. Whether organizations are using data to create new business products and services, improve user experiences, increase productivity, manage risk or influencing global power, we’ll see that fast and interconnected data, or “event streaming” is increasingly important.

After showing that data value can be quantified, the second half of this talk will explain the five steps to creating a business case.

Most businesses focus on:
-Making more money or conferring competitive advantage to make more money
-Increasing efficiency to save money and/or
-Mitigating risk to the business to protect money
-We’ll walk through examples of real business cases, discuss how business cases have evolved over the years and show the power of a sound business case. If you’re interested in big money and big business, as well as big data, this talk is for you.

NAV (Norwegian Work and Welfare Department) currently distributes more than one third of the national budget to citizens in Norway or abroad. We’re there to assist people through all phases of life within the domains of work, family, health, retirement and social security. Events happening throughout a person’s life determines which services we provide to them, how we provide them and when we provide them.

Today, each person has to apply for these services resulting in many tasks that are largely handled manually by various case workers in the organization. Their access to insight and useful information is limited and often hard to find, causing frustration to both our case workers and our users. By streaming a person’s life events through our Kafka pipelines, we can revolutionize the way our users experience our services and the way we work.

NAV and the government as a whole have access to vast amounts of data about our citizens, reported by health institutions, employers, various government agencies or the users themselves. Some data is distributed by large batches, while others are available on-demand through APIs. We’re changing these patterns into streams using Kafka, Streams API and Java microservices. We aim to distribute and act on events about birth, death, relationships, employment, income and business processes to vastly improve the user experience, provide real-time insight and reduce the need to apply for services we already know are needed.

This talk will touch on the following topics:
-How we move from data-on-demand to streams
-How streams of life events will free our case workers from mundane tasks
-How life and business events make valuable insight
-How we protect our users and comply with GDPR
-Why we chose Confluent Platform

Most organizations are faced with data sources from only one industry, be it telco, finance or retail. That’s hard enough. OVO is the payments, loyalty, and financial services app of the Lippo Group, a major conglomerate in Southeast Asia. OVO wanted to integrate data across different businesses into a single digital ecosystem to serve over 120 million consumers. Talk about a new level of complexity! OVO set out to consolidate customer data generated by their various subsidiaries into a centralized analytics platform to generate a 360-degree view of the customer. However, they found that their legacy databases and data lakes couldn’t analyze streaming data fast enough.

In this session, Irina Farooq, vice president of product at Kinetica, will share how OVO has leveraged Apache Kafka along with Kinetica’s GPU-accelerated engine to modernize their data stack and instantly ingest and analyze streaming data. This helps OVO to improve customer experience, deliver targeted marketing offers and increase operational efficiencies.

Learn how OVO moved to real-time streaming data analysis to power an innovative customer 360 initiative that stretches across their business units. OVO brings together customer data into a central analytic platform using multiple customer attributes including cross-channel behavior, demographic/behavioral/geographic data, brand sentiment, social media actions and purchasing patterns to enable personalized interactions throughout the customer journey. The technology behind this includes an analytic API that enables queries on huge volumes of streaming data in under seconds, compared to something that used to take minutes to hours. With Kafka and Kinetica, OVO can now analyze streaming data in real-time to gain insights into consumer trends and behavior, ultimately improving the customer experience, conversion rates, inventory life cycle management and future purchasing predictions.

You have data centers all over the world. You have set them up close to your customers and services. You have decentralized data handling with a network of data processing systems. The trouble is now how do you make sure your messages reach the appropriate data centers without getting into a complicated mess of dedicated topic-to-topic replication? What if your messaging bus was smart enough to figure out where to deliver each message? With Adobe’s Experience Cloud Pipeline, we have tackled this very issue.

Adobe’s Experience Cloud gives access to an integrated set of Adobe products and solutions that help Adobe’s customers to build campaigns, manage to advertise, gain profound insights and deliver personalized experiences all in near real time. To achieve this Adobe solutions must exchange data close to real time. Often data processed by the solutions need to travel across globally distributed data centers hosted on both public and private clouds.

Based on Apache Kafka, Adobe’s Experience Cloud Pipeline is a globally distributed, mission-critical messaging bus for asynchronous communication across Adobe solutions that process tens of billion messages each day. The Experience Cloud Pipeline also serves an entry point for the streaming data flowing from external systems into the Adobe Experience Cloud. Multi-data center deployment is commonplace to most large enterprises. Having a smart messaging bus that is capable of routing, filtering and enriching messages in near real time, in proximity to your solutions and customers is of paramount importance. Adobe’s Experience Cloud Pipeline demonstrates a successful large-scale deployment of such use case.

Breaking down monolithic applications into smaller manageable microservices can be a tough challenge. But the benefits are many. Faster changes, developer productivity, maintainability, scalability and high performance are a few of the motivators that make companies undertake this difficult journey.

At Intuit, we have our fair share of monolithic applications. One such application is Quickbooks Online, our accounting product for small businesses. In order to decompose the application, we needed to create new services, and reduce footprint of data in the monolith by moving it to new services in a phased manner. As more and more data and services keep moving out of the monolith, this data now distributed across multiple microservices needs to be synchronized in near real time to provide a seamless and fast experience to the customers of our product.

To achieve this, we are using Kafka as our eventing backbone that can aid us in keeping distributed data in sync, without compromising performance and user experience. Guaranteed publishing of financial events with no loss, high accuracy and performance is of utmost importance as majority of Intuit products deal with highly sensitive, financial data. Strong ordering guarantees is another important criteria that Kafka can provide with low latency and high throughput. Use cases for data and streaming analytics, insights, personalization, machine-learning-based predictions, can all be unlocked by adopting Kafka as our distributed streaming platform.

This talk will take you through Intuit’s journey of building a distributed, asynchronous system using Kafka. Specifically about the choices made, challenges faced, the adaptations clients had to make and how we see Kafka powering our future!

Commonwealth Bank of Australia (CBA) is Australia’s largest bank with over 15m customers, 50,000 employees and over USD700 billion in assets. We started the journey two years ago to transform our existing enterprise architecture into an “event driven” architecture. Since then, Kafka has become a mission critical platform in the Bank and it is the core component in our “event driven” architecture strategy.

In this talk, we will walk you through the journey of how we stood up the initial Kafka clusters, the challenges we encountered (both technical and organisational) and how we overcame those challenges.

We will also deep dive into one of the use cases for Kafka (with Kafka Streams and Connectors) in our new real time payment system that was introduced in Australia early this year. We will discuss why we think Kafka was the perfect solution for this use case, and the lessons learned.

Key Takeaways:
-Lessons learned from our experiences (that we think other companies could be able to benefit from)
-Our use cases for Kafka with a particular focus on the new real time payment systems (NPP) initiative in Australia