[Webinar] Mastering Kafka Security Across Hybrid Environments → Register Now


Dataflows for Machine Learning Operations

« Kafka Summit London 2023

Machine learning (ML) models are deployed for production use cases with ever increasing pace, driving the growing need for machine learning operations (MLOps) for the deployment, monitoring, and explainability of ML models at scale. With the rise of the data-centric AI movement, businesses are seeking solutions that will provide them with highly discoverable and available data for monitoring, governance, and compliance.

In this talk, we identify dataflow architectural principles to address these demands and discuss their application in an open-source ecosystem. We show how to create a decentralized dataflow engine underpinned by Kafka and the Kafka Streams client library, and how this can be leveraged for building flexible data processing pipelines on-the-fly.

We explore the challenges faced in creating such a dataflow engine and reflect on our journey with the Kafka ecosystem. We consider managing dynamically-created Kafka Streams topologies, multiplexing hundreds or even thousands of these topologies onto individual JVM instances, and the integration between Kafka and Kotlin, amongst other things.

Related Links

How Confluent Completes Apache Kafka eBook

Leverage a cloud-native service 10x better than Apache Kafka

Confluent Developer Center

Spend less on Kafka with Confluent, come see how