Register for Demo | Confluent Terraform Provider, Independent Network Lifecycle Management and more within our Q3’22 launch!

CI/CD with an Idempotent Kafka Producer & Consumer

Idempotence is a mathematical requirement of particular operations where the operation can be applied multiple times without changing the result beyond the initial application. The main driver behind the idempotency requirement is often to handle duplicated messages. As developers and architects, we need to pay close attention to how we deal with our production data during new deployments to ensure we are not losing any data, duplicating messages, or introducing malformed data into our system. Furthermore, we need to figure out how to automate the process and add testing guarantees to prevent any potential human error.

In this session, you will learn about the idempotent Kafka Producer & Consumer architecture and how to automate the CI/CD process with open-source tools.

Presenter

Eden Ohana

Eden Ohana is a Software Engineer at Treeverse, a company behind lakeFS, an open-source platform that offers a Git-like interface for object-storage data lakes. She is passionate about open source and giving back to the developer communities. Her experience includes designing and implementing efficient algorithms and developing distributed systems. When Eden is not coding, she's traveling the world and enjoying cultural food.