Confluent Cloud Q1 Launch: Build a Secure Shared Services Data Streaming Platform | Learn more

How to mutate your immutable log

Is your data pipeline under development and you simply want to iterate quickly? Immutability is one of the key and desirable features of Kafka. However, when mistakes happen and you are paged at night you sometimes wish there was an 鈥渆asy button鈥 to change the log.

This session first dives into some of the errors we have seen that caused outages for considerable durations of time. Recovery from the errors required late night code changes on consumers or simply waiting things out.Have you ever had your upstream producers write poisoned data that breaks your downstream consumers? Did Personal Identifiable Information (PII) land in a Kafka topic that wasn鈥檛 supposed to have it?

The next part of the session proposes a topic versioning scheme that allows us to recover from the examples that we mention. It segues into what it would take to build a control plane to manage and lifecycle these versioned topics. We鈥檒l cover the benefits and pitfalls of our proposed solution.


Andrey Falko