Does your organization struggle with updating of its Kafka Streams application? Releasing a new version of a Kafka Streams application can be challenging, especially if its state has to be preserved between releases. Consider these best-practices and architectural ideas to make this process smoother and improve your release process.
Having experienced accidental removal of change-log topics and needing to expand partitions, it is much easier to handle with some planning. With the proper planning, you can achieve easier application upgrades.
Key take-aways from the session include:
- How do minimize the rebuilding of the state-stores.
- How to change stream topologies without affecting the existing state stores.
- What you can do when you absolutely need to increase the number of partitions within your application.
- How to leveraging schemas for application releases.
- Measures to prevent data corruption, especially if Kafka is not only your system of record but also your source of truth.
- Techniques to support rolling back an application.
- The advantages of splitting apart a Kafka Streams application into multiple applications.
Neil Buesing has presented at Kafka Summit London 2019 and Kafka Summit San Francisco 2019. He was accepted to Kafka Summit London 2020 and leads up the real-time data practice at Object Partners, Inc.