Register for Demo | Confluent Terraform Provider, Independent Network Lifecycle Management and more within our Q3’22 launch!

Streampunk - The Difference Engine for Unlocking the Kafka Black Box

"Kafka is like a black box, you can't see what's inside". How many times do we think and hear this? We all love Kafka but despite the plethora of existing tools, none of them have been able to break the seal of the "Kafka black box" just yet.

With Streampunk, I have set out to change this. Streampunk is a polyglot programmable shell for Kafka. It works quite like kcat or the Apache Kafka commandline tools, but unlike them, it is programmable using your favourite programming language instead of having to resort to the typical eccentric mix of Unix-shell scripting capabilities and tools. It is also a lot faster because Streampunk commands do not know the concept of startup time.

This lightning talk starts with a demo of how you can conveniently fulfill basic tasks such as listing and consuming topics using Streampunk in the Python interpreter. After that, I'll lead you through real-life examples of increasingly difficult challenges, such as copying a portion of a set of topics from one cluster to another, including a single message transform, and I'll show you how easily and elegantly these challenges can be solved with Streampunk.

Presenter

Ralph M. Debusmann

Ralph is part technologist, part solution architect, part software engineer and part AI researcher working as the CTO of Forecasty.ai. He has received his PhD in computer science focusing on Natural Language Processing and Artificial Intelligence in 2006 and has spent 14 years at SAP and Bosch before joining Forecasty in 2021.