A Beginner's Guide to Realtime Data

Apache Kafka's an interesting beast. It solves many different data-processing problems by being part queue, part database, and part message bus. But it can be daunting to get started, and like many things, the only real way to understand Kafka is to build something with it. So let's do that!

In this live-coding session we'll take our first steps with Kafka, building a system that tracks the live status of metro trains. As we do so, we'll build up an understanding of some of Kafka's core ideas - topics, connectors, consumers and groups - to build up a mental model of how Kafka works, and where it works best.

(And for those of you who already know Kafka, you'll learn some modern JavaScript and see how quickly you can build a useful app.)

By the end you'll have a starter project, and the shape of Kafka in your mind so you can make faster progress building your own realtime applications.

We’ll explore the use of Kafka Connect to make sourcing London’s transport data trivial. We’ll use Kafka Streams to do live manipulations of that data into a form we want to present. And to get away from the usual charts and maps web tier, we’ll send our data out to a more exotic data source - a group of synthesizers that represent a well-running system by playing in harmony (and represent a broken one with something more discordant!)

You’ll see - and hear - how the whole thing fits together and get a better sense of what a complete Kafka solution looks like. You could use the same recipe as the backbone for a more traditional business dashboard, or maybe the next time there’s a crisis at work, you could have it accompanied by its own live backing track…