Unlock the full power of your event streams with Stream Governance | Learn More

πŸš‚ On Track with Apache Kafka: Building a Streaming ETL Solution with Rail Data

Do you want to know what streaming ETL actually looks like in practice? Or what you can REALLY do with Apache Kafka once you get goingβ€”using config & SQL alone?

This project integrates live data from the UK rail network via ActiveMQ along with data from other sources to build a fully-functioning platform. It includes analytics through Elasticsearch and exploring graph relationships in Neo4j, as well as real-time alerts delivered through Telegram.

This talk will show how I built the system, and include live demos and code samples of the salient integration points in ksqlDB and Kafka Connect.

The data may be domain-specific but the challenges of handling batch and stream data to drive both applications and analytics are encountered by many. This talk will give people lots of concrete examples of patterns and techniques for integration and stream processing.

Chinese Japanese Korean

Presenter

Robin Moffatt

Robin is a Developer Advocate at Confluent, the company founded by the original creators of Apache Kafka, as well as an Oracle Groundbreaker Ambassador. His career has always involved data, from the old worlds of COBOL and DB2, through the worlds of Oracle and Hadoop, and into the current world with Kafka. His particular interests are analytics, systems architecture, performance testing and optimization. He blogs at http://cnfl.io/rmoff and http://rmoff.net/ and can be found tweeting grumpy geek thoughts as @rmoff. Outside of work he enjoys drinking good beer and eating fried breakfasts, although generally not at the same time.