Log Compaction | Highlights in the Kafka and Stream Processing Community | September 2015

September is the start of the fall conference season. Between Strata + Hadoop World New York and ApacheCon: Big Data Europe, there is plenty to keep us busy learning.

Conferences can provide too many choices, and finding the most relevant and most exciting sessions can be a challenge. So I reviewed the Strata conference agenda and picked some of my favorites related to Apache Kafka and stream processing.

Strata + Hadoop World, New York – September 29 – October 1, 2015

• Kick off Strata with a Kafka deep dive: “Many Streams Lead to Kafka” tutorial with Jesse Anderson and Ewen Cheslack-Postava (who will take over for Neha Narkhede).
• For a complete Kafka tutorial day spend the afternoon in “Process, store, and analyze like a boss with Team Apache: Kafka, Spark, and Cassandra” with Patrick McFadin.
• Learn how to build end-to-end fraud detection application using Kafka, Flume, SparkStreaming, and HBase at the “Hadoop Application Architectures: Fraud Detection” tutorial.
• Keep up to date on the latest innovation in the Apache Kafka community with “Copycat: Fault tolerant streaming data ingestion powered by Apache Kafka” with Neha Narkhede.
• Learn how to never lose a Kafka message again by attending “When it absolutely, positively has to be there – reliability guarantees in Kafka” with Gwen Shapira and Jeff Holoman
• You can’t miss the newest stream processing innovation “Twitter Heron: Stream Processing at Scale”.
• Everyone’s favorite speaker, Martin Kleppmann will present “Data liberation and data integration with Apache Kafka”
• Are you a business leader who is wondering “what’s the deal with Kafka, Spark, Docker, Cloud, and where is Hadoop heading?” Then “The business case for Spark, Kafka, and friends is for you” with Ed Dumbill.
• Can you use Kafka for Analytics? Learn how at “Build real-time analytics stack with Kafka, Samza and Druid” with Fangjin Yang and Gian Merlino.
• Learn about the new features in SparkStreaming, including the exactly-one Kafka integration, from Tathagata Das in “What’s new in SparkStreaming”.

Apache Kafka NYC Meetup at ADP Innovation Lab

In addition to all of the Strata sessions, there is the NYC Kafka meetup. You will not want miss Jay Kreps‘  talk about Stream Processing – trust me on this one. Gwen Shapira will present “When Bad Things Happen to Good Kafka Clusters” – war stories from using Kafka in production and how to avoid the same mistakes.

Strata + Hadoop World Expo Hall – Confluent Booth #929

Drop by to meet the Confluent team and chat about Apache Kafka, stream processing, and other geeky topics. Meet Jay Kreps and pick up a free signed copy of his book, I Heart Logs, on Wednesday, September 30 at 12:45pm. And for those of you that only care about Kafka swag, the booth will be stocked with stickers and t-shirts.

ApacheCon: Big Data Europe, September 28-30, 2015

Going at at the same time as Strata NYC, ApacheCon: Big Data Europe is taking place in Budapest, Hungary, from Sep 28-30.  This is a great opportunity to catch up with the Big Data and Apache Kafka open source communities, particularly for European users.  If you are going to Apache: Big Data, here are our recommendations:

• “Being ready for Apache Kafka: Today’s Ecosystem and Future Roadmap”. Judging by all of the questions I get about the next release, I predict a standing-room-only for this session.
• “The best of Apache Kafka Architecture” which should describe how Kafka’s architecture makes it reliable, scalable and fast.
• “Apache Kafka for High Throughput Systems”. Jane Wyngaard ran Kafka on 10Gb/s network and shows the throughput she achieved and how.
• Jane Wyngaard also describes “Apache Kafka in a Production High Performance Computing (HPC) environment”.
• “Deploying SparkStreaming with Kafka: Gotchas and Performance Analysis”. It’s always good to learn from other people’s experiences.
• Another real world Kafka+SparkStreaming story is from Pearson: “Near Real Time Indexing Kafka Messages to Apache Blur Using SparkStreaming”

Looking forward to meeting you at the conferences!