Log Compaction | Highlights in the Kafka and Stream Processing Community | August 2015

Welcome to the first edition of Log Compaction, a monthly digest of highlights in the Apache Kafka and stream processing community. Today’s edition are the highlights from July and early August 2015. Got a newsworthy item? Let us know.

• Breaking news! Apache Kafka no longer supports Java 6 and Scala 2.9. This will allow us to use some of the language and API improvements that were introduced in later Java and Scala versions – Diamond operators, NIO improvements, Scala Futures, Scala string interpolation and more.
• Compression in Kafka was just made 34% faster. Contributor Yasuhiro Matsuda explains how compression works in Kafka and dives into various optimization techniques he used to improve performance.
• Apache Kafka community is now accepting contributions via Github pull requests, transitioning from the old “Attach patches to JIRA” process. We documented the new code contribution process here: https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Code+Changes
  We also welcome non-code contribution such as bug reports, feature requests and documentation improvements. We are planning to move our documentation to Github too so it will be easier to contribute documentation changes.
• Apache Kafka community is deep in discussion of what would a stream processing library that is tightly integrated with Apache Kafka look like: https://cwiki.apache.org/confluence/display/KAFKA/KIP-28+-+Add+a+processor+client We are still prototyping and gathering inputs and requirements – so join the discussion!
• Martin Kleppmann explained how stream processing with Kafka is a bit like working with Unix pipes.
• The Apache Flink community published a blog post that both explains some of the magic behind Flink and compares performance between Flink and other popular stream processing frameworks.
• Apache Kafka community is reviewing the first patch for our ingest/egress framework – CopyCat. You can take a look, try it out and comment.