Confluent
Log Compaction | Highlights in the Apache Kafka and Stream Processing Community | December 2016
Apache Kafka

Log Compaction | Highlights in the Apache Kafka and Stream Processing Community | December 2016

Apurva Mehta

This month saw the proposal of a few KIPs which will have a big impact on Apache Kafka’s semantics as well as Kafka’s operability.

  • KIP-95 : Incremental Batch Processing for Kafka Streams brings the first hint of batch processing to Kafka Streams and brings us a step closer to unifying batch and stream processing around the log. It enables streams tasks to ‘auto-stop’ when they reach the end of the log such that periodic invocations would process batches of messages.
  • KIP-97 : Improved Kafka Client RPC Compatibility Policy will enable newer versions Java Kafka clients to talk with older broker versions. Previously old clients could talk with newer brokers, but not vice-versa. Amongst other things, this KIP will make upgrading Kafka clusters easier since client and server upgrades can be decoupled.
  • KIP-98 : Exactly Once Delivery and Transactional Messaging has been proposed and is in discussion. This KIP is a major addition and will bring idempotent message production as well as transactional semantics to Kafka — features which have been long requested and heavily discussed over the years. The KIP has an associated detailed design document, so make sure you have plenty of coffee on tap!

Lots of interesting happenings occurred in the wider streaming community as well, notably:

And last but not least, Confluent released version 3.1 of Confluent Enterprise.

Happy Holidays!

Subscribe to the Confluent Blog

Subscribe
Email *

More Articles Like This

log compaction
Gwen Shapira

Log Compaction – Highlights in the Apache Kafka™ and Stream Processing Community – March 2017

Gwen Shapira . .

Big news this month! First and foremost, Confluent Platform 3.2.0 with Apache KafkaTM 0.10.2.0 was released! Read about the new features, check out all 200 bug fixes and performance improvements ...

log compaction
Gwen Shapira

Log Compaction: Highlights in the Apache Kafka and Stream Processing Community – January 2017

Gwen Shapira . .

Happy 2017! Wishing you a wonderful year full of fast and scalable data streams. Many things have happened since we last shared the state of Apache Kafka™ and the streams ...

log compaction
Gwen Shapira

Log Compaction | Highlights in the Apache Kafka and Stream Processing Community | November 2016

Gwen Shapira . .

Last month the Apache Kafka community released version 0.10.1.0, the announcement blog contains a good description of new features and major improvements. In other exciting news, the PMC for Apache ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Try Confluent Platform

Download Now