Confluent
Log Compaction – Highlights in the Apache Kafka and Stream Processing Community – February 2017
Log Compaction

Log Compaction – Highlights in the Apache Kafka and Stream Processing Community – February 2017

Gwen Shapira

As always, we bring you news, updates and recommended content from the hectic world of Apache KafkaTM and stream processing.

Sometimes it seems that in Apache Kafka every improvement is preceded by an involved KIP process. This month we’ve merged a great patch that improved the 99% latency of Kafka without requiring user visible changes: https://issues.apache.org/jira/browse/KAFKA-4614. Not only does it make a fast system even faster, the JIRA itself is worthy of study. I wish all JIRAs included this level of research.

Some important improvements do require KIPs. Here is what we’ve seen in active discussions this month:

  • KIP-112: Handle disk failure for JBOD and its close relative KIP-113: Support replicas movement between log directories. Both these KIPs improve Kafka’s behavior in the common case where the broker’s data is written to a number of directly mounted disks on the broker server (rather than using RAID). With these improvements, Kafka will be able to survive failure of a single disk without taking down an entire broker, and it will allow admins to control the placement of replicas on disk – useful in cases where disks or replicas have uneven sizes.
  • KIP-117: Add a public AdminClient API for Kafka admin operations: This lets developers create, modify and delete topics and ACLs without using internal APIs which are subject to incompatible changes and without requiring ZooKeeper connection from the applications.
  • KIP-98: The famous KIP that adds transactional semantics and exactly-once to Kafka is now under voting. This means that the Wiki now contains all the public changes. If you haven’t read it yet, now is a good time.
  • KIP-118 suggests we remove support for Java 7 in the next major release (0.11). We don’t know yet when 0.11 will get released, but we know it will be later than June.
  • KIP-110 suggests adding support for a new compression codec: ZStandard Compression. The new compression, written by Facebook, looks very promising.
  • KIP-109 suggests marking the old consumers as deprecated, as a hint for developers that they should start migrating to the new clients. As the KIP states, the old consumers are missing important features like security that were only added in the new clients.

Notable Blogs and Presentations:

Subscribe to the Confluent Blog

Subscribe
Email *

Leave a Reply

Your email address will not be published. Required fields are marked *

Try Confluent Platform

Download Now