Log Compaction | Highlights in the Apache Kafka and Stream Processing Community | November 2016
Log Compaction

Log Compaction | Highlights in the Apache Kafka and Stream Processing Community | November 2016

Gwen Shapira

Last month the Apache Kafka community released version, the announcement blog contains a good description of new features and major improvements. In other exciting news, the PMC for Apache Kafka has invited Jiangjie (Becket) Qin to join as a committer and we are pleased to announce that he has accepted.

As usual, there are a large number of improvement proposals being discussed and voted on:

  • KIP-72: Limiting the amount of memory used by incoming requests is in discussion. Currently admins can configure the number of requests in the request queue, but arrival of very large requests can cause brokers to run out of memory. This proposal will allow admins to avoid that by explicitly configuring the amount of memory available for incoming requests and not accept additional requests from clients until memory is available.
  • KIP-81: Bound Fetch memory usage in the consumer – Another memory improvement suggestion, this time to prevent consumers from running out of memory while fetching data. It is interesting to see how, even though KIP-72 and KIP-81 solve different problems in different components – we converged on very similar solutions, which is to limit the bytes the consumer / broker read from socket when it exhausts available memory.
  • KIP-84: Support SASL/SCRAM mechanisms is in discussion. This KIP adds another authentication mechanism – SCRAM supports convenient username and password authentication, but it addresses many of the security concerns involved with SASL/PLAIN authentication.
  • KIP-85: Dynamic JAAS configuration for Kafka clients was approved. Kafka clients currently rely on configuration file to configure SASL security. With this improvement, developers will be able to pass JAAS configuration as a normal configuration parameter – super useful for container deployments where clients don’t always have a persistent file system to use.
  • KIP-87: Add Compaction Tombstone Flag is in discussion. Right now, the way to remove keys from a compacted topic is by sending a record with the key and null value. This prevents both legitimate usage of nulls and is challenging in cases where the value has “magic bytes” or versions. This proposal will replace the null value with a message flag that will indicate a delete operation.

And few recommended presentations and blog posts relevant to Apache Kafka and streaming platforms:

Subscribe to the Confluent Blog

Email *

More Articles Like This

log compaction
Gwen Shapira

Log Compaction – Highlights in the Apache Kafka™ and Stream Processing Community – March 2017

Gwen Shapira . .

Big news this month! First and foremost, Confluent Platform 3.2.0 with Apache KafkaTM was released! Read about the new features, check out all 200 bug fixes and performance improvements ...

log compaction
Gwen Shapira

Log Compaction: Highlights in the Apache Kafka and Stream Processing Community – January 2017

Gwen Shapira . .

Happy 2017! Wishing you a wonderful year full of fast and scalable data streams. Many things have happened since we last shared the state of Apache Kafka™ and the streams ...

log compaction
Apurva Mehta

Log Compaction | Highlights in the Apache Kafka and Stream Processing Community | December 2016

Apurva Mehta . .

This month saw the proposal of a few KIPs which will have a big impact on Apache Kafka’s semantics as well as Kafka’s operability. KIP-95 : Incremental Batch Processing for ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Try Confluent Platform

Download Now