Register for Demo | Confluent Terraform Provider, Independent Network Lifecycle Management and more within our Q3’22 launch!

Log Compaction | Highlights in the Apache Kafka and Stream Processing Community | November 2016

Last month the Apache Kafka community released version, the announcement blog contains a good description of new features and major improvements. In other exciting news, the PMC for Apache Kafka has invited Jiangjie (Becket) Qin to join as a committer and we are pleased to announce that he has accepted.

As usual, there are a large number of improvement proposals being discussed and voted on:

  • KIP-72: Limiting the amount of memory used by incoming requests is in discussion. Currently admins can configure the number of requests in the request queue, but arrival of very large requests can cause brokers to run out of memory. This proposal will allow admins to avoid that by explicitly configuring the amount of memory available for incoming requests and not accept additional requests from clients until memory is available.
  • KIP-81: Bound Fetch memory usage in the consumer – Another memory improvement suggestion, this time to prevent consumers from running out of memory while fetching data. It is interesting to see how, even though KIP-72 and KIP-81 solve different problems in different components – we converged on very similar solutions, which is to limit the bytes the consumer / broker read from socket when it exhausts available memory.
  • KIP-84: Support SASL/SCRAM mechanisms is in discussion. This KIP adds another authentication mechanism – SCRAM supports convenient username and password authentication, but it addresses many of the security concerns involved with SASL/PLAIN authentication.
  • KIP-85: Dynamic JAAS configuration for Kafka clients was approved. Kafka clients currently rely on configuration file to configure SASL security. With this improvement, developers will be able to pass JAAS configuration as a normal configuration parameter – super useful for container deployments where clients don’t always have a persistent file system to use.
  • KIP-87: Add Compaction Tombstone Flag is in discussion. Right now, the way to remove keys from a compacted topic is by sending a record with the key and null value. This prevents both legitimate usage of nulls and is challenging in cases where the value has “magic bytes” or versions. This proposal will replace the null value with a message flag that will indicate a delete operation.

And few recommended presentations and blog posts relevant to Apache Kafka and streaming platforms:

Did you like this blog post? Share it now

Subscribe to the Confluent blog