Project Metamorphosis: Unveiling the next-gen event streaming platformLearn More

Log Compaction | Highlights in the Apache Kafka and Stream Processing Community | August 2016

It is August already, and this marks exactly one year of monthly “Log Compaction” blog posts – summarizing news from the very active Apache Kafka and stream processing community. Hope you enjoy them and as usual, let us know if you have news to share.

  • The Apache Kafka community is preparing to release a bugfix for version 0.10.0. The new release will be 0.10.0.1 and we are currently voting on a release candidate – hopefully we won’t find critical issues and the release will be available soon.
  • The on-going work on KIP-4 has seen significant progress. This work will allow all client libraries to manage topics without depending on core Kafka or Zookeeper:
  • API to create new topics through the wire protocol was voted in and committed
  • API to delete topics was voted in and a patch is currently under review
  • API to manage ACLs is currently under discussion
  • KIP-67 – adding queryable state to Kafka Streams was voted in and committed. This new feature will allow other applications to directly query the latest processing results of your Kafka Streams application (i.e. its current state).  This means that, for many use cases, you no longer need to operate and interface with external systems or databases to share data between applications. The result is a simplified, more app-centric architecture.
  • Michael Noll published two more blogs on Kafka Streams: Secure stream processing and Elastic Scaling in Kafka Streams.
  • Alex Loddengaard published his best practices for running Apache Kafka in AWS. There have been tons of questions in the community about this topic as cloud deployments are becoming more and more popular – so we shared our answers in this blog post.
  • Spark 2.0 was released last week with many improvements to Spark Streaming. This blog post gives an overview of what’s new in Spark Streamng.
  • Back in April, when we ran the Kafka Connect and Streams hackathon, one of my favorite projects was by SVDS. They streamed data from a Bluetooth brain monitoring device to Kafka, used Kafka Connect to stream data out to OpenTSDB, and then used Grafana to visualize the brain activity! How cool is that? SVDS blogged all the fun details of their brain monitoring project for your inspiration.
  • Recommendation powerhouse Yelp blogged about their real-time data pipeline architecture – and it is gorgeous. We recommend checking it out as a reference for anyone tasked with building similar infrastructure.
  • Apache Kafka training is offered by Confluent and our partners. New classes have just been published, including online-based training. www.confluent.io/training.

Did you like this blog post? Share it now

Subscribe to the Confluent blog

More Articles Like This

Kafka Streams Interactive Queries Go Prime Time

What is stopping you from using Kafka Streams as your data layer for building applications? After all, it comes with fast, embedded RocksDB storage, takes care of redundancy for you, […]

From Eager to Smarter in Apache Kafka Consumer Rebalances

Everyone wants their infrastructure to be highly available, and ksqlDB is no different. But crucial properties like high availability don’t come without a thoughtful, rigorous design. We thought hard about […]

Walmart’s Real-Time Inventory System Powered by Apache Kafka

Consumer shopping patterns have changed drastically in the last few years. Shopping in a physical store is no longer the only way. Retail shopping experiences have evolved to include multiple […]

Sign Up Now

Start your 3-month trial. Get up to $200 off on each of your first 3 Confluent Cloud monthly bills

New signups only.

By clicking “sign up” above you understand we will process your personal information in accordance with our Privacy Policy.

By clicking "sign up" above you agree to the Terms of Service and to receive occasional marketing emails from Confluent. You also understand that we will process your personal information in accordance with our Privacy Policy.

Free Forever on a Single Kafka Broker
i

The software will allow unlimited-time usage of commercial features on a single Kafka broker. Upon adding a second broker, a 30-day timer will automatically start on commercial features, which cannot be reset by moving back to one broker.

Select Deployment Type
Manual Deployment
  • tar
  • zip
  • deb
  • rpm
  • docker
or
Auto Deployment
  • kubernetes
  • ansible

By clicking "download free" above you understand we will process your personal information in accordance with our Privacy Policy.

By clicking "download free" above, you agree to the Confluent License Agreement and to receive occasional marketing emails from Confluent. You also agree that your personal data will be processed in accordance with our Privacy Policy.

This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising, and analytics partners.