[Demo+Webinar] New Product Updates to Make Serverless Flink a Developer’s Best Friend | Watch Now
Log Compaction – Highlights in the Apache Kafka® and Stream Processing Community – June 2017
We are very excited for the GA for Kafka release 0.11.0.0 which is just days away. This release is bringing many new features as described in the previous Log Compaction blog post.
The most notable new feature is Exactly Once Semantics (EOS). Kafka’s EOS capabilities provide more stringent idempotent producer semantics with exactly once, in-order delivery per partition, and stronger transactional guarantees with atomic writes across multiple partitions. Together, these strong semantics make writing applications easier and expand Kafka’s addressable use cases. You can learn more about EOS in the online talk on June 29, 2017.
The recent 2017 Apache Kafka Report showed that 37% of Kafka adopters are now using the Kafka Connect API. As part of the streaming ETL paradigm, Kafka Connectors make it very simple to load data from other systems into Kafka and to extract Kafka data into other systems. The full list of Kafka Connectors has recently been updated with some new ones contributed by the community including:
- AWS S3 sink connector
- Redis source connector
- RabbitMQ source connector
- GitHub source connector
- Amazon Kinesis source connector
- MongoDB sink connector
Video recordings and slide decks from the sessions at Kafka Summit NYC are now available. And in a few days, the Kafka Summit committee will announce the list of sessions for the upcoming Kafka Summit in San Francisco. If you would like to attend the conference which is happening on August 28, 2017, please register soon.
Notable blogs and presentations:
- Scrapping Your Yearlong Project: a CTO’s description of how the Kafka stream pipeline architecture compares to other approaches to data flow
- How Pandora Uses Kafka: Pandora transitioned its ad tracking system from a batch-oriented system into a real-time one, and adopted Kafka as a core component of its new stream processing system
- Scale-out for Complex Event Processing: horizontal scale-out architecture built around Kafka
- Real-Time Stream Processing for Internet of Things: stream processing enables quick processing of large amounts of data from multiple IOT sources, in real-time
- Configuring Kafka Connect to log REST HTTP messages to a separate file: helping developers work with log files in Kafka Connect
If you want to directly engage with the Kafka community, you can engage in a variety of ways: Google Group, Slack, Reddit, LinkedIn, Twitter, and please join us at a Kafka Meetup group in your area! Alternatively, please feel free to reach out to us at community@confluent.io
Did you like this blog post? Share it now
Subscribe to the Confluent blog
Handling the Producer Request: Kafka Producer and Consumer Internals, Part 2
In this post, the second in the Kafka Producer and Consumer Internals Series, we follow our brave hero—a well-formed produce request—which is on its way to be processed by the broker and have its data stored on the cluster.
How Producers Work: Kafka Producer and Consumer Internals, Part 1
The beauty of Kafka as a technology is that it can do a lot with little effort on your part. In effect, it’s a black box. But what if you need to see into the black box to debug something? This post shows what the producer does behind the scenes to help prepare your raw event data for the broker.