OSS Kafka couldn’t save them. See how data streaming came to the rescue! | Watch now
Kafka Connect
Introducing Confluent Hub
Today we’re delighted to announce the launch of Confluent Hub. Confluent Hub is a place for the Apache Kafka and Confluent Platform community to come together and share the components […]
Analytics on Bare Metal: Xenon and Kafka Connect
The following post is a guest blog from Tushar Sudhakar Jee, Software Engineer, Levyx responsible for Kafka infrastructure. You may find this post also on Levyx’s blog. Abstract As part […]
We ❤️ syslogs: Real-time syslog Processing with Apache Kafka and KSQL – Part 1: Filtering
Syslog is one of those ubiquitous standards on which much of modern computing runs. Built into operating systems such as Linux, it’s also commonplace in networking and IoT devices like […]
No More Silos: How to Integrate Your Databases with Apache Kafka and CDC
One of the most frequent questions and topics that I see come up on community resources such as StackOverflow, the Confluent Platform mailing list, and the Confluent Community Slack group, […]
KSQL in Action: Real-Time Streaming ETL from Oracle Transactional Data
In this post I’m going to show what streaming ETL looks like in practice. We’re replacing batch extracts with event streams, and batch transformation with in-flight transformation. But first, a […]
Handling GDPR with Apache Kafka: How does a log forget?
If you follow the press around Apache Kafka you’ll probably know it’s pretty good at tracking and retaining messages, but sometimes removing messages is important too. GDPR is a good […]
Apache Kafka Goes 1.0
It has been seven years since we first set out to create the distributed streaming platform we know now as Apache Kafka®. Born initially as a highly scalable messaging system, […]
Getting Started Analyzing Twitter Data in Apache Kafka through KSQL
KSQL is the streaming SQL engine for Apache Kafka®. It lets you do sophisticated stream processing on Kafka topics, easily, using a simple and interactive SQL interface. In this short […]
The Simplest Useful Kafka Connect Data Pipeline in the World…or Thereabouts – Part 3
We saw in the earlier articles (part 1, part 2) in this series how to use the Kafka Connect API to build out a very simple, but powerful and scalable, streaming […]
The Simplest Useful Kafka Connect Data Pipeline in the World…or Thereabouts – Part 2
In the previous article in this blog series I showed how easy it is to stream data out of a database into Apache Kafka®, using the Kafka Connect API. I […]
The Simplest Useful Kafka Connect Data Pipeline in the World…or Thereabouts – Part 1
This short series of articles is going to show you how to stream data from a database (MySQL) into Apache Kafka® and from Kafka into both a text file and Elasticsearch—all […]
Exactly-Once Semantics Are Possible: Here’s How Kafka Does It
I’m thrilled that we have hit an exciting milestone the Apache Kafka® community has long been waiting for: we have introduced exactly-once semantics in Kafka in the 0.11 release and […]
Building a Real-Time Streaming ETL Pipeline in 20 Minutes
There has been a lot of talk recently that traditional ETL is dead. In the traditional ETL paradigm, data warehouses were king, ETL jobs were batch-driven, everything talked to everything […]
Creating a Data Pipeline with the Kafka Connect API – from Architecture to Operations
Pandora began adoption of Apache Kafka® in 2016 to orient its infrastructure around real-time stream processing analytics. As a data-driven company, we have a several thousand node Hadoop clusters with hundreds of Hive tables critical to Pandora’s operational and reporting success...