Kora Engine, Data Quality Rules und mehr in unserem Q2 2023 Launch | FĂŒr die Demo registrieren

Fan-in Flames: Scaling Kafka to Millions of Producers

« Current 2022

At supermassive scale, a perennial problem with Kafka is ""high fan-in"" -- a large number of producers sending records to a small number of brokers. Even a relatively modest amount of data can overwhelm a broker when there are hundreds of thousands of concurrent producer requests.

This talk discusses a few real-world applications where high fan-in becomes a problem, and presents a few strategies for dealing with it. These include: fronting Kafka with an ingestion layer; separating brokers into read-only and write-only subsets; implementing specialized partitioning strategies; and scaling across clusters with ""smart clients"".

Related Links

How Confluent Completes Apache Kafka eBook

Leverage a cloud-native service 10x better than Apache Kafka

Confluent Developer Center

Spend less on Kafka with Confluent, come see how