[Demo+Webinar] New Product Updates to Make Serverless Flink a Developer’s Best Friend | Watch Now
Apache Kafka® is the standard for data streaming, and has one of the largest open source communities in the world. With its popularity, there are a growing number of offerings compatible with Kafka’s API. This page compares Apache Kafka with Redpanda, as well as two different cloud Kafka services - Confluent Cloud and Redpanda Cloud.
Apache Kafka® is an open source data streaming technology capable of handling trillions of events per day. It’s based on the abstraction of a distributed commit log, with functionality comprising pub/sub, permanent storage, and the processing of event streams. Created at LinkedIn and open sourced in 2011, Kafka has since been adopted by over 100k organizations worldwide and has a vast developer community and ecosystem.
Redpanda is a C++ clone of Apache Kafka. It is not open source, but a community edition is source available under the BSL license, with enterprise features available with commercial subscription. Redpanda provides a Kafka-compatible API on top of its own implementation of the distributed commit log. Originally founded in 2019, Redpanda is privately owned by a company of the same name.
Apache Kafka and Redpanda both implement their own versions of a highly-available distributed commit log. While there are commonalities between the two, there remain several key design differences that affect usage, performance, and adoption. What are the key differences between Kafka and Redpanda?
License |
Open source Under the Apache License governed by the Apache Software Foundation. |
Source available Under the Business Source License (BSL) with proprietary paid features available under an enterprise license. |
---|---|---|
Contribution model and commercial backing |
Open Actively managed and maintained by 1,000+ full-time contributors at over a dozen companies and commercially backed by a broad coalition of vendors. |
Restricted Solely developed and maintained by Redpanda, with restrictive commercial support from other vendors due to BSL license agreement. |
Source Language | Java | C++ |
ZooKeeper Dependency |
No dependency ZooKeeper was removed by KRaft since version 3.3+ |
No dependency ZooKeeper-free and uses the Raft consensus algorithm. |
Storage Pattern and Performance Impact | Consistent performance across most real-world workloads Kafka has a purpose-built log and replication layer optimized for sequential IO, which allows it to deliver high throughput and low latency across a broad set of hardware and workloads. |
Performance optimized for selective workloads Redpanda can demonstrate low latency and high throughput on simple workloads. However, because it’s optimized for random IO, its performance can significantly degrade over time. Several common production configurations, such as high producer count, over 30% disk utilization, enabling message keys, enabling TLS, or running for more than 24 hours can cause severe reductions in performance. |
Broker Framework |
Purpose-built immutable log Uses its own purpose-built framework. Data is written in large blocks as high throughput sequential IO, allowing for high performance on drives with even very low IOPS. |
Based on Seastar Uses the Seastar framework, popularized by the Scylla Database, to implement its immutable log. Writes data in small 16kB chunks by default, requiring very high IOPS SSDs. |
Tiered Storage |
Available - early access Released in Kafka 3.6 as early access through KIP-405 |
Requires Enterprise License Redpanda’s tiered storage requires the purchase of an enterprise license. |
Replication Protocol |
Kafka replication (ISR) Replication is synchronous but data is written to disk asynchronously by design. Brokers don’t need to fsync for correctness and have in-built data recovery and repair. |
Raft Both replication and writing to disk are synchronous. |
Breadth of adoption |
Vast developer ecosystem and community Apache Kafka is used by 100,000+ organizations, including 80% of F100 companies, including Goldman Sachs, Netflix and Uber Community stats: 200+ global meetups with 41,000+ attendees, 32,000+ stack overflow questions, and 44k Slack community members in the Confluent Community Slack. |
Limited adoption and community Redpanda is used by thousands of organizations (undisclosed) Community stats: Less than 100 stack overflow questions, and 5k Slack community members. |
Built by the original co-creators of Kafka in 2018, Confluent Cloud is a cloud-native data streaming platform, powered by Kora engine. Confluent has re-architected Kafka to create a fully-managed service with 10x elasticity, resiliency and performance. Confluent Cloud offers a complete set of enterprise features to relieve operational burdens and boost developer productivity.
Redpanda launched their fully-managed Kafka service in late 2022. Redpanda Cloud is based on a C++ clone of Kafka and includes all the features from Redpanda’s Enterprise license. The service is offered through three deployment models: single-tenant Dedicated clusters, Bring-Your-Own-Cloud (BYOC) clusters, and multi-tenant Serverless clusters (not production-ready yet).
Infrastructure responsibilities |
Fully-managed SaaS Automated and fully-managed Kafka clusters with zero provisioning, scaling, or operational burdens. |
Shared infrastructure responsibilities No self-serve options. Infrastructure operations and maintenance considerations shared between customer, Redpanda, and cloud provider. |
---|---|---|
Trust and security |
Enterprise grade security and governance Enterprise grade security (RBAC, authentication support, encryption, etc.) and data governance features; Built-in compliance with all major industry standards (SOC, ISO, HIPAA, etc.). |
Shared security model Security shared between customer, Redpanda and cloud provider; RBAC and authentication support, compliance with SOC 2 Type 1/2. |
Availability |
Comprehensive 99.99% SLA 99.99% SLA across 30K+ clusters and 4500+ customers globally, covering infrastructure, software, bug fixes, upgrades, patching, and more. |
99.5% to 99.95% SLA Limited number of publicly referenced customers; Fully managed not available for BYOC offering. |
Developer Productivity |
Complete set of tools 80+ fully-managed Kafka connectors, SQL-based stream processing for both simple and complex data transforms, and low-code visual pipeline builder. |
Limited tools 10+ fully-managed connectors, in-broker processing for basic data transform (not available in Cloud), and no pipeline building solution. |
Pricing |
Transparent pricing, volume-based discounts Publicly available, pay-as-you-go pricing; negotiated volume-based discounts for Kafka workloads with cloud providers. |
Pricing not published Pricing not publicly available and only include licensing costs; Pay cloud provider on a separate hidden bill for compute, storage, and networking. |
Developers, architects and operators need a complete, cloud-native data streaming platform that prioritizes ease of use and alleviates operational burdens. That means delivering a Kafka service with fully-managed infrastructure, built-in enterprise-grade tooling, and flexible deployment options.
Category | ||
---|---|---|
Serverless Offering | ||
Scalability | ||
Automatic Cluster Rebalancing | ||
Infinite Data Retention with Tiered Storage | ||
Resiliency Guarantees | ||
Upgrades and Patches |
Category | Feature | ||
---|---|---|---|
Productivity | Infra-as-code Deployment | ||
Productivity | Stream Processing | ||
Productivity | No-Code / Low-Code Data Pipelines | ||
Productivity | Kafka Connectors | ||
Governance | Schema Registry | ||
Governance | Stream quality enforcement | ||
Governance | Metadata Catalog | ||
Governance | Data Lineage | ||
Governance | Data product management | ||
Monitoring and Alerting | Monitoring | ||
Monitoring and Alerting | Notifications |
Category | ||
---|---|---|
Core Cloud Service Providers | ||
Cluster Synchronization |
Category | Feature | ||
---|---|---|---|
Security | Authorization | ||
Security | Authentication | Supports OAuth 2.0, OIDC, API Keys, and plain username/password. Integrates with existing SAML-based single sign-on (SSO) and identity providers (IdP) such as Google, GitHub, Okta, OneLogin, and Azure Active Directory (AD). |
Supports plain username/password, OIDC/SSO and mTLS. |
Security | Audit Log | ||
Security | Private Connection | ||
Security | VPC Peering | ||
Security | Data Encryption | Encrypt data at rest using your own Kafka cluster management tools. No BYOK option. |
|
Trust | Compliance | ||
Trust | Privacy | ||
Enterprise Support | Expertise | ||
Enterprise Support | Product Warranty |
Start testing and deploying fully-managed Kafka clusters in Confluent Cloud within minutes with a 30-day trial.