Kafka in the Cloud: Why it’s 10x better with Confluent | Find out more

 Redpanda vs Kafka vs Confluent

Apache Kafka® is the standard for data streaming, and has one of the largest open source communities in the world. With its popularity, there are a growing number of offerings compatible with Kafka’s API. This page compares Apache Kafka with Redpanda, as well as two different cloud Kafka services - Confluent Cloud and Redpanda Cloud.

Apache Kafka vs. Redpanda

What is Apache Kafka®?

Apache Kafka® is an open source data streaming technology capable of handling trillions of events per day. It’s based on the abstraction of a distributed commit log, with functionality comprising pub/sub, permanent storage, and the processing of event streams. Created at LinkedIn and open sourced in 2011, Kafka has since been adopted by over 100k organizations worldwide and has a vast developer community and ecosystem.

What is Redpanda?

Redpanda is a C++ clone of Apache Kafka. It is not open source, but a community edition is source available under the BSL license, with enterprise features available with commercial subscription. Redpanda provides a Kafka-compatible API on top of its own implementation of the distributed commit log. Originally founded in 2019, Redpanda is privately owned by a company of the same name.

Apache Kafka vs. Redpanda at a Glance

Apache Kafka and Redpanda both implement their own versions of a highly-available distributed commit log. While there are commonalities between the two, there remain several key design differences that affect usage, performance, and adoption. What are the key differences between Kafka and Redpanda?

 
License

Open source

Under the Apache License governed by the Apache Software Foundation.

Source available

Under the Business Source License (BSL) with proprietary paid features available under an enterprise license.

Contribution model and commercial backing

Open

Actively managed and maintained by 1,000+ full-time contributors at over a dozen companies and  commercially backed by a broad coalition of vendors.

Restricted

Solely developed and maintained by Redpanda, with restrictive commercial support from other vendors due to BSL license agreement.

Source Language Java C++
ZooKeeper Dependency

No dependency

ZooKeeper was removed by KRaft since version 3.3+

No dependency

ZooKeeper-free and uses the Raft consensus algorithm.

Storage Pattern and Performance Impact Consistent performance across most real-world workloads

Kafka has a purpose-built log and replication layer optimized for sequential IO, which allows it to deliver high throughput and low latency across a broad set of hardware and workloads.

Performance optimized for selective workloads

Redpanda can demonstrate low latency and high throughput on simple workloads. However, because it’s optimized for random IO, its performance can significantly degrade over time.

Several common production configurations, such as high producer count, over 30% disk utilization, enabling message keys, enabling TLS, or running for more than 24 hours can cause severe reductions in performance.

Broker Framework

Purpose-built immutable log

Uses its own purpose-built framework. Data is written in large blocks as high throughput sequential IO, allowing for high performance on drives with even very low IOPS.

Based on Seastar

Uses the Seastar framework, popularized by the Scylla Database, to implement its immutable log. Writes data in small 16kB chunks by default, requiring very high IOPS SSDs.

Tiered Storage

Available - early access

Released in Kafka 3.6 as early access through KIP-405

Requires Enterprise License

Redpanda’s tiered storage requires the purchase of an enterprise license.

Replication Protocol

Kafka replication (ISR)

Replication is synchronous but data is written to disk asynchronously by design. Brokers don’t need to fsync for correctness and have in-built data recovery and repair.

Raft

Both replication and writing to disk are synchronous.
Data must be written (fsynced) to disk synchronously, otherwise, it is possible to lose data during an election of a new leader.

Breadth of adoption

Vast developer ecosystem and community

Apache Kafka is used by 100,000+ organizations, including 80% of F100 companies, including Goldman Sachs, Netflix and Uber

Community stats: 200+ global meetups with 41,000+ attendees, 32,000+ stack overflow questions, and 41k Slack community members in the Confluent Community Slack.

Limited adoption and community

Redpanda is used by thousands of organizations (undisclosed)

Community stats: Less than 100 stack overflow questions, and 4.5k Slack community members.

Confluent Cloud vs. Redpanda Cloud

What is Confluent Cloud?

Built by the original co-creators of Kafka in 2018, Confluent Cloud is a cloud-native data streaming platform, powered by Kora engine. Confluent has re-architected Kafka to create a fully-managed service with 10x elasticity, resiliency and performance. Confluent Cloud offers a complete set of enterprise features to relieve operational burdens and boost developer productivity.

What is Redpanda Cloud?

Redpanda launched their fully-managed Kafka service in late 2022. Redpanda Cloud is based on a C++ clone of Kafka and includes all the features from Redpanda’s Enterprise license. The service is offered through two deployment models: single-tenant Dedicated clusters and Bring-Your-Own-Cloud (BYOC) clusters.

Confluent vs. Redpanda at a Glance

 
Infrastructure responsibilities

Fully-managed SaaS

Automated and fully-managed Kafka clusters with zero provisioning, scaling, or operational burdens.

Shared infrastructure responsibilities

Infrastructure operations and maintenance considerations shared between customer, Redpanda, and cloud provider.

Trust and security

Enterprise grade security and governance

Enterprise grade security (RBAC, authentication support, encryption, etc.) and data governance features; Built-in compliance with all major industry standards (SOC, ISO, HIPAA, etc.).

Shared security model

Security shared between customer, Redpanda and cloud provider; RBAC and authentication support, compliance with SOC 2 Type 1/2.

Availability

Comprehensive 99.99% SLA

99.99% SLA across 30K+ clusters and 4500+ customers globally, covering infrastructure, software, bug fixes, upgrades, patching, and more.

99.5% to 99.95% SLA

Limited number of publicly referenced customers; Fully managed not available for BYOC offering.

Developer Productivity

Complete set of tools

70+ fully-managed Kafka connectors, SQL-based stream processing for both simple and complex data transforms, and low-code visual pipeline builder.

Limited tools

10+ fully-managed connectors, in-broker processing for basic data transform (not available in Cloud), and no pipeline building solution.

Pricing

Transparent pricing, volume-based discounts

Publicly available, pay-as-you-go pricing; negotiated volume-based discounts for Kafka workloads with cloud providers.

Pricing not published

Pricing not publicly available; Pay your own negotiated cloud provider rates for compute, storage, and networking.

Deep-dive: Confluent Cloud vs. Redpanda Cloud BYOC

Developers, architects and operators need a complete, cloud-native data streaming platform that prioritizes ease of use and alleviates operational burdens. That means delivering a Kafka service with fully-managed infrastructure, built-in enterprise-grade tooling, and flexible deployment options.

Cloud-Native Capabilities and Performance
Category Confluent RedPanda
Serverless Offering Available Instantly provision fully-managed infrastructure without sizing, taking advantage of multi-tenancy to handle any irregular, cyclical, or spiky workloads. Pay per use and commit pricing available. Not available Serverless offering in roadmap; Currently must size workloads ahead of time and pay full price for cloud infrastructure regardless of use patterns.
Scalability Auto-scaling with high throughput limits Avoid increased latency or over-provisioning with auto-scaling on Basic, Standard and Enterprise clusters from 0 to 1 GBps, or one-click slider scaling up and down on Dedicated (max 30.4 GBps). Manual scaling within cluster tiers Must choose between 7 throughput tiers and manually scale within tier limits (max 4 GBps for Tier 7). Require over-provisioning to handle spikey worloads.
Automatic Cluster Rebalancing Intelligent, continuous data rebalancing based on 10+ factors Intelligently monitor and automatically rebalance partitions, considering node failures, disk usage, replication factors, network throughput, topics and partitions leadership, and other factors. Periodic data balancing based on two factors Monitor and automatically rebalance partitions considering node failures and disk usage.
Infinite Data Retention with Tiered Storage Available Offload data to object storage for easy scaling and cost savings. Available Offload data to object storage for easy scaling and cost savings.
Resiliency Guarantees 99.95% uptime SLA for Single Zone, 99.99% for Multi Zone SLA covers infrastructure, Kafka performance, critical bug fixes, security updates, and more, with service credit amount clearly defined. 99.5% uptime SLA for Single Zone, 99.95% for Multi Zone BYOC offering does not cover maintenance. Service credit not defined.
Upgrades and Patches Automatic upgrades to latest stable Kafka version Zero intervention as part of non-disruptive rolling upgrades to latest stable Kafka version that includes strategic patches ahead of scheduled Apache releases. Self-service upgrades Upgrade clusters to the latest version of the Redpanda platform at own pace. May take time for platform to catch up on latest Kafka version. BYOC offering does not cover maintenance.
Developer and Operator Tooling
Category Feature Confluent RedPanda
Productivity Infra-as-code Deployment Full Terraform support Easily provision & manage Kafka clusters with Terraform for all Confluent resources (e.g., Kafka, Flink, network, security) with automatic policy enforcement. No Terraform support Manual configuration, deployment and management of infrastructure.
Productivity Stream Processing Fully-managed ksqlDB and Apache Flink® (preview) Native fully-managed SQL-based stream processor and industry's only serverless Flink service that can handle both simple and complex data transforms without managing any infrastructure. Not available No stream processing provided. Need to rely on free community tools without support or paid third-party tools.
Productivity No-Code / Low-Code Data Pipelines No-code / No-code stream processing and pipeline builder Stream Designer for creating point-and-click streaming data pipelines with connect and process capabilities. 

Flink Actions (in earlly access) for pre-packaged, turn-key stream processing for topic data.
Not available No no-code / low-code stream processing or data pipelines solution.
Productivity Kafka Connectors 70+ fully-managed connectors Currently 70+ connectors available to integrate with a wide array of modern and legacy services. Fully-managed provisioning, testing, configurations, monitoring and security. 10+ built-in connectors Currently 10+ connectors available to integrate with object storage and several other analytic services.
Governance Schema Registry Fully-managed Central registry with 99.95% uptime SLA to ensure data compatibility. Supports Avro, JSON Schema, and Protobuf. Built-in Central registry to ensure data compatibility. SLA not published. Supports Avro and Protobuf.
Governance Stream quality enforcement Broker-side schema validation and Data Quality Rules Data Quality Rules enables users to enforce "data contract" on the structure and semantics of data to ensure data integrity and compatibility Broker-side schema validation Schema validation enables the broker to verify that data produced to a topic uses a valid schema according to the schema registry rules.
Governance Metadata Catalog Fully-managed Stream Catalog Classify, tag, and organize your metadata to easily search, discover, and browse relevant data streams. Also includes REST and GraphQL APIs to integrate with existing metadata management solutions. Not available No data catalog provided.
Governance Data Lineage Fully-managed Stream Lineage Built-in interactive, end-to-end data lineage that illustrates both current and past dependencies. Enables better understanding and discoverability of data dependencies. Not available No data lineage provided.
Governance Data product management Data Portal Developer-friendly, self-service UI that provides an easy and curated way to find, understand, access and enrich Kafka topics across the organization. Not available No data product tooling provided.
Monitoring and Alerting Monitoring Built-in Metrics API and integration with all major third-party monitoring tools Monitors health metrics at cluster or topic level through a REST API or first-class integration with New Relic Open Telemetry, Datadog, Dynatrace, Grafana Cloud, and Prometheus. Integration with third-party monitoring tools available Built-in Prometheus endpoint and integration with third-party monitoring systems like Datadog and Grafana.
Monitoring and Alerting Notifications Granular, rule-based notifications Manages notifications for accounts, billing, licenses, and service events. Also sets rules to route notification events to specific users through Cloud UI or REST API. Basic, self-configured alerts Generates and configures alerts through third-party monitoring systems based on available metrics.
Deployment Flexibility
Category Confluent RedPanda
Core Cloud Service Providers Full support for AWS, GCP, and Azure Also supports multi-cloud and hybrid deployment. Available on AWS and GCP No current support for Azure.
Cluster Synchronization Fully-managed Cluster Linking for high availability and disaster recovery Provides byte-for-byte replication, including topic permission settings and schema linking. Clients can failover and resume from the same offset on an identical copy of the data. Bidirectional links also integrate fully with open source Kafka clusters. Uses Mirror Maker 2.0 Synchronizes events but does not provide byte-for-byte replication. Client failovers require remapping consumer groups to new offsets.
Readiness for Mission-Critical, Production Workloads
Category Feature Confluent RedPanda
Security Authorization Granular RBACs Granularly control access at multiple levels with 10+ resource-specific roles for both users and service accounts. Fully compatible with ACLs. Basic RBAC in console Three basic roles for users without resource-specific access control.
Security Authentication Supports all common authentication methods

Supports OAuth 2.0, OIDC, API Keys, and plain username/password.

Integrates with existing SAML-based single sign-on (SSO) and identity providers (IdP) such as Google, GitHub, Okta, OneLogin, and Azure Active Directory (AD).

Supports basic authentication methods

Supports plain username/password (OIDC on roadmap) for Redpanda Cloud, and OIDC/SSO for Redpanda Console.

Security Audit Log Covers Kafka operations, security, API and user activities Structured user action logs to detect security threats & anomalies for Kafka cluster, Schema Registry, OAuth/OIDC operations, organization, networking, Streams Designer, and RBACs. Covers Kafka operations and API Captures interactions mainly for Redpanda cluster and Schema Registry.
Security Private Connection Available for all three major public clouds Supports Private Link for AWS, Azure, GCP, and AWS Transit Gateway. Available for AWS Supports AWS PrivateLink only.
Security VPC Peering Supports all three public clouds VPC Peering for AWS, Azure, and GCP. Supports AWS and GCP VPC Peering for AWS and GCP.
Security Data Encryption Self-Managed Keys Available Default at-rest data encryption and in-transit data protection with TLS or TLS 1.2. Also offers BYOK (Bring Your Own Key) option to ensure only the appropriate entity or user can decrypt it. Self-Managed Keys Unavailable Provide in-transit data encryption.
Encrypt data at rest using your own Kafka cluster management tools. No BYOK option.
Trust Compliance Comprehensive support SOC 1/2/3, ISO 27001, ISO 27701, PCI DSS, CSA Star, TISAX, and HITRUST compliant. (see more) Basic support SOC 2 Type 1 and 2 compliant.
Trust Privacy Comprehensive support GDPR, CCPA, and HIPAA readiness. Basic support GDPR readiness.
Enterprise Support Expertise Committer-led support with vertical expertise 100+ Kafka and Flink experts across all continents with 3M+ support hours. Vertical specialists in every component (e.g., Kafka, clients, connectors, inter-datacenter replication, Flink). Small support team with limited vertical expertise ~10 support engineers. No vertical specialists (e.g., no support on Connect).
Enterprise Support Commitment Public contractual SLAs 24x7x365 supports with public SLA, with committed response time as little as 30 min for critical issues and "Game Day" support for special events. 24x7 helpdesk with SLA SLA details not disclosed.

Why Confluent-Managed Kafka Is Trusted Industry-Wide

founded by the creators of Apache Kafka

support hours of Kafka expertise

of Apache Kafka commits come from Confluent

clusters run on Confluent Cloud

Leader in The Forrester Wave™: Streaming Data Platforms

Leader in The Forrester Wave™: Streaming Data Platforms

Google Partner of the Year

Google Technology Partner of the Year

Microsoft Commercial Marketplace Partner of the Year

Microsoft Commercial Marketplace Partner of the Year

Top 10 in Forbes Cloud 100

Top 10 in Forbes Cloud 100

J.P. Morgan Hall of Innovation

JP Morgan Hall of Innovation

Evaluate Confluent Cloud for Yourself

Start testing and deploying fully-managed Kafka clusters in Confluent Cloud within minutes with a 30-day trial.

Additional Resources

Kafka: The Definitive Guide by Kafka’s co-creators

Easily Offload Operational Complexities to Make Kafka Go Farther, Faster

Understanding the TCO and ROI of Apache Kafka & Confluent