Build Predictive Machine Learning with Flink | Workshop on Dec 18 | Register Now

Kafka vs. Pulsar vs. RabbitMQ: Performance, Architecture, and Features Compared

There are numerous messaging systems out there with use cases for message queuing, distributed messaging, and high-performance event streaming systems. Here we'll do a deep side-by-side comparison of Apache Kafka®, Apache Pulsar®, and RabbitMQ®—performance, architecture, features, and other differences to help you choose the best open source messaging system.

learn more

Apache Kafka

Apache Kafka is an open source distributed event streaming platform. Based on the abstraction of a distributed commit log, Kafka is capable of handling trillions of events a day with functionality comprising pub/sub, permanent storage, and the processing of event streams. The de facto transport for event streaming use cases Kafka is used by thousands of organizations, from internet giants to car manufacturers to stock exchanges and has more than 5 million lifetime downloads. Kafka is also available as managed service offerings on all major cloud platforms via Confluent Cloud and others.

Apache Pulsar

Apache Pulsar is an open-source distributed messaging system. Originally developed as a queuing system, it has been broadened in recent releases to add event streaming features. Pulsar makes use of Apache BookKeeper™ for its storage layer—a project created at Yahoo as a high-availability solution to Hadoop's HDFS NameNode (although not ultimately used for that use case). It shares properties with both Kafka and RabbitMQ. Pulsar is a largely community-led project with no enterprise-grade commercial backing today.

RabbitMQ

RabbitMQ (AMQP) is an open-source traditional message-oriented middleware that implements the AMQP messaging standard. Its capabilities include queuing, exchanges, routing, and low-latency messaging. Written in Erlang, RabbitMQ is developed and commercially supported by Pivotal Software, part of VMware.

Benchmark

Throughput

Kafka provides the highest throughput of all systems, writing 15x faster than RabbitMQ and 2x faster than Pulsar, based on the popular OpenMessaging Benchmark*

*Full results described in the associated: benchmarking comparison

Latency

Kafka provides the lowest latency (5ms at p99) at higher throughputs, while also providing strong durability and high availability*.

Kafka in its default configuration is faster than Pulsar in all latency benchmarks, and it is faster up to p99.9 when set to fsync on every message.

RabbitMQ can achieve lower end-to-end latency than Kafka, but only at significantly lower throughputs (30K messages/sec versus 200K messages/sec for Kafka), after which its latency degrades significantly.

*Full results described in the associated: benchmarking comparison

General

Apache Kafka

Apache pulsar

RABBITMQ (AMQP)

General
License	Apache v2	Apache v2	Mozilla Public
Components	Kafka + Zookeeper(ZK is being removed)	Pulsar + Zookeeper + BookKeeper + RocksDB	RabbitMQ
Message consumption model	Pull	Push	Push
Storage architecture	Log	Index	Index

License:

All three technologies are available under fully open source licenses. Confluent also has a Kafka-based distribution with added security, tiered storage and more.

Components:

Kafka makes use of Apache Zookeeper™. Pulsar makes use of Apache Zookeeper for consensus, Apache BookKeeper for message storage which in turn uses the RocksDB database. RabbitMQ has no distributed dependencies.

Message Consumption Model:

Kafka uses a pull-based architecture where consumers pull messages from the server and long-polling is used to ensure new messages are made available instantaneously. RabbitMQ uses a push-based approach synonymous with traditional messaging systems. Pulsar also uses a push-based approach but with an API that simulates consumer pulls. Pull based architectures are often preferable for high throughput workloads as they allow consumers to manage their own flow control, essentially fetching only what they need. Push based architectures require flow control and backpressure to be integrated into the broker.

Storage Architecture:

Kafka uses a distributed commit log as its storage layer. Writes are appended to the end of the log. Reads are sequential starting from an offset and data is zero-copied from the disk buffer to the network buffer. This works well for event streaming use cases. Conversely, RabbitMQ and Pulsar both use index-based storage systems. These keep data in a tree structure to provide the fast access necessary for acknowledging individual messages. The fast individual reads that tree structures provide come at the cost of write overhead, which can manifest as either decreased write throughput, increased write latency, or increased write amplification compared to a log. RabbitMQ and Pulsar differ in the design of the storage layer also. RabbitMQ is designed to store messages for a short period of time only. Pulsar like Kafka can retain messages indefinitely.

Ease of Use

Apache Kafka

Apache pulsar

RABBITMQ (AMQP)

Ease of Use
Operational simplicity
Documentation & learning
Open source ecosystem
Size of user community
Enterprise support
Managed cloud offerings
Management tooling built-in
Integrations (databases, REST, COTS, etc.)
Client library diversity

Operational Simplicity:

Kafka is a cluster-based technology with a medium-weight architecture requiring two distributed components: Kafka's own servers (brokers) plus ZooKeeper™ servers. Zookeeper adds an additional level of complexity but the community is in the process of removing the ZooKeeper component from Kafka which will significantly decrease Kafka's operational burden. The Kafka ecosystem includes two different mature Kubernetes operators, one open source and one commercial. These ease cluster management significantly. Pulsar has a heavy-weight architecture that is the most complex in this comparison. It requires four components be configured and understood: Pulsar servers (brokers) plus Apache BookKeeper (bookies) plus Apache ZooKeeper servers as well as the RocksDB database used by BookKeeper. RabbitMQ has a comparatively light-weight architecture, requiring only RabbitMQ's own servers (brokers). However it's distributed configurations are less complete and often more complex to manage compared to Kafka and Pulsar (e.g. the sharding plugin). RabbitMQ is currently without a production-grade Kubernetes operator (although a commercial one is in the works).

Documentation and Learning:

Kafka has over half a million words of official documentation, 13 textbooks, a rich site of tutorials, demos, podcasts, and video tutorials, more than 21,000 questions on Stack Overflow, online courses from Confluent, Udemy and more. RabbitMQ also has a rich suite of documentation, tutorials, conference talks, and online courses. Pulsar has comparatively lightweight documentation.

Enterprise Support:

Kafka's popularity means that enterprise-grade support offerings are available from a number of large reputable vendors including Confluent. Confluent also has a more complete enterprise distribution of Kafka with Confluent Platform and a fully managed service alongside Confluent Cloud which includes advanced security, management features, and tiered storage. RabbitMQ is developed and commercially supported by Pivotal Software, part of VMware. Pulsar is supported by a small startup, Streamnative.

Managed Cloud Offerings:

Enterprise cloud offerings exist for both Kafka and RabbitMQ on the three major clouds: Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). Confluent Cloud, for example, is a fully managed Kafka service that provides a “serverless” version of Kafka, i.e., the customer interacts only with topics and data, whereas all infrastructure concerns (Kafka servers, ZooKeeper, etc.) are transparently managed behind the scenes.

Ecosystem and Community:

Kafka has an extensive ecosystem, including open source clients, UIs, data balancers, Kubernetes operators, plugins, connectors and third-party tooling in both open source and commercial forms. It is the de facto standard transport for Spark, Flink and of course Kafka Streams and ksqlDB. Because of its wide-spread adoption, Kafka also has a large, active, and global user community that regularly participates in conferences and events. In comparison, RabbitMQ has a medium-sized community but a rich ecosystem that includes clients in many languages and tools that interface with the AMQP protocol. Pulsar's ecosystem and community is still in its infancy. The table below provides some base metrics.

Kafka

Pulsar

RabbitMQ

Number of Programming Languages Supported	17	6	22
Stack Overflow Questions	21,233	134	11,430
Slack Community Size	23,057	2,332	7,492
Job mentions in the USA on Monster.com	4,293	23	536
Meetups	486	1	1

Integrations (Big Data, databases, REST, COTS, etc.):

Apache Kafka includes the Kafka Connect framework, which allows customers to deploy ready-to-use “connectors” to continuously ingest data streams into Kafka from other systems like databases, messaging systems, and applications, and vice versa. Kafka's ecosystem includes more than 400 open source connectors (with more than a hundred officially supported by Confluent). These include interfacing to S3, Elasticsearch, MongoDB, Oracle, JMS, Syslog, and Salesforce. RabbitMQ also has a large ecosystem of interfaces but lacks the powerful distributed connector execution framework that Kafka offers. Pulsar has around 20 connectors, including MySQL, Twitter, and Kafka.

Client library diversity:

Apache Kafka officially supports 17 languages. RabbitMQ supports 22. Pulsar supports 6.

Performance & availability

Apache Kafka

Apache pulsar

RABBITMQ (AMQP)

Performance & availability
High-throughput workloads
Low-latency workloads
Elastic scaling
High availability
Global data replication
Ordering guarantees
Permanent storage

Throughput, Latency, and Scale:

Kafka provides the highest throughput and lowest latency of all three systems, based on measurements taken using the Open Messaging Benchmark configured for a typical event streaming use case. Its ability to operate at network-speeds and to process trillions of events (messages) per day has been demonstrated in production by power users such as Netflix, LinkedIn, Uber, Microsoft, Twitter, and Tencent. Large Kafka deployments can consist of hundreds of servers or more. Pulsar, by comparison, has good scalability properties but is limited by its comparatively complex storage architecture. This means that, for the same resources, it cannot match Kafka's high-end throughput (this complexity and the associated cost is why Twitter abandoned their own BookKeeper-based system, similar in architecture to Pulsar, in favor of Kafka). RabbitMQ can benchmark at millions of messages per second, but real-world performance tends to be many orders of magnitude lower, particularly if datasets exceed the memory available. Also, RabbitMQ does not provide first-class distributed systems primitives like sharding, distributed consumption, or data balancing in the same seamless way Kafka and Pulsar do. All three technologies provide broadly similar latencies.

High Availability:

All three systems include high-availability features. Kafka and Pulsar both provide seamless availability with sharded partitions, which are also replicated for high availability across machines or availability zones. RabbitMQ has a plugin for sharding and supports mirroring, but the result is less seamless than the other two.

Global Data Replication:

All three technologies include features for global data replication. This could be, between two datacenters or between two geo-regions in the cloud. Kafka and Pulsar both replicate in parallel allowing high throughput geo-replication.

Ordering Guarantees:

Kafka provides strong ordering guarantees at a partition level even when consuming a topic in parallel across many consumers. RabbitMQ provides strong ordering for publish-subscribe and single-consumer topics but queues subscribed to by multiple consumers may break ordering guarantees. Ordering violations are also possible in Pulsar's 'shared' and Kafka-like 'key-shared' modes.

Permanent Storage:

Kafka stores your data durably and reliably much like a normal database. Data retention is user configurable per Kafka topic. Companies like The New York Times store data forever in Kafka and use it as the source-of-truth for their business. Kafka features like topic compaction make such setups more efficient. Tiered Storage via KIP-405 further reduces the costs of long-term storage. RabbitMQ is different: by design, messages are removed once they are consumed and are not stored beyond that. So while RabbitMQ can store data, it is not designed to be held permanently. Pulsar is also designed to delete messages once they have been consumed but unlike RabbitMQ can store data for longer in its BookKeeper storage layer. Bookkeeper's retention capabilities are limited by the fine-grained metadata it stores in Zookeeper.

Features

Apache Kafka

Apache pulsar

RABBITMQ (AMQP)

Features
Built-in stream processing
Message replay, time travel
Exactly-once processing
Topic (log) compaction
Security

Built-in Stream Processing:

Kafka is the only technology that ships with functionality for stream processing. Its Kafka Streams library allows developers to implement elastic and scalable client applications that can leverage essential stream processing features such as tables, joins, aggregations, windowing, state management, fault-tolerance, security functionality, elastic scaling of client applications and exactly-once processing. The Kafka Streams ecosystem includes further tools for application development, including the Spring framework and ksqlDB an event streaming database built for Kafka. In comparison, Pulsar provides only basic functionality for stream processing using its Pulsar Functions interface. This is suited to simple callbacks, but it isn't a true stream processing offering. RabbitMQ provides no native stream processing features but can be integrated with external processing engines like Apache Flink®.

Message Replay, Time Travel:

Because both Kafka and Pulsar can store data permanently, they support use cases that a traditional messaging broker cannot. Data can be processed and reprocessed as needed to facilitate A/B testing, train analytic models, capture audit trails, recompute data due to (say) an application bug that produced incorrect results, and so forth. RabbitMQ can replay messages that have not been acknowledged, but the number of messages that can be retained and replayed efficiently is small in comparison to that of Kafka and Pulsar.

Exactly Once Processing:

Pulsar supports idempotent writes—meaning that duplicate messages are not stored in Pulsar's storage layer, but it does not support the transactional reads needed for exactly once (aka Effectively once) processing. For example, Pulsar cannot guarantee exactly-once semantics when messages are read and then written to a secondary topic, which is a common requirement for many practical use cases. Kafka supports at-most-once, at-least-once, and also high-throughput exactly once semantics specifically aimed at end-to-end stream processing use cases (Kafka Streams) and exactly-once enabled connectors (Kafka Connect). This is achieved with a combination of both idempotent writes to Kafka and its transaction API. RabbitMQ does not include transactions or idempotence in the broker, meaning it supports only at-least-once semantics.

Topic (Log) Compaction:

Kafka supports native topic compaction, which runs on all brokers. This runs automatically for compacted topics, condensing the log down to the latest version of messages sharing the same key. Pulsar supports a similar feature, but it runs over the network, streaming data from the storage layer to the broker layer, running compaction, then saving the result back. It also does not work in the same seamless way but instead creates a “snapshot” of a topic compacted at some prior point in time (with the original topic remaining). RabbitMQ does not support topic compaction.

Use Cases

Apache Kafka

Apache pulsar

RABBITMQ (AMQP)

Use Cases
Mission-critical
Event Streaming
Pub/sub
Message routing
Queueing

Mission Critical:

Kafka is used by tens of thousands of organizations across a plethora of industries, including more than 80% of Fortune 100 companies. Examples include mission-critical use cases spanning stock exchanges, internet, telco, retail, car-sharing, banking and insurance, automotive, healthcare, manufacturing, news and media, gaming, and many more. RabbitMQ is a popular traditional messaging system and, similar to Kafka, is widely used across industries in mission-critical applications. Examples include telcos, banks and insurances, retail, Internet companies, logistics, and more. Pulsar has significantly less adoption compared to Kafka or RabbitMQ, but there are production use cases notably at Yahoo where the project initiated and in China. Examples include online retail, media, and web hosting companies. Pulsar lacks public mission-critical use cases.

Event Streaming:

Kafka provides the rich suite of tools required for event streaming use cases, a category it helped define. These include strongly ordered parallel message delivery, message storage that decouples consumers from the rate of input events, and rich features for processing events and creating streaming pipelines. RabbitMQ lacks the ordered parallel message delivery properties needed for such use cases. Pulsar does better in this category but lacks the strong guarantees needed for event streaming pipelines.

Message Routing:

As a mature traditional messaging system, RabbitMQ has an extensive set of server-side (i.e., broker-side) message routing capabilities via the routing keys and exchanges described in the AMQP protocol. Kafka provides similar capabilities through Kafka Connect and Kafka Streams, including content-based routing, message transformation, and message enrichment. Unlike RabbitMQ these components run in a separate layer. Pulsar is similar to Kafka in this regard but with more limited routing capabilities in its Pulsar Functions processing layer. But like RabbitMQ, Pulsar runs its Pulsar Functions inside the broker.

Message Queuing:

RabbitMQ supports message queuing via AMQP e.g., for implementing a worker queue that delivers messages round-robin to competing consumers. Kafka focuses on event streaming and delivers messages based on the order of messages in a partition. Kafka's consumer groups feature provides a model similar to competing consumers. Pulsar also supports a queuing API that provides the highest offset delivery similar to RabbitMQ, though it is limited in comparison.

Summary

In reality, Kafka, RabbitMQ, and Pulsar are three very different systems. Kafka is a pure distributed log designed for efficient event streaming at a high scale. RabbitMQ is a traditional messaging system, designed to publish messages quickly and delete them. Pulsar sits somewhere in between. It's not a distributed log in the true sense, but it synthesizes some similar properties.

Which to choose should be a fairly straightforward decision: for lightweight messaging that requires request-response, queuing, and pub-sub RabbitMQ is well suited; Pulsar is really only for the brave at heart, but it may have a place in the future for those that require both queuing and event streaming in the same system; for event streaming use cases that require high throughput, scalability, and permanent message storage Kafka is the clear winner. To learn more, check out this blog post on benchmarking Kafka, Pulsar, and RabbitMQ.