Introducing Connector Private Networking: Join The Upcoming Webinar!
RabbitMQ is an open-source traditional message-oriented middleware (MOM) that implements the AMQP (Advanced Message Queuing Protocol) messaging standard for low-latency message queuing. Written in Erlang and built on the Open Telecom Platform framework in 2007, it’s an older messaging system often used as a message broker.
A common alternative to RabbitMQ, Confluent's data streaming platform comes with cloud-native Apache Kafka + Flink, 120+ pre-built connectors, and event-driven communication for modern, scalable, stream processing across all apps, systems, and sources.
RabbitMQ consists of a set of broker processes that host “exchanges” for publishing messages to and queues for consuming messages from. RabbitMQ is often likened to a post office for your applications and messages. RabbitMQ is frequently used to reduce load and delivery time with your messages.
RabbitMQ allows applications that may not have integrations with other applications to connect and communicate information with each other. RabbitMQ is aware of and tracks the status of each application that connects and manages the queue accordingly. This is often referred to as a smart broker/dumb consumer model where the onus is on RabbitMQ rather than the consumer.
RabbitMQ is typically run on a cluster of nodes with distributed queues to provide fault tolerance and high availability. RabbitMQ messages are delivered via a push model which enables low-latency messaging.
There are two main use cases where it makes the most sense to use RabbitMQ, long-running tasks that occur in the background and communication between applications, especially when using microservices.
A typical example of a long-running task would be processing an upload to a web server. Typically a user inputs a document, picture, or other media onto a website, and some processing needs to occur. You can configure your application so the user waits until the processing is complete before allowing the user to continue with the upload or conversion, or you can process the upload in the background and have the user continue with any additional tasks needed.
The other common use case is as a middleman for microservices. As microservices interact with each other, there needs to be a way for the messages to trigger events or tasks on other microservices. Using RabbitMQ, each application can avoid bottlenecks while passing messages.
RabbitMQ also has the ability to process a priority queue. An example of this might be with a batch processing job that runs every hour or every day, or manually on-demand. For this on-demand task, we enter the priority queue and process when submitted rather than waiting for the next scheduled task.
While there are some use cases where it makes sense to use RabbitMQ, there are some important considerations.
Once a message with RabbitMQ has been delivered it is removed from the queue. Whereas in a system like Apache Kafka, the messages are persistent and configurable by the user to a specified retention period, either a certain time or byte size. Depending on how you configure your retention period you have the ability to discover new insight later on that you didn’t think of when you initially ingested data.
RabbitMQ scales vertically whereas Kafka scales horizontally. Kafka was built to easily add additional machines to process your messages based on demand, whereas RabbitMQ relies on getting more powerful hardware to increase throughput.
RabbitMQ is built for traditional messaging. Delivering a large amount of messages to consumers. Kafka was built as a streaming platform where you implement messaging, distributed storage, and processing of data.
RabbitMQ stores messages in memory as long as there is space after which messages will be transferred to disk. This severely impacts the performance of the system. Kafka was built and designed for a wider scale and therefore can handle trillions of messages.
RabbitMQ cannot deal with high throughput, as it doesn’t support message batching, and is optimized for one message at a time instead.
Many of RabbitMQ’s disadvantages stem from being written in Erlang, however, it can also be difficult for a developer to read the source code and understand what’s going on when troubleshooting.
Because RabbitMQ manages the delivery of messages, it can make the overall system difficult to set up, use, and maintain. Message brokers are not as simple as they appear. They have a plethora of configuration and setup options. The size of queues and messages, queue behavior, delivery settings, and message TTL are just a few of the many options available.
From messaging queues, AMQP, to pub/sub and streaming, there are numerous messaging systems available. While Kafka and RabbitMQ are most commonly compared, below is the full list of commonly searched alternatives that are worth considering based on your use case:
While people commonly compare Kafka and RabbitMQ, in reality, they are very different systems. Kafka is a purely distributed log designed for efficient event streaming at a high scale. RabbitMQ is a traditional messaging system, designed to publish messages quickly and delete them.
Which to choose should be a fairly straightforward decision: for lightweight messaging that requires request-response, queuing, and pub-sub, RabbitMQ is well suited. For event streaming use cases that require high throughput, scalability, and permanent message storage, Kafka is the clear winner. To learn more, check out this blog post on benchmarking Kafka, Pulsar, and RabbitMQ.