What is Pub/Sub?

Both pub/sub messaging and messaging queues decouple the applications that send messages (producers from the ones that receive them (consumers). This arrangement allows for an asynchronous communication model: it removes or mitigates the delays and impedances associated with direct, synchronous communication between two applications.

Messaging systems decouple applications that produce messages from the ones that consume them. The role names “producer” and “consumer” themselves focus on an application’s immediate context rather than a static role in an architectural pattern (e.g., client-server). The emphasis shifts instead to the message as a time-based fact, or event.

The term publishing suggests an indefinite duration of a message’s usefulness when it is intended for an arbitrary number of consumers (subscribers). These subscribers, while interested in the same messages, all have their own use cases and life cycles. It often makes sense for the topic to persist messages in a way that allows subscribers to review the message store and leave room for later use cases.

Once a consumer or consumer group subscribes to a Kafka topic, it sets a position (offset) from which to read the events in the topic. Each message in a Kafka topic is an immutable record of some event. It can be deleted (e.g., when it expires the retention window of the topic), but it cannot be otherwise modified. Each record is structured as a key-value pair.

If the record key exists, then the producer uses it to determine which partition to send the record, so that every record of a given key will always be written to the same partition. Key-based partitioning and idempotent producers ensure per-partition ordering guarantees for all records with the same key. If the record key is null, the producer essentially sends the record to a random partition based on convenience.

The topic receives each record from a producer in the form of a serialized byte array. Consumers must know the proper deserialization method to decode the message contents. So the data format acts as an API between producers and consumers. Producers choose the schema and serialization format, and hopefully document that choice for consumers and make that schema available in a “schema registry” or searchable catalog.

Shifting these responsibilities – partitioning, serialization/deserialization, compression, etc – to producers and consumers lets a Kafka cluster focus on storage efficiency and optimal throughput, among other things, which in turn supports better performance relative to popular legacy messaging systems.

Both messaging types have been in use for decades, but some developments in this century have dramatically changed the IT landscape, including:

• The rise of cloud-based, distributed, virtualized computing platforms
• Lightweight synchronous communication models, such as REST APIs
• More and different user agents and types (browsers, web/mobile apps, other managed services).

These trends put greater pressure on all modern middleware services, including messaging, to incorporate resilience, high availability, and scalability into their design. Because of this need for real-time data ingestion and integration, Apache Kafka has become the de facto choice for messaging.

Apache Kafka uses a cluster architecture, to which an operator can add multiple brokers (servers). As topics are added, they are distributed among the brokers in the form of one or more partitions. These partitions are usually replicated across the cluster so disruptions to cluster operations are transparent to producers and consumers. Partitions are what allow Apache Kafka to scale to much higher throughput compared to other pub/sub messaging systems while maintaining low latency. Each partition is an append-only commit log, and the Kafka API allows for many individual consumer instances to form a consumer group so that each consumer reads only a subset of the partitions.

Learn more about Kafka architecture and internals.

Partitions are what allow Apache Kafka to scale to much higher throughput compared to other pub/sub messaging systems while maintaining low latency. Each partition is an append-only commit log, and the Kafka API allows for many individual consumer instances to form a consumer group so that each consumer reads only a subset of the partitions.

• When a meal for delivery is ordered on a mobile app, the customer “publishes” their order and delivery details. The specified delivery service is a subscriber that receives orders that are keyed for them. The customer then becomes a subscriber to events detailing the status of their order.
• The roles of message producer and consumer change with the state of the order. The application ensures that messages are routed correctly and promptly.

• Event ticketing is an interesting case for supporting dynamic updates. Say a customer wants to review a venue’s seating chart before making a selection. The application could update seat availability from time to time to keep browsing customers apprised of availability. Some customers choose a seat right away but abandon their cart. Once the timeout window expires, that seat could appear again for browning customers.

• Banks and credit card companies use pub/sub models to aggregate, correlate, and inspect transactions for potential fraud. If two transactions use the same card but occur in different states within a few minutes of each other, it’s a likely sign something is wrong. The pub/sub model could be used to alert the bank’s fraud agency and the affected customer as soon as practical.

• IoT devices are a natural fit with pub/sub messaging. Device data, such as location, and state changes, coupled with the data they capture, provide a rich field of data points useful to a broad number of consumer use cases from data cleaning, transforming, and enriching, all the way to visualizing the data and training inference models with it.
• One concrete example is remote patient monitoring. A hospital can track patient data in each room and pass it to the closest nursing station, the attending physician, and more. Patients recovering at home can use wearable devices to keep caregivers apprised of their condition.