Kafka In the Cloud: Why It’s 10x Better With Confluent | Get free eBook

Real-time Adaptation of Financial Market Events with Kafka

Trading never stops in global capital markets. Consider the relay race of trading between the U.S., Asia and European markets, and back to the U.S. in one day. In Over-The-Counter (OTC) markets, new contracts are issued and old contracts are deprecated constantly, which require financial applications to change configurations on-the-fly, while still supporting ongoing derivatives trading throughout the day. At Bloomberg, we strive to be first in delivering financial products and associated market data to clients. To achieve this, we built a real-time data pipeline that leverages Kafka Connect & Kafka Streams and replaces a two-step configuration update, application restart workflow -- while promising zero downtime.

Using JDBC Kafka Connect with custom transformation, we’re able to push configuration changes into Kafka topics with minimal coding. Kafka Streams provides a way to model live market data as a stream and configuration data as a table. We use high-level Kafka Streams Domain Specific Language (DSL) to join data streams and tables. Leveraging Kafka Connect and Kafka Streams together, our data pipeline handles dynamic configuration changes while providing continuous real-time pricing for millions of financial instruments each day. We hope our journey inspires other engineers to innovate and redesign systems using Kafka Infrastructure.


Cliff Cheng
Anuj Arora

Anuj is an Engineering Leader at Bloomberg with 10+ years of experience across software platforms, microservice architecture and distributed data processing. He leads the Derivatives Data Foundations team at Bloomberg. The team is responsible for building data pipelines for pricing and analytics of Over-The-Counter (OTC) Derivatives products. In his free time, Anuj loves to travel and is a big foodie!