Real-time Adaptation of Financial Market Events with Kafka

Trading never stops in global capital markets. Consider the relay race of trading between the U.S., Asia and European markets, and back to the U.S. in one day. In Over-The-Counter (OTC) markets, new contracts are issued and old contracts are deprecated constantly, which require financial applications to change configurations on-the-fly, while still supporting ongoing derivatives trading throughout the day. At Bloomberg, we strive to be first in delivering financial products and associated market data to clients. To achieve this, we built a real-time data pipeline that leverages Kafka Connect & Kafka Streams and replaces a two-step configuration update, application restart workflow -- while promising zero downtime.

Using JDBC Kafka Connect with custom transformation, we’re able to push configuration changes into Kafka topics with minimal coding. Kafka Streams provides a way to model live market data as a stream and configuration data as a table. We use high-level Kafka Streams Domain Specific Language (DSL) to join data streams and tables. Leveraging Kafka Connect and Kafka Streams together, our data pipeline handles dynamic configuration changes while providing continuous real-time pricing for millions of financial instruments each day. We hope our journey inspires other engineers to innovate and redesign systems using Kafka Infrastructure.