Elevate your stream processing w/ The Force of Kafka + Flink Awakens | Read Now


Real-time Adaptation of Financial Market Events with Kafka

« Kafka Summit Americas 2021

Trading never stops in global capital markets. Consider the relay race of trading between the U.S., Asia and European markets, and back to the U.S. in one day. In Over-The-Counter (OTC) markets, new contracts are issued and old contracts are deprecated constantly, which require financial applications to change configurations on-the-fly, while still supporting ongoing derivatives trading throughout the day. At Bloomberg, we strive to be first in delivering financial products and associated market data to clients. To achieve this, we built a real-time data pipeline that leverages Kafka Connect & Kafka Streams and replaces a two-step configuration update, application restart workflow -- while promising zero downtime.

Using JDBC Kafka Connect with custom transformation, we’re able to push configuration changes into Kafka topics with minimal coding. Kafka Streams provides a way to model live market data as a stream and configuration data as a table. We use high-level Kafka Streams Domain Specific Language (DSL) to join data streams and tables. Leveraging Kafka Connect and Kafka Streams together, our data pipeline handles dynamic configuration changes while providing continuous real-time pricing for millions of financial instruments each day. We hope our journey inspires other engineers to innovate and redesign systems using Kafka Infrastructure.

Related Links

How Confluent Completes Apache Kafka eBook

Leverage a cloud-native service 10x better than Apache Kafka

Confluent Developer Center

Spend less on Kafka with Confluent, come see how