Tick, Tick, Tick….Time is Money When Processing Market Data

At Bloomberg, we process hundreds of billions of financial market events (aka ""ticks"") daily. The Derivatives Data Streaming Engineering team is responsible for processing these ticks in a scalable, performant, and robust manner to provide insights to our clients who trade in the $544 trillion derivatives industry. In most real-time streaming applications, we use event time semantics where possible. However, sometimes processing and wall-clock time are the proper abstractions to use.

This talk will cover two of our real-time streaming systems that rely on different notions of time:

• A Kafka Streams snapshotting system that provides consistent, time-based ""snapshots"" of market events using both event and wall-clock time.

• An Apache Flink-based monitoring system that monitors the quality of ticks published by other engineering teams using both processing and wall-clock time.

In both use cases, we must deal with time across many global markets, different time zones, holidays, and daylight savings time changes. To further complicate matters, how do we handle time in the presence of failures?

After this talk, you will have a better understanding of how we apply the different notions of time in Flink and Kafka Streams, as well as the challenges we faced with each.