Aggregating Ad Events with Kafka Streams and Interactive Queries at Invidi

Invidi ad decisioning engine needs semi-realtime feedback on the performance of the ad campaigns it runs.

In the heart of this feedback loop there is a service that aggregates 1B+ daily ad tracking events and serves campaign performance time series to the ad decisioning engine over http. Recently we successfully rewrote it as a pure Kafka Streams application with all data being stored in Kafka and served via Interactive Queries.

The experience was surprisingly not straightforward and we had to trade off some of the simplicity of our processing topology to increase scalability and lower resource consumption.

In this talk we plan to go over the system architecture and share the issues we faced and how we solved them.

Here are some highlights:

• The distribution of our aggregation keys is very skewed, so early repartitioning resulted in poor scalability. To mitigate this issue we used a scatter-gather approach avoiding repartitioning and combining results in IQ. To minimize memory consumption we had to combine the above approach with pre-aggregating events before re-partitioning in a lambda-architecture style.

• Due to multiple stores sharing buffer memory we had to resort to manually deleting entries in our live windowed stores to avoid premature flushing of the aggregates due to cache thrashing.

• We had to implement our own in-memory windowed store to increase IQ performance

We hope that our findings will be helpful to a wider audience.

We also plan to file and fix the issues we discovered in the near future.