Register for Demo | Confluent Terraform Provider, Independent Network Lifecycle Management and more within our Q3’22 launch!

Evergreen: Building Airbnb’s Merge Queue With Kafka Streams

Most of Airbnb's development happens in several large monolithic repositories. With thousands of changes merged every day there is a high probability that changes that pass verification independently fail to integrate with other changes on the mainline.

Evergreen is a system which guarantees serializability of changes. It enforces that any mainline commit passes all automated checks like compilation, static analysis or tests as if they were merged sequentially without overlapping in time. This is made possible by utilizing optimistic concurrency controls and build system topology in order to parallelize the verification of merging changes.

Evergreen uses an actor model built on top of Kafka Streams. At its core is a state machine that applies a pure function on events and transforms them into actions which are then executed by workers that in turn might produce additional events. We found this architecture particularly well fitted for Kafka Streams with its exactly-once-processing, load balancing capabilities and minimal dependencies. However using Kafka Streams was not without its challenges: one-record-at-a-time processing quickly became a bottleneck and it was not easy to replay records in order to debug the service.

In our talk we will explore Evergreen's architecture and share our learnings from utilizing Kafka Streams in a mission critical system.

Presenters

Janusz Kudelka

Janusz Kudelka is a Staff Software Engineer working on Developer Infrastructure at Airbnb. Before Airbnb he worked on building key-value stores and p2p systems at Facebook. He is passionate about efficiency with a background in distributed systems and high performance computing.

Joel Snyder

Joel is a Software Engineer working on Developer Infrastructure at Airbnb. Prior to developing Evergreen, he worked on source control scalability for large monolithic repositories.