Celebrating Excellence: Kora wins ‘Best Industry Paper’ at 2023 VLDB Conference

The Very Large Data Bases (VLDB) conference is a premier conference for data management systems and is renowned for showcasing cutting-edge research and industry systems. We are delighted to share that our paper, titled "Kora: A Cloud-Native Event Streaming Platform For Kafka" was awarded the Best Industry Paper at VLDB 2023 out of 29 other prestigious research papers from Meta, Google, Microsoft, Alibaba, Bytedance, and other innovative companies. We’re honored to be part of this group and to be able to share our cutting-edge system with our peers across the industry.

The paper introduces Kora, our next-generation enginethat serves up the Kafka protocol for our thousands of customers and their tens of thousands of clusters in the cloud. It describes Kora's design that enables Confluent Cloud to meet its cloud-native goals, such as reliability, elasticity, and cost-efficiency. Kora's abstractions allow users to think in terms of their workload requirements and not the underlying infrastructure. The paper also discusses how Kora is designed to provide consistent, predictable performance across cloud environments with diverse capabilities. 

VLDB and ‘Best Industry Paper’ Award

VLDB (Very Large Data Bases) is an annual international conference on data management, database and information systems and is considered to be one of most prestigious forums in the field of database technology, attracting researchers, practitioners, and industry experts from around the world. 

Every year, the VLDB program committee goes through a rigorous review process to select papers that present original and innovative research or design that also has practical implications for industry. Just one "Best Industry Paper" award is given per year to a paper that made a significant contribution to the field of database management and has demonstrated the ability to apply research to solve practical problems.

Kora: A Cloud-Native Event Streaming Platform For Kafka

When we announced Kora a couple months ago, our CEO and cofounder Jay Kreps wrote a blog about why we built it. Essentially, we saw a market demand for a cloud-native Kafka service that truly captures all the benefits cloud computing can bring and offers much better scalability, elasticity, system resiliency, performance, and cost efficiency for users. We knew that, compared to the easy path of simply hosting Kafka on the cloud or asking a customer to bring your own cloud, we had chosen a much harder path to completely rearchitect Kafka layer by layer (and doing it three times across AWS, Azure, and GCP), while ensuring 100% compatibility with the Kafka protocol. But we decided this harder path was a worthy one because the outcome of it would be transformative—and our engineers here at Confluent love a challenge.

Over the past five years of developing Kora Engine, we faced numerous challenges and made a series of breakthroughs. You can read about them in the paper itself, but here are a few that we spent many hours debating and designing:

• Cellular architecture to support multi-tenancy at scale with data and performance isolation

• Serverless abstraction to free users from the burden of thinking about low-level details like the amount of memory or CPU type, network bandwidth, or IOPS/throughput/storage bandwidth

• Self-balancing clusters (SBC) and tiered storage architecture to support automatic and elastic expansion, shrink, and balancing

• Automatic detection and mitigation to guarantee high availability with 99.99% uptime SLA on multi-zone clusters

• Seamless replication, backup, and restoration leveraging highly durable object storage and continual automatic durability audits to ensure data durability

Acknowledgments

Kora was the result of effort from numerous individuals, especially our current and former teammates who worked tirelessly over the years to realize the vision of a true cloud-native event streaming platform. The work done to build Kora is a testament to the team's technical skill and commitment to delivering a high-quality product. 

We would like to take this opportunity to express our gratitude to our customers for their feedback and support. Your input has been invaluable in helping us continuously improve Kora and deliver a better product. We would also like to thank the Apache Kafka community for their contributions. The development of Kora would not have been possible without the open-source Kafka project, and we are grateful for the community's ongoing support and collaboration to advance the field of cloud-native data storage. 

Experience Kora today

As mentioned above, Kora is not just an idea on paper—it has been powering 30,000 Confluent Cloud clusters and thousands of customers for years with better scalability, reliability, performance and cost efficiency. You can give it a try today!

P.S.: For any further questions about the Kora paper, feel free to reach out to kora-paper@confluent.io, or meet us in person! We are presenting our research at the VLDB conference in Vancouver today and participating at Current 2023 in San Jose on September 26-27, 2023. Look forward to seeing you there!