Project Metamorphosis: Unveiling the next-gen event streaming platformLearn More

The Source of Truth: Why The New York Times Stores Every Piece of Content Ever Published in Kafka

Speaker: Boerge Svingen, Director of Engineering, The New York Times

As one of the major news organizations in the world, The New York Times published its first issue on September 18, 1851. In the ensuing 164 years, we have published approximately 16 million articles, for the past 20 years also online. We are now building a new publication pipeline around Kafka. This pipeline functions as the source of truth for all published content, and decouples producers from consumers of content. The pipeline is implemented as an immutable log – all content is published to the log, and all the back-end systems driving the different online experiences access content by consuming this log. I want to explain how this publishing pipeline works, how it interacts with other system, and what our experiences have been so far.

<< Back

Sign Up Now

Start your 3-month trial. Get up to $200 off on each of your first 3 Confluent Cloud monthly bills

New signups only.

By clicking “sign up” above you understand we will process your personal information in accordance with our Privacy Policy.

By clicking "sign up" above you agree to the Terms of Service and to receive occasional marketing emails from Confluent. You also understand that we will process your personal information in accordance with our Privacy Policy.

Free Forever on a Single Kafka Broker
i

The software will allow unlimited-time usage of commercial features on a single Kafka broker. Upon adding a second broker, a 30-day timer will automatically start on commercial features, which cannot be reset by moving back to one broker.

Select Deployment Type
Manual Deployment
  • tar
  • zip
  • deb
  • rpm
  • docker
or
Auto Deployment
  • kubernetes
  • ansible

By clicking "download free" above you understand we will process your personal information in accordance with our Privacy Policy.

By clicking "download free" above, you agree to the Confluent License Agreement and to receive occasional marketing emails from Confluent. You also agree that your personal data will be processed in accordance with our Privacy Policy.

This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising, and analytics partners.