Project Metamorphosis: Unveiling the next-gen event streaming platformLearn More

101 Ways to Configure Kafka – Badly

Watch Video

Kafka Summit 2016 | Operations Track

Nearly 3 years of using Kafka without really understanding the documentation has taught us a lot.

At FINN.no, Norway’s biggest online classifieds site, we have been using Kafka since version 0.8-beta in early 2013. Kafka was introduced as a major part of a proof of concept for collecting click events from our web front end, which had approximately 40 million page views a day. The Kafka cluster became popular, and more and more teams started using it, with our default settings. Still we did not have anyone dedicated to operating and configuring Kafka – it was a best effort, and usually the latest person touching the defaults won. As the traffic, number of use cases, and topics grew, we got some scaling problems, stability problems, lost messages and a general feeling among developers that “”our kafka cluster was unstable””. In the end we had to put some effort into find out what we ran, which properties we really needed, and get control over the configuration, our library code and how our clients used Kafka. Since August we haven’t had a complaint, nor downtime on our Kafka cluster.

We will cover our top 5 mistakes:
-No consideration of data on the inside vs outside
-Lack of schema
-One configuration fits all
-128 partitions for everyone
-Running on already overloaded servers

The Germans say “”Schadenfreude ist die schönste Freude””. At this session we’ll tell you what we learned, so you can avoid doing the same.

Speakers:

Henning Spjelkavik, Architect, FINN.no

 

Audun Fauchald Strand, Lead Developer, FINN.no

Sign Up Now

Start your 3-month trial. Get up to $200 off on each of your first 3 Confluent Cloud monthly bills

New signups only.

By clicking “sign up” above you understand we will process your personal information in accordance with our Privacy Policy.

By clicking "sign up" above you agree to the Terms of Service and to receive occasional marketing emails from Confluent. You also understand that we will process your personal information in accordance with our Privacy Policy.

Free Forever on a Single Kafka Broker
i

The software will allow unlimited-time usage of commercial features on a single Kafka broker. Upon adding a second broker, a 30-day timer will automatically start on commercial features, which cannot be reset by moving back to one broker.

Select Deployment Type
Manual Deployment
  • tar
  • zip
  • deb
  • rpm
  • docker
or
Auto Deployment
  • kubernetes
  • ansible

By clicking "download free" above you understand we will process your personal information in accordance with our Privacy Policy.

By clicking "download free" above, you agree to the Confluent License Agreement and to receive occasional marketing emails from Confluent. You also agree that your personal data will be processed in accordance with our Privacy Policy.

This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising, and analytics partners.