Show Me How: Build Streaming Data Pipelines for Real-Time Data Warehousing | Register Today

Introducing the OCTOlog

Watch demo: Kafka streaming in 10 minutes

Get started with Confluent, for free

Written By

As humans, we have always relied on opinion. The world is too complex to understand from first principles, so we rely on the views and observations of others to build our own worldview. There is an old parable that highlights how this can prove troublesome. In it, 3 blind men encounter an elephant. The men haven’t encountered an elephant before, and they each approach and touch a different part of the elephant’s body: its leg, tusk, and trunk. They then describe the elephant based on their experience, and their descriptions differ wildly. The blind man that touches the trunk likens the elephant to a snake, the one that touches the leg likens it to a tree, and so on. The story underlines an unfortunate consequence of the human condition: our tendency to claim absolute truths based on limited and subjective experience. Much like today’s fledgling version of ChatGPT, we all have a tendency to be ‘confidently wrong’ ;-)

The scientific method provides a solution through hypotheses, induction, and empirical measurement, and it works well in most sciences, including computer science. But in socio-technical fields–and this includes software engineering–it is harder to apply successfully. Our behavior doesn't map to predictable, consistent outcomes, and organizations of people don't lend themselves to causal analysis. Instead, we inevitably rely on the lived experience of others for opinions.

If you frequent the comments section of Hacker News, you’ll know what I mean. Whether you’re investing in microservices, toying with AI, or building your whole world on top of Postgres - there are almost infinite opinions on tap. Some are insightful and helpful, but none are likely to match your exact situation–and this is the rub. You need to determine whether each opinion is credible, contains bias, which parts apply to your situation, and weave them into your own mental model of the world.

In Confluent’s Office of the CTO, we’ve tried to create a specialization in this, bringing us to the subject of the new column—the OCTOlog. A weekly exploration of the assumptions and opinions we use daily, how we make decisions about the software we build, and ultimately how we can be more successful.

Each new column grasps blindly at a different part of the proverbial elephant. Like the parable’s blind men, columns focus on one part of the bigger picture, piecing together the elephant in its full glory, one week at a time. We humbly invite you to stick with us. See you next week.

  • Ben is the lead technologist in the Office of the CTO at Confluent where he has worked on a wide range of projects, from implementing the latest version of Apache Kafka’s replication protocol through to developing strategies for streaming applications. Before Confluent Ben led the design and build of a company-wide data platform for a large financial institution, as well as working on a number of early service-oriented systems, both in finance and at Thoughtworks.

Watch demo: Kafka streaming in 10 minutes

Get started with Confluent, for free

Did you like this blog post? Share it now