Show Me How: Build Streaming Data Pipelines for Real-Time Data Warehousing | Register Today
We live in exciting times, a moment rich with dramatic tech innovations and transformative, data-driven business changes. While the opportunities before us are great, so are the challenges. Big ideas can help us dream of a new world, but it takes builders to get us there. That’s one reason I joined Confluent after many years at major cloud providers in the data analytics space.
Confluent has a vibrant engineering culture that supports grand ideas and those who pursue them. Just as important, it’s in the right place at the right time. We provide a simple model for addressing a core challenge for real-time data: organizing and managing the massive amounts of critical information that is continually pulsing through a modern organization.
Most of the toil and expense of processing data comes from collecting the raw events, cleaning them up in a batch-based data pipeline, then inserting them into a data warehouse where they can be quickly assembled in a form that is easily analyzed. It’s very difficult for customers to work across these stages, given the number of different tools typically used. The batch nature of processing makes this slow and expensive, and it’s challenging to debug or reprocess data when an issue in the final output is found.
This processing problem is best solved by starting from where the data originates and building from there. Most companies build products that optimize for working with data in a single product. Not Confluent. We recognize that people have data in multiple systems, locations, and formats, and they need a way to quickly and easily stitch it together.
But coming up with innovative solutions is not enough. The products must be created by innovative teams and then used and loved on a global scale to make a difference. That process starts with building an engineering team inspired by tackling ambitious problems. It means building a trusting tech culture, a meritocracy that welcomes diverse perspectives. Our teams connect and collaborate, take appropriate risks, and have the grit and drive to get things done.
To get the smartest minds working on the toughest problems, we onboard new hires quickly. We do this in recognition of a leading challenge in any software engineering culture: new people have a hard time understanding what’s going on. There are millions of lines of code in any given big project. Making our engineers knowledgeable about them as quickly as possible puts us on the path to productivity.
This is also a process by which we continually grow smarter. I’m in constant communication with managers on how to optimize everything we do, from the code we create to getting new folks in the mix quickly. By working together, blending our mission and our engineers’ skills, we can create great things. When we empower engineers to do their best and most innovative work, everyone benefits.
Perhaps the best indicator of success for a developer or engineering leader is a passion for their work. And yet, our industry is filled with stories of brilliant people who had great ideas but lacked the company support, resources, and customer focus to see those visions through.
In my own case, in an earlier role, I saw multiple communications platforms briefly win the hearts and minds of users, only to be usurped by a more compelling upstart. Building software is complicated and hard. There are numerous competitors driven by an urgency to get something done. It’s often unclear what the perfect solution will be. It takes grit to keep focused and make tough (sometimes risky) decisions, all while staying true to the passion of creation.
We don’t have all the answers, but I do know this: There is still plenty of great work ahead. We need help to solve our industry’s toughest challenges. And the best place for a sharp engineer to make a mark is with a company that has the talent and vision to support their skills and dreams.
At Confluent, we are creating the next-generation system for how a variety of industries will make the most of their most valuable asset—their data. It’s time to build the road to that business future, together.
When Jade Bowen joined Confluent as an account executive for the enterprise market and 11th employee in the ANZ region, there was only one client for her to work with. Three years later, she’s heading up the entire APAC Customer Success team.
Three years in, Marcus Greer is still excited about the work he does at Confluent. As a software engineer in the Cloud Manageability organization, Marcus helps make customers’ lives easier – giving them insight into the complex systems their businesses depend on.