Confluent
Introducing Self-Service Apache Kafka for Developers
Apache Kafka

Introducing Self-Service Apache Kafka for Developers

Santosh Raghuram

New Confluent Cloud Professional, and Ecosystem Expansion to Google Cloud Platform

The vision for Confluent Cloud is simple. We want to provide a streaming data service to support all customers on their journey to an event-driven enterprise. Realizing that vision requires two things. First, a streaming data service that provides the reliability and scalability required by operations teams, but also delivers a lightweight and agile option for developers. Second, an ecosystem of developer tools around the streaming platform, which may be provided by the open source ecosystem or the leading public cloud providers.We introduced Confluent Cloud Enterprise a year ago, as the first step in realizing our cloud vision. Cloud enterprise enables enterprises in their transition to an event-driven organization, without the burden of operating Kafka at scale. We have been successfully running Confluent Cloud Enterprise for a year now, far exceeding the 99.95% availability SLA committed to our customers. Our customers are now benefiting from a fully-managed, private, secure, and high-performance Kafka- based streaming platform for their mission-critical applications. This has established the first milestone in our vision for Confluent Cloud.

But this initial introduction of Confluent Cloud was only the first step towards realizing our vision. It did not cover the needs of development teams that are asking us for a self-service offering that can be provisioned in minutes, and charged based on actual usage. And it was only available on AWS, limiting access to leading public cloud services like Google’s BigQuery and TensorFlow.

The Developer Challenge

While our enterprise customers are already leveraging the benefits of Confluent Cloud in a simple and highly available managed service, developers wanting to build their applications on a Kafka-based streaming platform continue mostly to operate all the parts of the platform themselves. This means you have to plan infrastructure to run Kafka and its various platform components before you can get started writing your application.The following examples might sound familiar to you:

  • Bob is a machine learning developer building an application to predict what a customer is most likely to purchase next. He wants to use TensorFlow time series algorithms on data being served from Kafka. This requires him to run Kafka on GCP, pull data from its current source, and then finally load it into TensorFlow. Below is an illustration of where Bob spends most of his time today, before he gets his data into TensorFlow. For any developer focused on an end-user’s problem, this is extremely frustrating!

 

 

  • Mary is an analytics developer in a ride hailing startup trying to identify the highest-demand locations at various times of day and various days of the week. While this data comes from Kafka in real time, she needs to run BigQuery on historic data in the data warehouse to do so. However, her Kafka deployment and the data warehouse are both on a different cloud provider! To write this application, she’ll have to mirror Kafka to GCP and migrate historic data to Google Cloud Storage before she could even start writing her application.

These examples expose some things that a cloud-native developer needs. Specifically:

  1. A lightweight, easy-to-use streaming platform based on Apache Kafka that you don’t have to setup and operate yourself.
  2. The need to use an ecosystem of tools around the streaming platform. The tools you want may be offered by specific cloud providers or available in the open source ecosystem.
  3. A path for your operations team (or you, the consummate devops professional) to run it at scale, without adding to the security, compliance, or operational needs of your organization. This is addressed emphatically by Confluent Cloud Enterprise.

Offering cloud independence and liberating developers from the operational burden of running a streaming platform has been a key focus of Confluent Cloud’s vision. To address this, you need a fully managed, low-cost, self-service, Kafka-based event streaming platform, and the freedom to access this service and your data on any public cloud provider.

It is precisely these pain points that we want to address. To this end, we are excited to announce two exciting new capabilities of Confluent Cloud. First, the launch of the self-service Confluent Cloud Professional. And second, the expansion of Confluent Cloud Enterprise and Professional to Google Cloud Platform.

Confluent Cloud Professional: Liberating the Cloud First Streaming Developer

Today, we are announcing the public availability of Confluent Cloud Professional. Running a streaming platform isn’t easy, and Confluent Cloud Professional frees you from that operational burden by offering a fully managed Apache Kafka service, built on the Confluent Platform. It is also perfect for projects under development or early pre-production applications. It offers the legendary performance of Apache Kafka, compatibility with open source Kafka tools and APIs, automatic upgrades to the latest version of Apache Kafka built at Confluent, and additional features that enable you to focus on actually building your application.If you’re a cloud-native developer (or just aspire to be one), then this is how Confluent Cloud Professional helps you build your event-driven applications right away:

  • Self-service provisioning with zero upfront commitment. You don’t even need to enter a credit card until you actually spin up a cluster.
  • Real-time provisioning of Kafka cluster capacity based on throughput and storage.
  • World-wide availability, ensuring your data is adjacent to your applications or in your jurisdiction.
  • The same open-source Apache Kafka APIs and wire protocol you would expect. This means you can integrate any library or client application compatible with Kafka from the large and thriving open-source ecosystem around it.
  • Pricing based on hours of usage to help keep your costs under control.
  • Cloud independence: bridging hybrid clouds to make your data available on GCP or AWS. To the streaming developer, this provides Confluent Cloud on multiple cloud providers, giving users the ability to leverage various cloud-native services along with Kafka.  

Confluent Cloud on GCP: Expanding the Kafka Ecosystem

To that end, we are pleased to announce that we are also expanding Confluent Cloud to  Google Cloud Platform. This is a significant development because many Kafka developers and enterprise customers want to use  GCP services such as BigQuery and TensorFlow. The new Confluent Cloud on GCP enables users to combine the leading Kafka service with GCP services to support a variety of use cases. Let’s now take a look at how Bob and Mary can use Confluent Cloud on GCP to start focusing on their applications in minutes:

  1. Bob signs up for Confluent Cloud Professional and creates a cluster on the Google Cloud Platform. He then uses a source connector to connect his data source to the cloud and writes up a client based on the language of his choice to push data into this cluster. Once this data is in the Kafka cluster, Bob can leverage Kafka’s storage capabilities to retain data, process data (using Kafka-native frameworks like Kafka Streams or KSQL or GCP-native tools like Dataflow or Dataproc), or stream data to TensorFlow, until he completes his time series models. With Cloud Professional, Bob is now able to focus on the time series that will make a difference to his end user!

 

 

  1. While Mary’s scenario is slightly more complex, Confluent Cloud Professional helps her start writing her application with ease. Once Mary creates herself a cluster on GCP and her other cloud provider, she will use a Kafka Connect connector to load data from her existing data warehouse to Confluent Cloud. After that, the Confluent Replicator will keep the GCP cluster in sync with all the real-time and historic data while even keeping consumer offsets straight between the two clusters and also maintain the right offsets for her. All of this data can be streamed into Google Cloud Storage, which makes it available to her BigQuery applications.

How can I get started with Confluent Cloud?

Below is a quick demonstration of how you can get started within minutes.

Click here to experience the ease of building your applications on Confluent Cloud Professional. And as with the first launch of any product, we look forward to your feedback on the user experience, the product and features that’ll help you build applications on Confluent Cloud more effectively. Join us in our Slack Community in the #confluent-cloud channel.

Summing It Up

The twin capabilities of offering a self-service Apache Kafka for developers and access to a broad cloud ecosystem are highly significant developments for you, the Kafka developer. It now allows you to focus on leveraging Kafka without worrying about the operational burden of running one. It also offers a choice of technology and tools, that are best suited to the application. At a grassroots level, these are stepping stones that every developer has wanted. At Confluent, these capabilities help us provide you with the key cornerstones of every enterprise’s journey to become an event-driven organizationbuilding applications that analyze data at massive scale, without an operational burden on the developer!

 

Subscribe to the Confluent Blog

Subscribe

More Articles Like This

dmt-welcome
Neha Narkhede

Welcoming the Distributed Masonry Team to Confluent

Neha Narkhede . .

Streaming platforms force a fundamental rethink of the way companies treat the data that describes and drives their business, taking us away from the static notion of data which sits ...

CCloud_ExternalDataSources
Yeva Byzek

Stream Processing Made Easy With Confluent Cloud and KSQL

Yeva Byzek . .

Confluent Cloud™ is a fully-managed streaming data service based on open-source Apache Kafka. With Confluent Cloud, developers can accelerate building mission-critical streaming applications based on a single source of truth. ...

Neha Narkhede

Confluent Cloud: Enterprise-Ready, Hosted Apache Kafka is Here!

Neha Narkhede . .

Today, 40,000 people in Las Vegas are thinking about the Cloud—not because the weather is dry in Nevada, but because AWS re:Invent is in full force. Today is a perfect ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Try Confluent Platform

Download Now

We use cookies to understand how you use our site and to improve your experience. Click here to learn more or change your cookie settings. By continuing to browse, you agree to our use of cookies.