New Confluent Cloud Professional, and Ecosystem Expansion to Google Cloud Platform
The vision for Confluent Cloud is simple. We want to provide a streaming data service to support all customers on their journey to an event-driven enterprise. Realizing that vision requires two things. First, a streaming data service that provides the reliability and scalability required by operations teams, but also delivers a lightweight and agile option for developers. Second, an ecosystem of developer tools around the streaming platform, which may be provided by the open source ecosystem or the leading public cloud providers.We introduced Confluent Cloud Enterprise a year ago, as the first step in realizing our cloud vision. Cloud enterprise enables enterprises in their transition to an event-driven organization, without the burden of operating Kafka at scale. We have been successfully running Confluent Cloud Enterprise for a year now, far exceeding the 99.95% availability SLA committed to our customers. Our customers are now benefiting from a fully-managed, private, secure, and high-performance Kafka- based streaming platform for their mission-critical applications. This has established the first milestone in our vision for Confluent Cloud.
But this initial introduction of Confluent Cloud was only the first step towards realizing our vision. It did not cover the needs of development teams that are asking us for a self-service offering that can be provisioned in minutes, and charged based on actual usage. And it was only available on AWS, limiting access to leading public cloud services like Google’s BigQuery and TensorFlow.
The Developer Challenge
While our enterprise customers are already leveraging the benefits of Confluent Cloud in a simple and highly available managed service, developers wanting to build their applications on a Kafka-based streaming platform continue mostly to operate all the parts of the platform themselves. This means you have to plan infrastructure to run Kafka and its various platform components before you can get started writing your application.The following examples might sound familiar to you:
- Bob is a machine learning developer building an application to predict what a customer is most likely to purchase next. He wants to use TensorFlow time series algorithms on data being served from Kafka. This requires him to run Kafka on GCP, pull data from its current source, and then finally load it into TensorFlow. Below is an illustration of where Bob spends most of his time today, before he gets his data into TensorFlow. For any developer focused on an end-user’s problem, this is extremely frustrating!
- Mary is an analytics developer in a ride hailing startup trying to identify the highest-demand locations at various times of day and various days of the week. While this data comes from Kafka in real time, she needs to run BigQuery on historic data in the data warehouse to do so. However, her Kafka deployment and the data warehouse are both on a different cloud provider! To write this application, she’ll have to mirror Kafka to GCP and migrate historic data to Google Cloud Storage before she could even start writing her application.
These examples expose some things that a cloud-native developer needs. Specifically:
- A lightweight, easy-to-use streaming platform based on Apache Kafka that you don’t have to setup and operate yourself.
- The need to use an ecosystem of tools around the streaming platform. The tools you want may be offered by specific cloud providers or available in the open source ecosystem.
- A path for your operations team (or you, the consummate devops professional) to run it at scale, without adding to the security, compliance, or operational needs of your organization. This is addressed emphatically by Confluent Cloud Enterprise.
Offering cloud independence and liberating developers from the operational burden of running a streaming platform has been a key focus of Confluent Cloud’s vision. To address this, you need a fully managed, low-cost, self-service, Kafka-based event streaming platform, and the freedom to access this service and your data on any public cloud provider.
It is precisely these pain points that we want to address. To this end, we are excited to announce two exciting new capabilities of Confluent Cloud. First, the launch of the self-service Confluent Cloud Professional. And second, the expansion of Confluent Cloud Enterprise and Professional to Google Cloud Platform.
Confluent Cloud Professional: Liberating the Cloud First Streaming Developer
- Self-service provisioning with zero upfront commitment. You don’t even need to enter a credit card until you actually spin up a cluster.
- Real-time provisioning of Kafka cluster capacity based on throughput and storage.
- World-wide availability, ensuring your data is adjacent to your applications or in your jurisdiction.
- The same open-source Apache Kafka APIs and wire protocol you would expect. This means you can integrate any library or client application compatible with Kafka from the large and thriving open-source ecosystem around it.
- Pricing based on hours of usage to help keep your costs under control.
- Cloud independence: bridging hybrid clouds to make your data available on GCP or AWS. To the streaming developer, this provides Confluent Cloud on multiple cloud providers, giving users the ability to leverage various cloud-native services along with Kafka.
Confluent Cloud on GCP: Expanding the Kafka Ecosystem
- Bob signs up for Confluent Cloud Professional and creates a cluster on the Google Cloud Platform. He then uses a source connector to connect his data source to the cloud and writes up a client based on the language of his choice to push data into this cluster. Once this data is in the Kafka cluster, Bob can leverage Kafka’s storage capabilities to retain data, process data (using Kafka-native frameworks like Kafka Streams or KSQL or GCP-native tools like Dataflow or Dataproc), or stream data to TensorFlow, until he completes his time series models. With Cloud Professional, Bob is now able to focus on the time series that will make a difference to his end user!
- While Mary’s scenario is slightly more complex, Confluent Cloud Professional helps her start writing her application with ease. Once Mary creates herself a cluster on GCP and her other cloud provider, she will use a Kafka Connect connector to load data from her existing data warehouse to Confluent Cloud. After that, the Confluent Replicator will keep the GCP cluster in sync with all the real-time and historic data while even keeping consumer offsets straight between the two clusters and also maintain the right offsets for her. All of this data can be streamed into Google Cloud Storage, which makes it available to her BigQuery applications.
How can I get started with Confluent Cloud?
Click here to experience the ease of building your applications on Confluent Cloud Professional. And as with the first launch of any product, we look forward to your feedback on the user experience, the product and features that’ll help you build applications on Confluent Cloud more effectively. Join us in our Slack Community in the #confluent-cloud channel.
Summing It Up
The twin capabilities of offering a self-service Apache Kafka for developers and access to a broad cloud ecosystem are highly significant developments for you, the Kafka developer. It now allows you to focus on leveraging Kafka without worrying about the operational burden of running one. It also offers a choice of technology and tools, that are best suited to the application. At a grassroots level, these are stepping stones that every developer has wanted. At Confluent, these capabilities help us provide you with the key cornerstones of every enterprise’s journey to become an event-driven organization—building applications that analyze data at massive scale, without an operational burden on the developer!