Kafka in the Cloud: Why it’s 10x better with Confluent | Find out more

Announcing the Confluent Q1 ‘22 Launch

Written By

The Confluent Q1 ‘22 Launch is live and packed full of new features that enable businesses to continue innovating quickly with real-time experiences fueled by data in motion. Our quarterly launches provide a single resource to learn about the accelerating number of new features we’re bringing to Confluent Cloud, our cloud-native data streaming platform.

Here’s an overview of everything you’ll find within this blog:

Simplified real-time data integrations and massive-throughput elasticity with the Confluent Q1 ‘22 Launch

Q1 '22 launch pillars
Cloud Native, Complete, and Everywhere: These three pillars are paramount to a data-in-motion platform and structure each of our quarterly launches

Complete: Simplified real-time data integrations with 50+ fully managed connectors

Providing both customers and internal decision-making teams with real-time experiences requires integrating data systems and apps from across your entire business—an ever-evolving challenge. With firsthand knowledge of what it takes to stand up and manage a custom source or sink integration—~3–6 engineering months and deep Kafka expertise—we’re excited to announce the availability of 50+ fully managed connectors on Confluent Cloud. This is the largest portfolio of fully managed connectors in the market and includes more than 30 new connectors since our last quarterly launch! Our fully managed connectors eliminate operational overhead and are packed with productivity features like single message transforms and data previews, enabling teams to accelerate their data in motion journey and lower their total cost of ownership (TCO) for Apache Kafka®.

With our new first-class integrations for Datadog and Prometheus, gain deep visibility into Confluent Cloud from within the monitoring tools you already use.

Learn more about all the updates to our Complete pillar

Cloud native: Massive-throughput elasticity with Dedicated cluster expand and shrink

Teams across every industry working with Apache Kafka face the challenge of ensuring their services remain highly available at all times, no matter what. This is a highly expensive, time-consuming, and risky effort, especially for high throughput use cases. Confluent’s Dedicated clusters now allow customers to run their largest workloads more cost effectively with self-service, programmatic capacity shrink available alongside the previously existing capacity expansion. With controls available through the Confluent Cloud UI, CLI, and API alongside built-in safeguards like automatic load re-balancing, you can dynamically expand capacity to handle any workload, then shrink it back down to control costs without updating clients.

New single CSU application sizes available within ksqlDB provide pricing options fit for any budget, meaning more teams can leverage the power of stream processing.

Learn more about all the updates to our Cloud Native pillar

Everywhere: Globally controlled data quality with Schema Linking

Globally consistent data quality controls are critical for maintaining a highly compatible Kafka deployment fit for long-term, standardized use across the organization. With Schema Linking, you now have an operationally simple means of maintaining trusted, compatible data streams across cloud and hybrid environments with shared schemas that sync in real time. Leveraged alongside Cluster Linking, schemas are shared everywhere they’re needed, providing an easy means of maintaining high data integrity across the entire business.

Additionally, manage all of Confluent, across clouds and on-premises, from a single interface with the new Confluent v2 CLI.

Learn more about all the updates to our Everywhere pillar

Start building with features in the Confluent Q1 ‘22 Launch

Ready to get started? Be sure to register for the Q1 ‘22 Launch demo webinar where you’ll learn how to put these new features to use!

Register Now

And if you haven’t done so already, sign up for a free trial of Confluent Cloud. New signups receive $400 to spend within Confluent Cloud during their first 60 days. Use the code CL60BLOG for an additional $60 of free usage.*

Complete

Complete: Seamlessly connect your entire business in real time with 50+ fully managed connectors

To deliver real-time experiences, companies need high-performance, streaming data pipelines across all of their business systems and apps.

We’re excited to announce our milestone of over 50 (and counting) fully managed connectors on Confluent Cloud—the largest portfolio of fully managed connectors in the market. Our newest additions include ActiveMQ source, Databricks Delta Lake sink, Google BigTable sink, IBM MQ source, MQTT source, MQTT sink, PagerDuty sink, RabbitMQ source, RabbitMQ sink, Redis sink, SFTP source, SFTP sink, Splunk sink, and Zendesk source connectors.

With Confluent’s fully managed connectors, you’re able to:

  • Eliminate operational burdens of self-managing connectors and reduce total cost of ownership with the only portfolio of 50+ expert-built, fully managed connectors in the market
  • Operate your business in real time by breaking data silos with high-performance, streaming data pipelines into your data warehouse, database, and data lake
  • Improve data portability and developer workflow with rich, out-of-the-box features that seamlessly transform and monitor data within the connector rather than in downstream systems

50+ fully managed connectors
Our portfolio of 50+ fully managed connectors enables real-time integrations with popular data systems and apps in just a few clicks

“We are a small team tasked with making real-time user interaction data available in our data warehouse and data lake for immediate analysis and insights. By using Confluent’s fully managed Amazon S3, Elasticsearch, Salesforce CDC, and Snowflake connectors, we were able to quickly and easily build high-performance streaming data pipelines that connect our business through Confluent Cloud without any operational burden, accelerating our overall project timeline.”
-Joe Burns, CIO, TeePublic

Our journey to fully managed connectors

We are continuing to make significant investments into our fully managed connector portfolio, as we’ve seen firsthand the challenges that arise when building and managing connectors. For example, developers may use the open source Kafka Connect framework to build custom connectors. This requires significant Kafka knowledge, Connect worker management, and an average of 3–6 engineering months to rigorously design, build, and test each connector.

Confluent offers a pre-built ecosystem of 120+ connectors ready for production use to avoid the challenges of developing connectors. Our next goal was to take pre-built connectors further by offering them as fully managed on Confluent Cloud to remove all of your operational and low-level infrastructure burdens. With fully managed connectors, you can now effortlessly integrate your data systems and shift your focus to business-differentiating projects instead. Only Confluent offers 50+ connectors that are fully managed across the entire stack and enriched with out-of-the-box usability features, built by the world’s foremost Kafka experts.

Benefits of fully managed connectors

  • Fully managed across the stack
    Confluent abstracts away complexities around the Connect worker by managing internal topics, worker configs, monitoring, and security, while giving you the control to elastically scale up and down task counts to fit your business needs.

    While other cloud-hosted Kafka offerings may also manage the underlying Connect worker, they stop short of providing a vast portfolio of fully managed connectors. Without them, you’ll still need to develop, test, maintain, and support each connector in your architecture—missing the opportunity to fully offload non-differentiated development and operational burdens. Additionally, Confluent provides the ongoing support, maintenance, and connector updates you need to power ongoing, mission-critical apps and use cases.

    Ease of use based on provider-managed features
    Only Confluent offers expert-built, fully managed connectors across the entire stack to eliminate operational burdens and risks

  • Modernize your data systems
    Connectors play a critical role in enabling a broad range of streaming use cases, the most popular of which are building streaming ETL pipelines to data warehouses, databases, and data lakes. By moving away from batch processing, you start operating your business in real time.

    Data warehouse, database, and data lake connectors

    Confluent’s portfolio of fully managed connectors enable you to modernize your data warehouses, databases, and data lakes with real-time data pipelines

  • Rich, value-added features: Single message transforms, data preview, and Connect logs
    Beyond source and sink coverage, it’s equally important that our fully managed connectors come with rich features to provide additional value at no extra charge and improve your productivity. Over the past quarter, we’ve brought Connect log events and single message transforms (SMTs) to the cloud so that you can monitor connect logs for contextual information and also perform lightweight data transformations, such as masking and filtering messages in flight within the connector. These SMTs improve scalability and reduce the total cost of ownership by eliminating the need for costly, high-volume data processing in downstream systems.

    Confluent is also the only provider that offers data preview functionality, letting you test a source connector’s output with actual connector configurations prior to launching the connector. This helps with iterative testing, saving you valuable time and enabling you to confidently launch connectors into production.

Complete: Integrated monitoring with Datadog and Prometheus

Maintaining a deep, confident understanding of your entire IT stack is critical to consistently delivering the high-quality services customers demand and expect. However, modern stacks are becoming increasingly complex with the rise of distributed systems and the growing popularity of technologies offloaded to managed services.

Confluent Cloud customers can now use Datadog to monitor their data streams directly alongside the rest of their technology stack. The integration provides businesses with a single source of truth for monitoring and managing mission-critical systems across their entire business and is set up with just a few simple clicks. This allows them to drastically reduce operational complexities when compared to the setup of manual solutions and returns time for focus on higher value-add projects. DevOps teams have easy access to end-to-end visibility of Confluent Cloud within the monitoring tool they already use, providing an easier means to identifying, resolving, or altogether avoiding any issues that may occur.

“Delivering a best-in-class experience for homeowners and housing professionals around the world requires a holistic view of our business. Confluent’s integration with Datadog quickly syncs real-time data streams with our monitoring tool of choice with no operational complexities or middleware required. Our teams now have visibility into the health of all of our systems for reliable, always-on services.” -Mustapha Benosmane, Product leader, ADEO

The new Datadog integration includes:

  • Unified monitoring: Monitor the health of Confluent Cloud resources alongside the rest of your technology stack to get a holistic, end-to-end view within a single platform
  • Serverless operation: Configure your integration in minutes with just a few clicks and zero ops burden
  • Out-of-the-box dashboards: Visualize key cluster health and performance metrics including i/o record counts in a purpose-built dashboard
  • Proactive alerting: Set alerts based on performance against user-defined service level objectives (SLOs)

Deep visibility in Confluent Cloud
Gain deep visibility into Confluent Cloud from within Datadog’s purpose-built dashboard and visualize key cluster metrics including i/o record counts

For representation of custom data streaming use cases and dashboards within Datadog, Confluent makes it simple to access the data you need with our fully managed Datadog Metrics sink connector.

Alongside our new Datadog integration, Confluent customers can now integrate Confluent Cloud with Prometheus for unified monitoring of data streams within their chosen open source platform by using the new /export endpoint in the Confluent Cloud Metrics API.

The Datadog and Prometheus integrations represent the first of many options we’ll be building for customers to monitor Confluent within their chosen third-party tools.

Complete: Additional updates

  • Securely connect to data systems over the public internet with Static Egress IPs (AWS)
    Our fully managed connectors now support