Confluent Releases Managed V2 Connector for Apache Kafka® for Azure Cosmos DB

We’re excited to announce the General Availability (GA) of the Confluent fully managed V2 connector for Apache Kafka® for Azure Cosmos DB! This release marks a major milestone in our mission to simplify real-time data streaming from and to Azure Cosmos DB using Apache Kafka®.

The V2 connector is now production-ready and available directly from the Confluent Cloud connector catalog. This managed connector allows you to seamlessly integrate Azure Cosmos DB with your Kafka-powered event streaming architecture—without worrying about provisioning, scaling, or managing the connector infrastructure.

Whether you’re building a real-time analytics pipeline, syncing operational data between services, or enabling event-driven microservices, this connector empowers you to unlock the full potential of change data capture (CDC) and low-latency writes across your Azure Cosmos DB containers.

Why V2 of the New Connectors?

The release of version 2.0 isn’t just a simple upgrade. It’s a complete architectural overhaul that addresses the limitations of version 1.0 and introduces scalability, performance, and flexibility across both the source and sink connectors. Check out migration guidance for detailed comparisons between V1 and V2 and what to keep in mind when upgrading.

Key motivations behind V2 include:

• Scalability Bottlenecks

In V1, every container required a separate Kafka task for CDC. V2 eliminates this constraint by allowing multiple containers to be processed by a single Kafka task, dramatically improving task efficiency and lowering operational cost.

• Improved Performance

By adopting the pull-based change feed model, the source connector reduces memory usage and thread contention—leading to more stable and performant stream processing.

• Flexible Write Patterns

The sink connector now supports multiple write strategies, enabling much finer control over how data lands in Azure Cosmos DB, especially in concurrent or distributed write scenarios.

• Better Integration With Kafka Infrastructure

Metadata management now leverages Kafka’s native offset tracking, removing the dependency on Cosmos-specific lease containers and ensuring safer, more reliable checkpointing.

In short, V2 was built from the ground up to be cloud-native, more intelligent, and aligned with Kafka best practices.

Advantages of the New Connectors

The GA version of the Confluent-managed Azure Cosmos DB Kafka connector introduces critical enhancements that make it a powerful tool for enterprise-grade streaming workloads:

• High-Throughput Change Feed Reads Thanks to container batching and pull-model support, you can ingest high volumes of changes from many containers without a proportional increase in Kafka tasks.

• Advanced Sink Strategies Choose from multiple write modes, which offer fine-grained control over upserts, deletes, and conditional updates—ideal for microservices architectures and stateful event handling:

  • Item Override (default)

  • Create Only

  • Delete

  • Update if ETag Matches

• Built-in Throughput Control Prevent throttling and manage RU usage more effectively with native throughput control, essential for cost and performance optimization in Azure Cosmos DB.

• Integrated Metrics and Observability Deep integration with Confluent monitoring and Azure software development kit metrics means you get full visibility into connector health, lag, throughput, and errors—perfect for site reliability engineering teams and production troubleshooting.

• Improved Security and Authentication It now supports Microsoft Entra ID service principals using client secrets, with future enhancements planned for managed identities and cert-based authentication, enabling secure enterprise integration.

• More Reliable Metadata Handling It eliminates reliance on lease containers by adopting Kafka-native offset management and introduces a low-frequency metadata topic for robust handling of partition splits and merges—ensuring resilience during Azure Cosmos DB’s dynamic scaling events.

With these capabilities, V2 is fully equipped to support high-scale, production-ready, event-driven solutions in the cloud.

Getting Started With the New Connectors

The new connectors are very easy to operationalize. Before getting started on the setup itself, please note the prerequisites:

Azure Cosmos DB Source Connector Prerequisites

1. Authorized access to a Confluent Cloud cluster on Microsoft Azure

2. The Confluent CLI installed and configured for the cluster (see Install the Confluent CLI)

3. Schema Registry enabled to use a Schema Registry-based format (for example, Avro, JSON_SR [JSON Schema], or Protobuf)

4. Authorized access to read data in Azure Cosmos DB (for more information, see Security in Azure Cosmos DB)

The Azure Cosmos DB is now configured to use the NoSQL API.

Azure Cosmos DB Sink Connector Prerequisites

1. Authorized access to a Confluent Cloud cluster on Microsoft Azure

2. The Confluent CLI installed and configured for the cluster (see Install the Confluent CLI)

3. Schema Registry enabled to use a Schema Registry-based format (for example, Avro, JSON_SR [JSON Schema], or Protobuf)

4. At least one source Kafka topic in your Confluent Cloud cluster

5. The Azure Cosmos DB and the Kafka cluster in the same region

6. The Azure Cosmos DB having an ID field in every record

Connector Setup

Setting up the new connectors is very similar to the regular setup experience with Confluent Cloud. Connectors can be set up via UI, CLI, or API.

Steps to Set Up the Azure Cosmos DB Source V2 Connector

1. Log in to Confluent Cloud:

   1. Go to https://confluent.cloud/ and sign in.

   2. Select your environment and cluster.

2. Create the connector:

   1. Navigate to the "Connectors" tab in the left sidebar.

   2. Click "Add connector."

   3. Search for "Azure Cosmos DB" and select "Azure Cosmos DB Source Connector V2."

3. Authenticate with the Kafka cluster:

   1. Authenticate via service account or API key.

4. Set Azure Cosmos DB connection settings: Connect to the DB with the following:

   1. Cosmos Endpoint – the Azure Cosmos DB endpoint URL 

   2. Cosmos Database Name – the name of your Azure Cosmos DB

   3. Cosmos Connection Auth Type – MasterKey or ServicePrincipal

5. Configure:

   1. Choose the appropriate output record value format (AVRO, JSON_SR, Protobuf).

   2. Set up a topic-container map (a comma-delimited list of Kafka topics mapped to Cosmos containers).

6. Set sizing:

   1. Specify the number of tasks required for the connector.

7. Review and launch:

   1. Verify all settings.

   2. Click "Launch" to deploy the connector.

Here is a video from Confluent Cloud that details the setup process:

Steps to Set Up the Azure Cosmos DB Sink V2 Connector

1. Log in to Confluent Cloud:

   1. Go to https://confluent.cloud/ and sign in.

   2. Select your environment and cluster.

2. Create the connector:

   1. Navigate to the "Connectors" tab in the left sidebar.

   2. Click "Add connector."

   3. Search for "Azure Cosmos DB" and select "Azure Cosmos DB Sink Connector V2."

3. Choose the right topic from the topics list.

4. Authenticate with the Kafka cluster:

   1. Authenticate via service account or API key.

5. Set Azure Cosmos DB connection settings. Connect to the DB with the following:

   1. Cosmos Endpoint – The Azure Cosmos DB endpoint URL 

   2. Cosmos Database name – the name of your Azure Cosmos DB

   3. Cosmos Connection Auth Type – MasterKey or ServicePrincipal

6. Configure:

   1. Choose the appropriate input record value format (AVRO, JSON_SR, Protobuf).

   2. Specify a topic-container map (a comma-delimited list of Kafka topics mapped to Azure Cosmos DB containers).

   3. Choose the right Azure Cosmos DB Item write strategy. Note that this defaults to ItemOverwrite.

7. Set sizing:

   1. Specify the number of tasks required for the connector.

8. Review and Launch:

   1. Verify all settings.

   2. Click "Launch" to deploy the connector.

Here is a video from Confluent Cloud that details the setup process:

Coming Soon: Direct Integration With Azure Cosmos DB

Both the Confluent source and sink V2 connectors are soon to be natively integrated with Azure Cosmos DB. This means users will be able to configure and deploy the connectors directly from the Azure interface, making setup faster and more seamless than ever. Here is a preview of how the integration works.

Setup of the source connector from Azure Cosmos DB:

Setup of the sink connector from Azure Cosmos DB:

Conclusion

The release of the new fully managed Azure Cosmos DB V2 source and sink connectors marks a significant milestone for organizations leveraging both Kafka and Microsoft's globally distributed Azure Cosmos DB service. These source and sink connectors eliminate the complexity previously associated with integrating these powerful technologies, providing a streamlined path for real-time data movement between Azure Cosmos DB and Kafka ecosystems.

By implementing these connectors, data teams can now:

• Capture Azure Cosmos DB change feeds directly into Kafka topics with minimal latency

• Stream data from Kafka into Azure Cosmos DB collections without custom code development

• Maintain consistency across distributed systems through configurable transformation options

• Leverage Confluent Cloud fully managed services to lower operational complexity and reduce costs

For organizations building event-driven architectures, these connectors represent a critical building block that simplifies infrastructure while improving reliability. The ability to seamlessly connect Azure Cosmos DB's multi-model capabilities with Kafka's streaming prowess opens new possibilities for real-time analytics, microservices synchronization, and data replication scenarios.

As cloud-native architectures continue to evolve, having certified, well-maintained integration points between key services becomes increasingly vital. We're excited to see how these connectors will empower developers to build more responsive, resilient data pipelines that leverage the best of both platforms.

Apache®, Apache Kafka®, Kafka®, and the Kafka logo are registered trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by the Apache Software Foundation is implied by the use of these marks. All other trademarks are the property of their respective owners.