Version 1.5.0
Available fully-managed on Confluent Cloud
Plugin type:
Sink
Enterprise support:
Confluent supported
Verification:
Confluent built
Author:
Confluent, Inc.
Documentation

Kafka Connect Azure Data Lake Storage Gen2

The Azure Data Lake Gen2 Sink Connector integrates Azure Data Lake Gen2 with Apache Kafka. The connector can export data from Apache Kafka® topics to Azure Data Lake Gen2 files in either Avro or JSON formats. Depending on your environment, the Azure Data Lake Gen2 connector can export data by guaranteeing exactly-once delivery semantics to consumers of the Azure Data Lake Gen2 files it produces.

The Azure Data Lake Gen2 sink connector periodically polls data from Kafka and in turn uploads it to Azure Data Lake Gen2. A partitioner is used to split the data of every Kafka partition into chunks. Each chunk of data is represented as an Azure Data Lake Gen2 file. The key name encodes the topic, Kafka partition, and start offset of this data chunk.

Show more

Installation

Confluent Hub CLI installation

Use the Confluent Hub client to install this connector with:
confluent-hub install confluentinc/kafka-connect-azure-data-lake-gen2-storage:1.5.0
Copy

Download installation

Or download the ZIP file and extract it into one of the directories that is listed on the Connect worker's plugin.path configuration properties. This must be done on each of the installations where Connect will be run.
By downloading you agree to the terms of use and software license agreement.
Configure an instance of your connector
Once installed, you can then create a connector configuration file with the connector's settings, and deploy that to a Connect worker.

Support

This connector is supported by Confluent as part of a Confluent Platform subscription.