Unlock the Power of Your Data Warehouse: Introducing the Snowflake Source Connector for Confluent Cloud

Organizations have mastered collecting and storing vast amounts of data in cloud data warehouses like Snowflake. This central repository has become the single source of truth for analytical insights, business intelligence, and reporting. However, the true potential of this data remains trapped if it's confined to the warehouse, creating a disconnect between rich analytical insights and real-time operational systems.

Today, we're thrilled to announce the launch of the Snowflake Source Connector for Confluent Cloud. This new, fully managed connector bridges the gap between your data warehouse and operational systems, enabling you to stream your Snowflake data to real-time applications and intelligent customer experiences.

With the Snowflake Source connector, you can:

• Democratize warehouse data across your organization for real-time business applications.

• Accelerate time to value with enterprise-ready features and flexible deployment options.

• Eliminate operational overhead through fully managed infrastructure and automatic scaling.

Ready to transform your data warehouse strategy? Continue reading to discover how you can unlock the full potential of your Snowflake data.

From Insight to Action: The Rise of Reverse ETL

For years, the primary flow of data has been into the data warehouse. ETL pipelines have been the workhorse of data analytics, moving data from transactional databases, logs, and software-as-a-service (SaaS) applications into a central store for analysis.

However, a new paradigm has emerged: Reverse ETL.

Reverse ETL closes the loop on your data strategy. Instead of just analyzing what has happened, you can now use the rich, aggregated, and cleansed data in Snowflake to influence what happens next. This approach transforms your data warehouse from a passive reporting repository into an active driver of business operations.

The Snowflake Source connector makes this transformation possible by providing a seamless, real-time bridge between Snowflake and Apache Kafka®, democratizing valuable warehouse data across your organization for immediate operational use.

Snowflake contains some of your organization's most valuable data assets—enriched customer profiles, refined business metrics, and processed analytical insights. However, this data often remains siloed within the warehouse, accessible only to analysts and business intelligence tools.

The Snowflake Source connector transforms your warehouse into a real-time data source for your entire organization. By streaming data from Snowflake to Kafka, you can trigger workflows, personalize customer interactions, and enrich applications with the latest analytical insights, moving beyond historical reporting to build proactive, data-driven systems.

This unlocks high-value use cases across your organization.

Marketing Personalization and Customer 360: Stream enriched customer profiles, purchase history, and engagement metrics to power real-time personalization engines, enabling dynamic website content, personalized email campaigns, and targeted product recommendations.

Real-Time Dashboards and Analytics: Move beyond static, historical dashboards by streaming processed metrics to build real-time analytical views that empower business users with up-to-the-minute insights.

Data Synchronization for Microservices: Use the connector as a central bus for broadcasting data changes from your warehouse to downstream microservices, ensuring application consistency while providing fresh data streams for machine learning models.

The connector supports multiple data capture modes to accommodate different use cases: bulk loading for initial data synchronization, incrementing column capture for immutable data streams, and timestamp-based polling for detecting changes.

Accelerate Time to Value With Enterprise-Ready Features

Getting data from Snowflake into production applications shouldn't require weeks of development and configuration. Our connector is designed for rapid deployment while delivering enterprise-grade capabilities out of the box.

Multiple Data Capture Modes: Whether you need a one-time snapshot, ongoing incremental updates, or timestamp-based change detection, the connector adapts to your specific requirements without custom development.

Flexible Data Formats: Native support for Avro, JSON Schema, and Protobuf ensures seamless integration with your existing data ecosystem and Confluent Schema Registry, maintaining data quality and governance standards.

Secure and Reliable Operations: Built-in authentication mechanisms protect your data while automated error handling and retry logic ensure reliable data delivery, even during network interruptions or temporary service issues.

These enterprise-ready features eliminate the typical integration complexity, allowing you to move from concept to production in days rather than months.

Eliminate Operational Overhead

Managing Kafka Connect infrastructure for data warehouse integrations presents significant challenges. Traditional approaches require dedicated resources for scaling, maintenance, and monitoring—overhead that diverts focus from building valuable applications.

Our fully managed Snowflake Source connector removes these operational burdens entirely. Confluent handles the complete underlying infrastructure, including automatic scaling based on your data volume, seamless updates and patches, and 24/7 monitoring and maintenance. This allows your teams to focus on creating business value rather than managing infrastructure.

The connector employs intelligent polling strategies specifically optimized for Snowflake's characteristics, ensuring efficient resource utilization while maintaining the responsiveness your applications demand. This straightforward JDBC approach significantly lowers the barrier to entry for integrating Snowflake data into Kafka.

Demo: Configuring the Snowflake Source Connector

The Snowflake Source connector is designed for straightforward implementation while delivering powerful capabilities. Our technical demo walks you through a comprehensive setup, but here are some key prerequisites and deployment steps for setting up the connector.

Prerequisites

1. Confluent Cloud Account: An active Standard, Enterprise, or Dedicated cluster. Snowflake Account: Access to the Snowflake account, database, schema, and target table(s).

2. Snowflake User and Role: Create a dedicated Snowflake user and role for the connector. Grant necessary privileges: USAGE on the target database and schema, SELECT on the target table(s), and USAGE on the warehouse the connector will use. Ensure that the user is configured for the chosen authentication method (Password or Key Pair). Refer to Snowflake documentation for detailed instructions on user and role management.

3. Kafka Topic(s): While the connector can automatically create topics using the convention <topic.prefix>.<database>.<schema>.<tableName>, you might want to pre-create topics to customize partition counts or other settings. Auto-created topics typically default to 1 partition and a replication factor of 3.

4. Schema Registry: This must be enabled in your Confluent Cloud environment if you plan to use schema-based message formats such as Avro, JSON Schema, or Protobuf.

5. Network Connectivity: Ensure that Confluent Cloud's managed connector workers can reach your Snowflake instance. This might involve:

   • Allowlisting Confluent Cloud's public egress IP addresses in your Snowflake network policies if using public connectivity.

   • Configuring private networking options such as AWS PrivateLink, Azure Private Link, Google Cloud Private Service Connect, virtual private cloud/virtual network peering, or AWS Transit Gateway if your Snowflake instance is not publicly accessible.

Configuration Steps (Using Confluent Cloud User Interface)

1. Navigate to Connectors: In your Confluent Cloud cluster, go to the "Connectors" section in the left-side menu and click "+ Add connector."

   ‎ 

   ‎ 

2. Select Connector: Search for "Snowflake Source" and select the connector card.

   ‎ 

   ‎ 

3. Kafka Credentials: Choose how the connector authenticates to Kafka. Using a Service Account is recommended for production environments. Alternatively, you can provide an API key and secret directly. Click "Continue."

   ‎ 

   ‎ 

4. Snowflake Connection Details: Provide the necessary information to connect to your Snowflake instance.

   • Snowflake Connection URL: Enter the JDBC URL. The standard format is <account_identifier>.snowflakecomputing.com/?<connection_params>. Replace <account_identifier> with your Snowflake account identifier (e.g., orgname-accountname or the account locator). You can append connection parameters such as db=<database_name>, schema=<schema_name>, warehouse=<warehouse_name>, and role=<role_name> directly to the URL.

   • Snowflake User: Enter the username of the dedicated Snowflake user created earlier.

   • Credentials Source: Select either PRIVATE_KEY or PRIVATE_KEY_PASSPHRASE based on how you configured the Snowflake user.

     • If PRIVATE_KEY: Paste the user's RSA private key (the content between -----BEGIN PRIVATE KEY----- and -----END PRIVATE KEY-----) into the "Private Key" field. If the key is encrypted, enter the passphrase in the "Private Key Passphrase" field.

     • If PRIVATE_KEY_PASSPHRASE: Enter the Private Key Passphrase in addition to above details.

   • Click "Continue."

     ‎ 

     ‎ 

5. Connector Configuration: Configure the core behavior of the connector.

   • Output Messages: Select the desired format for Kafka message keys and values (e.g., JSON_SR, Avro, Protobuf). If using a schema-based format (Avro, JSON_SR, Protobuf), ensure that Schema Registry is enabled and configure any related properties if necessary.

   • Topic Prefix: Define a prefix that will be added to the names of the Kafka topics that are automatically created by the connector.

   • Mode: Select the desired polling mode: bulk, incrementing, timestamp, or timestamp+incrementing.

   • Mode-Specific Columns:

     • If using incrementing or timestamp+incrementing, provide the name of the strictly increasing numeric column in the Incrementing column name.

     • If using timestamp or timestamp+incrementing, provide the name(s) of the timestamp column(s) (comma-separated if multiple) in Timestamp column name(s).

   • Table Selection:

     • Tables: Enter a comma-separated list of tables to poll in the format <database>.<schema_name>.<table_name> (e.g., CONFLUENT.PUBLIC.CUSTOMERS,CONFLUENT.SALES.ORDERS). This uses the table.whitelist parameter internally. 

   • Advanced Configurations:

     • Poll Interval (ms): Set the frequency (in milliseconds) at which the connector polls Snowflake (e.g., 60,000 for 1 minute). Default values should be checked in the UI. Balance desired latency against the load placed on Snowflake.

     • To set other advanced configurations, refer to the connector documentation.

       ‎ 

       ‎ 

6. Sizing: Choose the number of tasks for the connector. For fully managed source connectors like this, it’s often fixed at 1 task ("tasks.max": "1"). Click "Continue."

   ‎ 

   ‎ 

7. Review and Launch: Carefully review the generated JSON configuration. Provide a descriptive name for your connector instance. Click "Launch."

Verification

1. Connector Status: Monitor the connector's status in the Confluent Cloud UI. It should transition from "Provisioning" to "Running" within a few minutes.

2. Logs: Check the connector's "Logs" tab for any error messages or successful polling/batch processing confirmations.

3. Topic Data: Navigate to the "Topics" section, select the target topic(s), and use the "Messages" tab to inspect the data being ingested from Snowflake.

Ready to Activate Your Snowflake Data?

The Snowflake Source connector represents a paradigm shift in how organizations leverage their data warehouse investments. By bridging the gap between analytical insights and operational systems, you can transform static warehouse data into dynamic, real-time business capabilities.

The fully managed nature of this connector removes traditional barriers to implementation, enabling rapid deployment without operational overhead. Combined with flexible polling modes and enterprise-ready features, this solution accelerates your journey from data warehouse to real-time intelligence.

Get started today:

• Deploy the Snowflake Source connector in Confluent Cloud.

• Explore our comprehensive Quick Start guide.

• Discover additional connectors in Confluent Hub.

Transform your Snowflake data into real-time competitive advantage with Confluent Cloud.

Sign up for a free trial of Confluent Cloud to explore the new Snowflake Source connector. New sign-ups receive $400 to spend within Confluent Cloud during their first 30 days. Use the code CL60BLOG for an additional $60 worth of free usage.*

The preceding outlines our general product direction and is not a commitment to deliver any material, code, or functionality. The development, release, timing, and pricing of any features or functionality described may change. Customers should make their purchase decisions based on services, features, and functions that are currently available.

Confluent and associated marks are trademarks or registered trademarks of Confluent, Inc.

Apache®, Apache Kafka®, Kafka®,  Apache Flink®, and Flink® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by the Apache Software Foundation is implied by using these marks. All other trademarks are the property of their respective owners.