Tableflow: Turn Kafka Topics into Iceberg Tables

TL;DR: Tableflow is a Confluent Cloud feature that materializes Apache Kafka topics as Apache Iceberg or Delta Lake tables, eliminating custom data pipelines by automatically handling schematization, type conversions, schema evolution, CDC stream materialization, catalog publishing, and table maintenance.

What Is Tableflow?

Tableflow is a managed feature in Confluent Cloud that continuously converts Kafka topics into open table formats (Apache Iceberg or Delta Lake) so analytics engines can query streaming data as tables—without writing or maintaining ETL pipelines. The core problem it solves is familiar: operational data lives in Apache Kafka, while analysts and BI tools expect data in data lakes or warehouses. Traditionally, teams bridge this gap with custom connectors, scheduled ETL jobs, and schema handholding—an approach that is brittle under schema evolution and expensive to operate at scale.

How Tableflow Works

Tableflow uses Confluent’s Kora storage layer to transform Kafka topic data in Avro, JSON Schema, or Protobuf into structured Parquet files, then generates Iceberg metadata or Delta Lake metadata and exposes those tables through the built-in Iceberg REST Catalog or supported external catalog integrations. Schema Registry is the source of truth for schema mapping and schema evolution.

Step-by-step:

1. Kafka topic ingestion — Tableflow consumes committed Kafka segments.

2. Kora storage layer — Segments are staged and prepared for columnar conversion.

3. Parquet conversion — Records are written as Parquet files optimized for analytics.

4. Table metadata — Iceberg (GA) or Delta Lake (GA) metadata is generated.

5. Catalog publish — Tables are registered in an Iceberg REST catalog (or external catalogs).

6. Schema governance — Schema Registry governs compatibility and evolution.

Supported table formats:

• Apache Iceberg (GA)

• Delta Lake (GA)

Architecture Flow 

What Can You Query Tableflow Tables With?

You can query Tableflow-materialized tables using common analytics engines and warehouses, including:

• Snowflake

• Databricks

• AWS Athena

• Amazon Redshift

• Trino

• BigQuery

Catalog options:

• Built-in Iceberg REST catalog (managed)

• AWS Glue

• Unity Catalog

Tutorial: Enable Tableflow on a Kafka Topic

Prerequisites

1. A Confluent Cloud account

2. An AWS account with:

• An Amazon S3 bucket for Tableflow storage

• Permissions to use AWS Glue and Amazon Athena

3. A Kafka cluster, topic, and schema (created in the steps below)

Note: This tutorial explicitly uses Amazon S3 as the storage layer. Managed storage and other cloud providers (GCS / ADLS) are supported but not covered here.

Step 0 — Create a Kafka Cluster in Confluent Cloud

1. Log in to Confluent Cloud

2. Click Environments and click “Add cloud environment” 

3. Then click that environment to create a cluster

4. Click Create Cluster

5. Choose AWS as the cloud provider

6. Select a region close to your S3 bucket (for example: us-east-1)

7. Choose Standard cluster type

8. Click Launch Cluster

Step 1 — Create a Kafka Topic and Register a Schema

Create a Topic

1. Open your cluster → Topics

2. Click Create Topic

3. Enter a topic name (for example: orders)

4. Choose the number of partitions (default is fine)

5. Click Create

Register a Schema

1. Once the topic is created, create data contract

2. Click message values and click Add data contract for selected topic (orders-value)

3. Register an Avro / JSON / Protobuf schema. For example, see the following Avro schema:

Once the schema is registered, produce a few test records to the topic.

4. Click Actions -> Produce new message

5. Insert the record as shown in figure and click produce

6. You should see the inserted record in the Messages tab of your topic.

Step 2: Configure your Amazon S3 Bucket and Provider Integration

We need an Amazon S3 Bucket to write our Apache Iceberg data to. 

1. If you already have an Amazon S3 bucket, feel free to skip to the next section

2. In a new tab, navigate to the Amazon S3 console or click here

3. Click on Create Bucket on the top right.

4. Leave everything default and give your bucket a name. The name of the bucket must be unique so let’s name it tableflow-s3-bucket.

5. Leave the rest of the options as default, scroll down and click Create Bucket. 

Step 3: Configure Storage Provider

1. Back on Confluent Cloud, navigate to the Tableflow page by clicking on Tableflow on the left-hand side of the menu bar within your “Confluent Environment —-> Kafka Cluster”.

2. In the center of the page, you’ll see an option to configure a Storage Connection. Click on Go to Provider Integrations in order to configure the connection to Amazon S3.

3. From Provider Integrations, click on  + Add Integration.

4. You can either create or use an existing role. For the sake of this example, we will create a new role. Click Continue.

5. On the Create Permission Policy in AWS page, select Tableflow S3 Bucket in the dropdown.

6. You will need to copy this permissions-policy.json for use in the next step

Step 4 : Create a new permissions policy on AWS

We need to create a permissions policy for Amazon S3 Access. In this section, we will create a permissions policy to be later used with an AWS Identity Access Management (IAM) Role.

1. In a new tab, open your Amazon Web Services (AWS) account and navigate to IAM Policies or click this link.

2. Click Create Policy

3. From the Create Policy screen, switch to the JSON editor

4. Copy the permissions-policy.json from the Confluent Provider Create Wizard into your policy editor on AWS. Ensure to replace the values for the Amazon S3 bucket we created above (-).

// Example structure from the document - ensure your bucket name is correct

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::<<Your S3 Bucket Name>>" // Replace with your S3 bucket
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectTagging", // May differ slightly based on wizard version
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::__<<Your S3 Bucket Name>>/*" // Replace with your S3 bucket__
            ]
        }
    ]
}

5. Click Next

6. Give the policy the name tableflow-s3-access-policy and click Create Policy.

7. Click Create Policy.

8. Once the policy has been created, you can click Continue on Confluent Cloud.

The next page, Create role in AWS and map to Confluent will present a trust-policy.json that we will leverage for the next section of these instructions. Copy this down for later use.

Step 5 : Create role in AWS and map to Confluent

1. Next, we are going to create a role in AWS that will leverage the policy we just created, plus a trust-policy.json to allow Confluent to assume this role. 

2. Click here or navigate to the Roles sub-menu under the IAM service on AWS

3. Once on the Roles sub-menu, on the top right of the screen, click Create role in order to begin creating the role for Tableflow to assume.

4. Select Custom Trust Policy and paste in the trust-policy.json from Confluent.

5. Scroll down and click Next

6. In the next page, Add Permissions, search for the policy you created in the previous step, named tableflow-s3-access-policy and attach it to the role. Click Next.

7. Give the role a name, s3-tableflow-assume-role, scroll down and click Create Role

8. Once the role is created, you should see a green banner at the top of the console stating that the role was created. Click View Role to see the role details.

9. Copy the AWS ARN of the role and paste it in the Mapping component on the Provider Integration wizard on Confluent Cloud.

10. Give the Provider integration the name tableflow-s3-integration and click Continue.

11. You will now copy the updated Trust Policy from Confluent which contains the Confluent External ID and role. Copy the trust-policy.json from Confluent to your clipboard.

12. Go back to the AWS IAM Role you created (e.g., s3-tableflow-assume-role).

13. Select the Trust relationships tab.

14. Click Edit trust policy (or Edit trust relationship).

15. Replace the entire existing JSON with the updated trust-policy.json you copied from Confluent Cloud in the previous step. This adds the necessary External ID condition.

16. Click Update policy (or Save changes).

17. Return to the Confluent Cloud wizard one last time and click Continue (or Finish/Create).

Step 6 : Enable Tableflow on Your Kafka Topics

With the Provider Integration successfully configured, you can now enable Tableflow for your desired Kafka topics.

1. In your Confluent Cloud console, go to your Environment, then select your Kafka Cluster.

2. In the left-hand navigation menu for your cluster, click on Topics. You should see a list of your topics.

3. Find the specific topic you want to enable Tableflow for in the list.

4. On the right-hand side of the row for that topic, in the "Tableflow" column, click the Enable Tableflow button/link.

5. You will be asked to choose storage. Select Configure custom storage. Note: In this tutorial, we explicitly use Apache Iceberg as the table format (not Delta Lake), with Amazon S3 as the storage layer.

6. In the next menu, you will be able to choose the Provider Integration we created in the previous section. You can identify it by either the name of the provider integration or the IAM Role you created.

7. Provide the AWS S3 bucket name (tableflow-s3-bucket)

8. In the next screen, review the details and then click Launch.

9. You will now see Tableflow sync pending at the top of your topic information. This should transition to the Syncing status shortly.

10. You can verify it by navigating inside s3 bucket

11. Go inside your environment folder and then your cluster folder, you’ll see data folder which contains the kafka topics data in parquet format.

Schema Evolution and Ongoing Maintenance

Tableflow follows Confluent Schema Registry compatibility rules and automatically applies compatible schema changes during materialization. Supported changes include adding new nullable columns with a default of null, removing columns where compatibility rules allow it, and widening supported types such as int to long, float to double, and increasing decimal precision. If a schema change is not compatible with the configured compatibility mode, manual intervention is required, and Tableflow can suspend materialization for the affected topic.

What to Do After Your Kafka Data Is in Amazon S3

Now that Tableflow has materialized your Kafka topic into Apache Iceberg tables stored in Amazon S3, here are the next steps—kept short and practical:

1. Verify data in Amazon S3

   • Confirm that Parquet data files and Iceberg metadata are being written to your S3 bucket.

   • Ensure new Kafka records continue to appear as new files over time.

2. Validate table structure

   • Check that the Iceberg table reflects the expected schema and partitions.

   • Make a small, compatible schema change in Schema Registry to confirm automatic propagation.

3. Prepare for analytics access (next step)

   • Register the Iceberg tables in a metastore such as AWS Glue or another Iceberg-compatible catalog.

   • This enables SQL-based access from engines like Amazon Athena or Trino.

4. Plan consumption patterns

   • Decide how downstream teams will query or share the data (batch analytics, BI, ad-hoc exploration).

   • Use Iceberg features such as snapshot isolation and time travel for safer analytics.

5. Operational checks

   • Monitor Tableflow status in Confluent Cloud.

   • Apply S3 IAM policies and encryption settings for secure access.

Bottom line: Kafka remains your real-time system of record, while Iceberg tables on Amazon S3 become your analytics-ready foundation—kept continuously in sync by Tableflow, with catalogs and query engines added as the next layer when needed.

What’s Next?

• Tableflow + Snowflake: Hands-on tutorial for querying Iceberg tables from Snowflake

• Tableflow + Flink: Streaming ETL with Apache Flink into Iceberg

• Product page: Confluent Tableflow

• Try it free: Confluent Cloud Free

FAQ

What is Confluent Tableflow? Tableflow is a Confluent Cloud feature that automatically converts Kafka topics into Iceberg or Delta Lake tables without ETL code.

How do I convert Kafka topics to Iceberg tables? Enable Tableflow on a topic in Confluent Cloud, choose Iceberg, and query the resulting table via the REST catalog.

Can I query Kafka data from Snowflake, Databricks, or Athena? Yes. Tableflow publishes Iceberg/Delta tables that these engines can query.

What is zero-ETL for Kafka? Zero-ETL means no custom pipelines or jobs—Tableflow manages ingestion, conversion, metadata, and cataloging.

How does Tableflow handle schema evolution? Schema Registry governs compatibility; Tableflow propagates compatible changes automatically.