Kafka Copy Paste (KCP): How to Migrate to Confluent Cloud in Days, Not Weeks

While Apache Kafka® is incredibly powerful, self-managing brokers, upgrades, capacity, security, and incidents can quickly distract teams from what matters most: building real-time applications and delivering business value.

Confluent Cloud can remove that operational burden, yet migration can still be seen as risky and tedious. That’s why we developed a simpler way for customers to migrate their Kafka workloads to Confluent Cloud using Cluster Linking for data replication and a new, open source tool called Kafka Copy Paste (KCP). This blog post explores how KCP works and how this new approach lets you migrate to Confluent Cloud in days instead of weeks.

The Traditional Apache Kafka® Migration

Kafka migrations of the past often required so much effort that many teams settled on “good enough” Kafka platforms instead of moving to something better.

Teams spend weeks manually inventorying clusters, untangling access control lists (ACLs), schemas, and connectors, and handcrafting complex infrastructure-as-code (IaC). Once this mountain of work is done, they still have to coordinate their risky, late‑night cutovers, for which they must stop, reconfigure, and restart every client.

Modern migrations don’t have to look like this. A typical migration can be broken down into four steps: discovery and planning, infrastructure provisioning, data migration, and client migration.

Confluent offers Cluster Linking, a fully managed, byte-for-byte data replication service that preserves offsets. This eliminates the need to manage replication infrastructure and allows clients to switch over seamlessly, picking up exactly where they left off. Cluster Linking simplifies data movement from any self-managed Kafka environment or hosted Kafka service.

But migration isn’t just about data. What about connectors, ACLs, schemas, and the rest of the migration workflow? How do you make every step simpler? That’s where KCP comes in.

What Is KCP?

KCP is an open source tool built by Confluent to orchestrate the entire migration workflow. KCP coordinates discovery, provisioning, and migration across clusters so you can move to Confluent Cloud in days, not weeks, without manual, error-prone steps.

Today, KCP supports migrations from Amazon Managed Streaming for Apache Kafka (MSK) to Confluent Cloud. Support for migrations from self-managed Kafka is coming soon.

KCP orchestrates cloud-native technologies and services under the hood, so it’s the only tool you need to interact with during migration. Here’s how it works:

1. Discovery and Planning: With no more than five commands, KCP automatically discovers your hosted Kafka clusters across selected Amazon Web Services (AWS) regions and scans each cluster to build a detailed inventory. This includes throughput, estimated costs, networking setup, authentication mechanisms, topics, ACLs, schemas, connectors, and clients, giving you a complete, data-driven view of your existing environment before migration. You can also visualize this data in the KCP user interface (UI) for easier analysis.

2. Provisioning Infrastructure: Once you understand your existing setup, KCP helps map it to the equivalent Confluent Cloud architecture. It generates the required Terraform scripts to provision your Confluent Cloud environment, including networking components such as AWS PrivateLink, so your target infrastructure is ready before any data is moved. This lowers the risk and barrier to entry, letting you set up a secure, production-ready environment without needing deep networking expertise.

3. Data Migration: KCP handles Kafka topic data migrations as well as the migration of ACLs, schemas, and connectors, mapping them to Confluent Cloud resources and automatically generating the Terraform scripts to deploy them. For topic data, it generates Terraform scripts that provision the necessary migration infrastructure that uses Cluster Linking to seamlessly move topic data from your existing clusters to Confluent Cloud.

4. Client Migration: Coming in the first half of 2026, KCP will offer a single command to migrate clients. By leveraging Confluent Cloud Gateway, KCP enables client teams to migrate any group of clients—including their topics, offsets, consumer groups, ACLs, and more—with little to no changes. Simply change the gateway endpoint from your source cluster to your target cluster to cut over all clients simultaneously.

You don’t need to be an expert in Cluster Linking, Confluent Cloud infrastructure, or AWS networking. KCP handles all of that, letting you focus on executing the migration while it takes care of the infrastructure provisioning and orchestration behind the scenes.

How to Execute Your 4-Step Migration Process

Let’s see KCP in action and explore how it orchestrates your migration to Confluent Cloud. We’ll use Amazon MSK as our hosted Kafka environment.

Step 1. Discover your existing setup to plan your migration.

Start by creating a complete, accurate view of your existing Kafka environment. Use KCP to generate a single state file through its discovery process, which will be used to power all remaining migration steps.

To get a full picture of your existing setup, KCP discovers clusters, topics, ACLs, connectors, schemas, and clients, ensuring that no detail is missed before you start migrating.

1a. Discover your Kafka environment.

First, discover your existing setup (clusters and MSK Connect connectors) in one or more regions simultaneously with the following command:

kcp discover --region us-east-2,us-east-1

This command scans all existing clusters in the specified region(s), which is especially useful in environments with multiple teams using Kafka. It provides a high-level view of all available clusters in the different regions and produces:

• kcp-state.json – a shared state file used for reports, the UI, and migration assets. This file serves as the single source of truth for orchestrating the entire migration process across all steps.

• cluster-credentials.yaml – a template for connection details required for deeper Kafka scans. 

1b. Quantify cost and load.

With the kcp-state.json, you can run detailed cost reports.

kcp report costs   --state-file kcp-state.json
kcp report metrics --state-file kcp-state.json

KCP uses the AWS Cost Explorer API and Amazon Cloudwatch APIs to generate markdown files providing a detailed view of source cluster costs, throughput, storage, and sizing—helping you define migration scope and prioritize clusters effectively.

You can also visualize this data in the KCP UI. Run the following command on the machine where KCP is installed and then upload the generated state file to the UI:

kcp ui

1c. Complete the inventory.

Now that we have a high-level understanding of our existing clusters and costs, we can dive deeper to discover topics, ACLs, schemas, clients, and connectors. Using the state and credentials files generated in the discovery step, KCP can enrich your environment inventory across multiple dimensions.

• Clusters: KCP can gather additional cluster-level details, including topics, cluster IDs, ACLs, retention policies, partition counts, and replication factors. Use the following command to enrich your state file with this information; enter your cluster credentials here to enable KCP to perform detailed cluster discovery.

kcp scan clusters \
    --state-file kcp-state.json \
    --credentials-file cluster-credentials.yaml

• Schemas: To discover schemas, KCP can scan your Confluent Schema Registry and capture all schema subjects along with their compatibility settings. It supports both unauthenticated and Basic Auth connections to the Schema Registry. Here is an example of connecting to an unauthenticated local schema registry.

kcp scan schema-registry \
    --state-file kcp-state.json \
    --url http://localhost:8081 ...

• Clients: KCP can inventory all active Kafka clients in your environment, including producers and consumers. It does this by scanning Kafka broker logs stored in Amazon S3 to identify active clients. KCP parses KafkaApi TRACE lines for FETCH and PRODUCE requests, extracting client metadata such as client ID, topic, role, authentication type, and principal.

kcp scan client-inventory \
    --region us-east-1 \
    --s3-uri s3://<MSK_BROKER_LOGS_LOCATION>

Running these three commands enriches the same kcp-state.json file with topics, ACLs, connectors, schemas, and active clients, giving you a single source of truth to plan the rest of your migration.

Step 2. Provision infrastructure for migrated clusters.

Cluster Linking makes migrations simpler by enabling data movement from any hosted Kafka cluster to any Confluent Cloud cluster, regardless of the networking method used. With the newly introduced External Cluster Linking over PrivateLink, you can now connect a private, self-managed Kafka or hosted cluster directly to a private Confluent Cloud cluster over AWS PrivateLink. Virtual private cloud (VPC) peering connections, Transit Gateway, and public cluster migrations are also supported. 

Previously, migrations from private Kafka clusters to a private Confluent Cloud cluster backed by PrivateLink required creating a jump cluster and performing a two-step data transfer. With external Cluster Linking over PrivateLink, you can connect both clusters directly using reverse PrivateLink. This involves deploying a PrivateLink endpoint service in front of your Kafka cluster and creating Egress Endpoints in Confluent Cloud that have access to this service.

Establishing the Cluster Link, configuring PrivateLink, and creating Egress Endpoints can be tedious and complex if done manually. Luckily, KCP automates the provisioning of all infrastructure with Terraform, including the networking components and endpoint configuration so that you can focus on the migration itself.

With a clear picture of your existing setup, you can now use KCP to provision the target Confluent Cloud environment and supporting infrastructure. At a high level, you:

• Provision Confluent Cloud resources, including destination cluster(s), networking (PrivateLink), and access control

• Stand up migration infrastructure, including bastion hosts, reverse proxies, and Cluster Linking plumbing where needed

With just two commands, KCP can generate the IaC templates for both your target Confluent Cloud environment and migration infrastructure, allowing you to provision consistently instead of handcrafting resources. You choose the networking pattern that matches your source cluster and Confluent Cloud setup (i.e., public or PrivateLink); KCP emits the right building blocks for the supported combinations.

Target Infrastructure

To generate the target infrastructure Terraform script, run the create-asset target-infra  command:

kcp create-asset target-infra \
--state-file kcp-state.json \
--cluster-arn arn:aws:kafka:us-east-1:XXX:cluster/XXX/1a2345b6-bf9f-4670-b13b-710985f5645d-5 \
--needs-environment true \
--env-name example-environment \
--needs-cluster true \
--cluster-name example-cluster \
--cluster-type enterprise \
--needs-private-link true \
--subnet-cidrs 10.0.0.0/16,10.0.1.0/16,10.0.2.0/16 \
--output-dir confluent-cloud-infrastructure

This command takes the kcp-state.json file and the Amazon Resource Name (ARN) of the existing cluster you want to migrate and then generates Terraform scripts in the confluent-cloud-infrastructure directory. These scripts provision a Confluent Cloud environment and an Enterprise cluster, forming the foundation of your target infrastructure.

You can either use an existing network connection or provision PrivateLink as part of this step. In most cases, creating a new PrivateLink connection is recommended. KCP will configure the required VPC endpoints in the same VPC as your existing cluster; you simply need to provide the subnet CIDRs where those endpoints should be created.

Once you have the Terraform script, deploy them by running the following command from the confluent-cloud-infrastructure directory:

terraform apply

Migration Infrastructure

Now that we have target infrastructure in place, we can use KCP to deploy the migration infrastructure.  This includes Cluster Linking and AWS resources needed for external Cluster Linking over PrivateLink to work. 

To generate the migration infrastructure Terraform script, run:

kcp create-asset migration-infra /
--state-file kcp-state.json /
--cluster-arn arn:aws:kafka:us-east-1:XXX:cluster/XXX/1a2345b6-bf9f-4670-b13b-710985f5645d-5 /
--existing-internet-gateway /
--existing-private-link /
--output-dir migration-infrastructure  /
--type 2 /
--cluster-link-name migration-link /
--target-environment-id <TARGET_CONFLUENT_ENVIRONMENT> /
--target-cluster-id <TARGET_CONFLUENT_CLUSTER_ID> /
--target-rest-endpoint https://<TARGET_CONFLUENT_CLUSTER_ID>.<AWS_REGION>.aws.private.confluent.cloud:443 /
--target-bootstrap-endpoint <TARGET_CONFLUENT_CLUSTER_ID>.<AWS_REGION>.aws.private.confluent.cloud:9092

This command uses Confluent Cloud cluster details generated by the target-infra command, along with the MSK cluster ARN, to create Terraform scripts that provision a cluster link named migration-link.

KCP supports multiple migration patterns to accommodate different networking and authentication configurations:

• MSK with SASL/SCRAM authentication (public or private endpoints). If you don’t have SASL/SCRAM configured, it’s simple to add a SASL/SCRAM listener exclusively for Cluster Linking traffic.

• MSK with identity and access management (IAM) authentication using a jump cluster pattern

The example above illustrates a direct cluster link approach with a private MSK cluster with SASL/SCRAM authentication. If SASL is not an option, you’ll need to go down the jump cluster path. Here KCP automates the jump cluster infrastructure as well.

For more information on how this works, see KCP documentation.

Step 3. Migrate data from hosted Kafka to Confluent Cloud.

With the target and migration infrastructure in place, you can now migrate configuration and data: ACLs, connectors, schemas, and topics. KCP’s pattern is the same each time: Generate assets and then apply them in a controlled way.

Migrating IAM-Based Permissions and ACLs

Using the discovery data collected earlier, KCP migrates both IAM-based permissions and Kafka ACLs from your source cluster to Confluent Cloud. IAM principals are extracted from the previously captured client inventory, while Kafka ACLs are mapped directly from the cluster-level metadata.

KCP reuses the same kcp-state.json state file to translate these existing permissions into their Confluent Cloud equivalents, automatically generating the required Terraform assets.

# IAM → Confluent Cloud ACLs
kcp create-asset migrate-acls iam \
  --user-arn arn:aws:iam::1234:user/kcp-demo-iam-kafka-user \
  --target-cluster-id <TARGET_CONFLUENT_CLUSTER_ID> \
  --target-rest-endpoint https://<TARGET_CONFLUENT_CLUSTER_ID>.<AWS_REGION>.aws.private.confluent.cloud:443

# Kafka ACLs → Confluent Cloud ACLs
kcp create-asset migrate-acls kafka \
  --state-file kcp-state.json \
  --cluster-arn arn:aws:kafka:eu-west-3:1234:cluster/xxxx/1234 \
  --target-cluster-id <TARGET_CONFLUENT_CLUSTER_ID> \
  --target-rest-endpoint https://<TARGET_CONFLUENT_CLUSTER_ID>.<AWS_REGION>.aws.private.confluent.cloud:443

You can either apply all generated ACLs or selectively pick principals via the KCP UI.

Migrating Kafka Connectors 

KCP discovers both MSK Connect and self‑managed connectors and then helps you move them to fully managed Confluent Cloud connectors. KCP doesn’t migrate the actual connectors; it generates the connector configs in JSON so that you can use a tool of your choice to migrate the connectors.

The recommended tool is the Connect Migration Utility, a free, open source, CLI-based tool available on GitHub. It provides a guided, end-to-end migration experience from MSK Connect or self-managed connectors to fully managed Confluent Cloud connectors. You start paying for managed connectors once cutover is complete, giving you a low-risk path to migration.

To generate the JSON configs, run the following:

kcp create-asset migrate-connectors connector-utility \
  --state-file kcp-state.json

This command produces connector configurations in JSON format, which can then be consumed by the Connect Migration Utility. The utility:

• Translates configurations into their fully managed Confluent Cloud connector equivalents

• Handles v1 to v2 connector class changes

• Preserves connector offsets to ensure a smooth transition

If you prefer an IaC approach, KCP can also generate per-connector Terraform modules for both MSK Connect and self-managed connectors, allowing you to apply them using your existing IaC workflows.

Replicating and Exporting Schemas

After the discovery step, KCP can migrate schemas by generating Terraform assets that configure schema exporters to replicate from your source Schema Registry into Confluent Cloud.

Generate the migration assets with:

kcp create-asset migrate-schemas \
  --state-file kcp-state.json \
  --url http://localhost:8081

KCP creates a migrate_schemas/ folder containing Terraform for an exporter that continuously syncs all schemas to Confluent Cloud, preserving contexts and compatibility settings as they exist in the source registry.

Moving Topic Data to Confluent Cloud With Cluster Linking

Topic migration is powered by Cluster Linking, with KCP generating the Terraform scripts and supporting the configuration needed for your specific topology.

KCP automates the setup of all necessary infrastructure, including Cluster Linking plumbing so that you can move topic data from MSK to Confluent Cloud seamlessly and reliably. We’ve already created the target and migration infrastructure; now we start moving data by running:

kcp create-asset migrate-topics \
  --state-file kcp-state.json \
  --cluster-arn <MSK_CLUSTER_ARN> \
  --migration-infra-folder migration_infra

This command generates Terraform scripts in the migration_infra/ folder that:

• Creates cluster links between your existing cluster and Confluent Cloud

• Mirrors topics, keeping your data in sync—including consumer group offsets—until you’re ready to cut over

Step 4. Migrate clients.

Client migrations are often the most complicated part of the migration process, not because of technical complexity but because of the cross-team coordination required to plan and execute client cutovers.

There are two common approaches to migrating clients today, each with their own trade-offs. 

• For a seamless and predictable client cutover experience, users can leverage Cluster Linking for the client migration. This approach preserves offsets but requires a small amount of client downtime to perform the cutover.

• Users who need to perform a zero-downtime migration can instead dual-produce to both the source and target clusters. This requires more involved management of offsets during migration.

Today, KCP can assist with client migrations by scanning logs and creating an inventory of active clients using the kcp scan client-inventory command. However, much of the client migration process is still manual.

In the first half of 2026, KCP will greatly simplify this step with Confluent Cloud Gateway compatibility. With Confluent Cloud Gateway support in KCP, you’ll be able to connect all clients to the Gateway endpoint and then cut over all clients at once by directing the Gateway to your target cluster.

Start Planning Your Migration to Confluent Cloud With KCP and Cluster Linking

KCP transforms what is often a complex and manual migration process into a streamlined, automated workflow. By orchestrating steps like inventory discovery and cost assessment, infrastructure provisioning, and data and client migration, KCP both accelerates migration timelines and reduces the work and complexity involved.

It’s never been easier to move to Confluent Cloud. Get started today with KCP by exploring the documentation, visiting the GitHub repo, and trying out our migration workshop to see the tool in action. Once you’ve taken those first steps, Confluent can help you even more: To receive a granular cost breakdown, simply navigate to the Migration hub in Confluent Cloud, upload your kcp-state.json file, and select “Request cost estimate” to have a Confluent representative reach out with a detailed analysis.

If you haven’t done so already, sign up for a free trial of Confluent Cloud. New sign-ups receive $400 to spend within Confluent Cloud during their first 30 days. Use the code CLOUDBLOG60 for an additional $60 worth of free usage.

Apache®, Apache Kafka®, Kafka®, and the Kafka logo are registered trademarks of the Apache Software Foundation. No endorsement by the Apache Software Foundation is implied by the use of these marks.