Register for Demo | Confluent Terraform Provider, Independent Network Lifecycle Management and more within our Q3’22 launch!

Trials, Tribulations, and Triumphs: Migrating from Self-Managed Kafka to Managed Kafka in the Cloud

Compare the Market has long used Kafka as part of its event data streaming architecture. Whilst Kafka proved a reliable and scalable solution, hosting it in AWS and managing it ourselves came with its own challenges. These challenges included administration of the platform and security, not to mention handling incidents and needing highly skilled talent in the teams to do all of this - and were all contributing factors in driving the decision to migrate.

After much discovery, we landed on ​​Confluent Cloud as our provider of choice to replace our existing Self-Managed Kafka infrastructure. Attend this session to learn about how we went about the migration and issues faced and how this will power our next generation data platform. We will discuss how we overcame the following challenges and more:

  • Identifying a method of replicating topics between platforms
  • Reproducing our existing infrastructure as code with the Confluent Platform and Terraform
  • Coordinating the migration of 100+ consumers and producers, used across the business


Adrian Sibilla

Adrian is an experienced technology professional with a passion for leading great teams and great technology to great outcomes.

Dewi Rees

Passionate about technology and building things with engineering teams.