Kafka In the Cloud: Why It’s 10x Better With Confluent | Get free eBook

Developing Custom Transformation in the Kafka Connect to Minimize Data Redundancy

Compacted topics grow over time and are often utilizing high performance, low latency and relatively expensive storage solutions. Reducing duplicated data plays a critical role in the size of compacted topics. with less data on the topics, the Kafka cluster consumes less disk space which in turn it leads to lower operation cost. In this use case-driven talk, we are going to demonstrate how our team at UnitedHealth Group leveraged existing transformers to extract data from the message metadata in the topic as well as how we developed our customized transformers to minimize the amount of duplicated data in each message in the topic.


Siavash Sedghi

Siavash is based in Minneapolis, MN, where he works as a senior software engineer for Optum P360 team. He is an Oracle certified Java programmer with 10 years of experience working with Java eco-system with a focus on APIs, Microservice architectures, and design and architectural patterns. He recently joined the Java Community Process (JCP) as an individual member. Siavash started his career in 2011 in a GIS (Geographic Information System) startup where he was responsible for developing geo-spatial software solutions, deploy them to bare-metal servers and maintain and troubleshoot the application. He later joined The Walt Disney Company in California to develop internal software solutions for ABC Television Group.