Manage Consistent Configurations Across Multiple Kafka Environments

Conventional Kafka developers are liable to perform monotonous functions like creating topics, managing topic configurations, granting access for producers/consumers, topic sizing, capacity planning and other day-2 operations. At Fidelity Investments we support hundreds of applications publishing and subscribing to tens of thousands of event streams across business units. Changes to applications are constant. Expecting developers to manually repeat commands across environments slows development and increases the chance for misconfigurations, ultimately degrading the quality of work.

In this talk, we will discuss how Fidelity Investments modeled a unique API to seamlessly lift and shift application topics, ACLs, quotas, and every other entity from lower environment to higher environment clusters, thus resulting in the propagation of exact configurations across environments with zero human errors. We will also highlight the salient features of this in-house REST API in providing options to scale differently in higher environments. Flexibility of modifying users across environments while preserving complete Kafka cluster configuration is an added bonus. This entire process empowers Kafka applications teams to promote version-controlled releases across multiple environments with just a single click and a safety net to rollback to older configurations.