Knock Knock, Who’s There?

When managing multi-tenant Kafka clusters, the ability to identify individual clients is crucial but can be challenging to achieve.

Previously at Shopify, a single SSL certificate was used by nearly all clients to connect to our Kafka clusters. As Kafka distinguishes users based on their certificate’s subject, all clients were masked as the same user, and thus we were unable to identify who was connecting and what operations they were enacting. As a result, ensuring proper data usage and ownership was not possible, as the producers and consumers of each Kafka topic were unknown. Without this insight, protective measures such as request quotas and access-control were rendered impractical.

We set out to provide our thousands of Kafka clients with their own unique identity, by automating certificate management within our Kubernetes platform via controllers. In providing our clients with their own identities, we subsequently improved observability around data usage/ownership by implementing a custom Kafka authorizer to gather client request metrics. Lastly, we enhanced our Kafka infrastructure’s resiliency by introducing request quotas and protected our clusters from unintended operations with user access-control.