Project Metamorphosis Month 6: Secure Apache Kafka in Confluent Cloud

The cloud opens up exciting new opportunities for information gathering, analysis, and sharing that can make every organization’s products and services better. Thanks to the cloud and its decentralized nature, organizations can now collect, analyze, and interpret data across their entire footprint and exchange information with each other across the world in real time at an affordable cost. This same openness and ease of use for data ingress, propagation, and syndication also raises significant questions about security and data integrity across the information supply chain. To help address these concerns, we would like to share this month’s Project Metamorphosis theme: Secure.

Strong security requires an approach that is fundamentally defense in depth covering everything from communication, storage, access control, and even audit trails. Fitting all of these pieces together can be time consuming and requires specialized skills in order to execute correctly. Confluent Cloud has always put security as a top priority so that you can feel confident that our cloud service will meet your needs by being reliable, secure, and easy to use.

Securing Kafka

To deliver on this promise, Confluent Cloud requires that all traffic be encrypted in transit using TLS for Apache Kafka^® and SSL for HTTP traffic. Additionally, all data in Confluent Cloud is encrypted at rest. By default, we use cloud provider managed keys, but we understand that some organizations need even more control. To expand our security posture and give you even more control of your data, we now support Bring Your Own Key (BYOK) encryption in AWS to allow you to control the keys for your own data. This capability leverages Amazon KMS to deliver a native cloud service integrated encryption experience.

If network security beyond TLS and SSL are required, Confluent Cloud supports AWS PrivateLink to enable secure access to your streaming service via a private endpoint in your VPC. PrivateLink provides a single directional network link so that traffic cannot originate from outside of your VPC. PrivateLink is considered the most sophisticated cloud network security available today. We plan to bring similar functionality to our Azure and Google Cloud customers.

Encryption in transit and at rest are only part of the story: Authentication is another. Confluent Cloud requires all API calls and user sessions to be authenticated. This is an option for Apache Kafka, but one we enforce and do not allow you to disable. For Kafka, we leverage SASL PLAIN to ensure that only authenticated calls to our service can be made and that those calls happen over a secure transport. Additionally, Confluent Cloud supports Single Sign-On (SSO) so you can use your existing SAML-based identity provider (IdP) to control access to your streaming resources in Confluent Cloud. This makes it possible to have centralized management and control of authentication.

Beyond authentication, there is authorization to control who has access to what resources. Unlike Apache Kafka, which has a cluster as the central resource, in Confluent Cloud you have an organization, which contains environments, which in turn contains clusters. Confluent Cloud will support Role-Based Access Control for management operations so that users can have their privileges scoped to see and interact with only those environments and clusters for which they have been authorized. To help control data access, Apache Kafka and Confluent Cloud support Access Control Lists (ACLs) to lock down access to specific topics.

Finally, Confluent Cloud will feature an Audit Log to track changes being made to clusters, including creation and modification of topics and ACLs. This visibility helps you see what has been changed, when, and by whom.

Looking ahead

We’re excited to bring these core security features to you and look forward to providing deeper integrations and more granular controls and visibility on a regular basis. For more information about compliance and security initiatives in Confluent Cloud, please see the Trust and Security page.

To start using the most secure streaming service in the cloud with $400 of free usage during your first 60 days, sign up for Confluent Cloud. In addition, you can use the promo code CL60BLOG for an additional $60 of free Confluent Cloud usage.*

To learn more about these specific security features, please see the Secure page dedicated to this announcement.