Live demo: Kafka streaming in 10 minutes on Confluent | Register now

Simplifying Apache Kafka Multi-Cluster Management Using Control Center and Cluster Registry

Most companies who have adopted event streaming are running multiple Apache Kafka® environments. For example, they may use different Kafka clusters for testing vs. production or for different use cases. Self-managing multiple Kafka clusters, however, can pose a challenge to system administrators.

To help Kafka administrators, we introduced the Cluster Registry in Confluent Platform 6.0, through which you can centrally register Kafka clusters in the Confluent Platform Metadata Service (MDS). MDS manages important information, such as authorization data about Confluent Platform environments, as we will explain in more detail below.

Confluent Platform also comes packaged with Confluent Control Center to manage and monitor multiple Kafka clusters. Often the set of clusters managed by Control Center is the same as the set registered with Cluster Registry in MDS. Because Cluster Registry and Control Center are essentially storing the same information, this leads to the following drawbacks:

  • Duplicate work: User must register Kafka clusters in both tools
  • Error prone: Different places mean that cluster name could very well end up being different for the same cluster

In Confluent Platform 6.0.1, we have added support in Control Center for reading the cluster information from Cluster Registry to avoid the above pitfalls.

Note
If self-management is not a requirement, Confluent Cloud is recommended. In this cloud-native and fully managed service, infrastructure management and operational tasks such as cluster scaling and upgrades are taken care of for you. You can sign up for Confluent Cloud and get $200 of free usage each month for your first three months, plus an additional $60 of free usage when you use the promo code CL60BLOG.*

What is Control Center?

Confluent Control Center is a web-based tool for managing and monitoring Apache Kafka as well as Confluent software components, such as Kafka connectors, Schema Registry, and ksqlDB. Control Center provides a user interface that allows developers and operators to get a quick overview of cluster health; to observe and control messages, topics, and Schema Registry; and to develop and run ksqlDB queries.

Control Center overview

Manage and monitor multiple clusters

You can use Control Center to manage and monitor not just one but multiple Kafka clusters. To do so, each Kafka cluster must be specified in the Control Center configuration file. In the example below, we specify a cluster named lon-prod for our London production traffic:

confluent.controlcenter.kafka.lon-prod.bootstrap.servers=localhost:9192

The above cluster in turn will appear in Control Center UI as shown below:

Managing multiple clusters in Control Center

Please refer to the documentation on multi-cluster configuration for details.

Prior to Confluent Platform 6.0.1, the aforementioned approach was the only option to configure multiple Kafka clusters for use with Confluent Control Center. In the following sections, we explain how the latest release makes multi-cluster management more convenient through the Cluster Registry.

What is Cluster Registry?

Built on the top of Confluent Platform Metadata Service (MDS), Cluster Registry enables you to keep track of which clusters you have installed. For those who are not familiar with MDS, the Confluent Platform Metadata Service manages a variety of metadata about your Confluent Platform installation. Specifically, the MDS:

  • Hosts the Cluster Registry that enables you to keep track of which clusters you have installed
  • Serves as the system of record for cross-cluster authorization data
  • Provides a convenient way to manage audit log configurations across multiple clusters

A prerequisite for using Cluster Registry is enabling RBAC first. RBAC, aka Role-Based Access Control, is a method for controlling system access based on roles assigned to users within an organization. RBAC leverages the Confluent Platform Metadata Service to configure and manage its implementation from a centralized configuration context.

Register clusters

You can use either curl commands or the Confluent Platform CLI to register clusters. When registering a Kafka cluster in the Cluster Registry, you must specify the following information:

  • Cluster name: The name of the Kafka cluster
  • Cluster ID: The ID of the Kafka cluster
  • Host name and port number: The connection information for the cluster
  • Protocol used by the hosts and ports: The protocol should be SASL_SSL for Kafka clusters (or SASL_PLAINTEXT for non-production Kafka clusters) and HTTP or HTTPS for Kafka Connect, ksqlDB, and Schema Registry clusters

Register clusters using the CLI

To register a new Kafka cluster using the CLI:

  1. Log in to the MDS cluster first:
    confluent login --url <protocol>://<host>:<port>
  2. Register the new Kafka cluster by running the following command:
    confluent cluster register \
      --cluster-name <exampleKafka> \
      --kafka-cluster-id <kafka-ID> \
      --hosts <10.6.6.6:9000,10.3.3.3:9003> \
      --protocol SASL_PLAINTEXT
    
  3. Register a new Connect cluster associated with a new Kafka cluster:
    confluent cluster register \
      --cluster-name <exampleConnect> \
      --kafka-cluster-id <kafka-ID> \
      --connect-cluster-id <connect-ID> \
      --hosts <10.6.6.7:8443> \
      --protocol HTTP
    

Note: Substitute the values in < > with your information

For a complete guide, please refer to the Cluster Registry documentation.

Control Center + Cluster Registry

As mentioned in the introduction, in Confluent Platform 6.0.1, we added support in Control Center to read cluster information from Cluster Registry. This means that Control Center can use Cluster Registry as the primary source of cluster information over its configuration properties file.

The diagram below shows how Control Center discovers the cluster information when Cluster Registry is enabled and disabled.

Control Center and Cluster Registry

 

Enable Cluster Registry in Control Center

To enable reading cluster information from Cluster Registry, you can follow these steps:

  1. First, add the following configuration in Control Center’s properties file: confluent.metadata.cluster.registry.enable=true
  2. Second, add RBAC settings to the control-center.properties file as explained in this guide

After performing #1 and #2, start Control Center or reboot if it’s already running. Once started, Control Center should read the cluster information from Cluster Registry.

A few things to note:

  • When Cluster Registry is enabled, there is no need to specify Kafka settings in Confluent Control Center’s configuration files. It is only needed if we need to specify extra properties for communicating with the Kafka cluster (e.g., ssl.* settings for encrypted communication).
  • Control Center’s bootstrap cluster information still needs to be configured in the properties file. We cannot tell which cluster is the bootstrap cluster from Cluster Registry alone.

Migration scripts

To make it easier to migrate cluster information from Control Center to Cluster Registry, we have added the scripts below to Confluent Platform.

Control-center-export

You can use this script to export the existing cluster information from Control Center:

control-center-export --cluster control-center.properties --outfile exported-cluster-info.json

The exported data looks like this:

[
  {
    "clusterName": "lon-prod",
    "scope": {
      "path": [],
      "clusters": {
        "kafka-cluster": "jRCEm14eR_SadwCWqhXV-w "
      }
    },
    "hosts": [
      {
        "host": "localhost",
        "port": 9193
      }
    ],
    "protocol": "SASL_PLAINTEXT"
  }
]

Cluster-information-migration-script

This script provides a way to import or export data off Cluster Registry. Here is how you can import cluster information into Cluster Registry:

cluster-information-migration-script -i --url http://mds:8090 -f exported-cluster-info.json

To export cluster information from Cluster Registry, you would perform a similar action:

cluster-information-migration-script -e --url http://mds:8090 -f cluster-info-to-export.json

Update cluster name from Control Center

Control Center has always allowed you to update a Kafka cluster’s name via a UI as shown below.

Updating a cluster name

This is true when Cluster Registry is enabled as well. Control Center can not only read the cluster information from Cluster Registry, but it also allows you to edit the cluster name. The change is persisted in Cluster Registry.

For example, if the user changed the name from lon.prod to say nyc.prod, the result of the confluent cluster list command would be this:

> confluent cluster list
>  Name           |      Type       |  Kafka ID               | Component ID |     Hosts        |    Protocol
>+----------------+-----------------+-----------+--------------+------------------------------+----------------+
>  nyc.prod       | kafka-cluster   | jRCEm14eR_SadwCWqhXV-w  |              | localhost:9192   | SASL_PLAINTEXT

Conclusion

In summary, we encourage you to take advantage of Cluster Registry as the source of cluster information for Control Center. This is just the first step. We expect to add more to the integration story in the future, so stay tuned.

If you haven’t already but would like to get started with Cluster Registry today, you can download the Confluent Platform.

Download Now

Shailesh Panwar joined Confluent in 2019, where he is responsible for Confluent Control Center, the Monitoring and Management product for Confluent Platform. For the last 10+ years, Shailesh has worked in the cloud and consumer productivity space, building cloud Infrastructure products and collaboration suites such as Microsoft Team. His interests lie in distributed computing platforms and software development.

Did you like this blog post? Share it now

Subscribe to the Confluent blog

More Articles Like This

How to Better Manage Apache Kafka by Removing Residue Data with Control Center Cleanup Script

This blog post is the fourth in a four-part series that discusses a few new Confluent Control Center features that are introduced with Confluent Platform 6.2.0. It focuses on removing

How to Better Manage Apache Kafka with Improved Topic Inspection via Last-Produced Timestamp

This blog post is the third in a four-part series that discusses a few new Confluent Control Center features that are introduced with Confluent Platform 6.2.0. It focuses on inspecting

How to Better Manage Apache Kafka by Exporting Kafka Messages via Control Center

This blog post is the second in a four-part series that discusses a few new Confluent Control Center features that are introduced with Confluent Platform 6.2.0. This blog post focuses