Level Up Your Kafka Skills in Just 5 Days | Join Season of Streaming
In the latest major version update of the Confluent CLI, we’ve packed all of the functionality from our cloud-based ccloud CLI into the existing confluent CLI client! This is a pivotal step for Confluent as we move towards providing a single platform that serves cloud and on-prem users simultaneously. The newest Confluent Platform release, Confluent Platform 7.1, ships with this major version update, Confluent CLI v2.
Cloud users who update from ccloud v1.x to confluent v2.x will receive multi-org support and helpful command syntax changes. On-prem users will receive similar usability improvements and unlock all of the functionality of the Cloud CLI, making it easier than ever to migrate infrastructure to the Cloud. Most importantly, hybrid users who need to frequently switch between cloud and on-prem deployments can now use a single CLI for any Confluent task. The complete list of changes can be found in the documentation.
In this blog post, we’ll give a quick breakdown of the key features included in the new CLI and demonstrate an example workflow for a hybrid user who needs to link two clusters across Confluent Platform and Confluent Cloud and mirror a topic between them.
To get your hands on the new CLI, download the unified CLI client as part of Confluent Platform 7.1. Alternatively, if you are already a Confluent Cloud user, simply perform a major version update with ccloud update --major, or follow the installation instructions in the documentation.
Users of the new unified CLI will find that its interface hasn’t changed noticeably, with the most obvious exception that, for Cloud users, the name has changed from ccloud to confluent.
Logging in is exactly the same as before for all users: confluent login authenticates against Confluent Cloud by default, but if a Metadata Service (MDS) URL is provided with the --url flag, the user can log in to Confluent Platform.
Confluent Cloud | Confluent Platform |
confluent login |
confluent login --url https://example.com:8090 |
After logging in, a user will have access only to the commands that pertain to them. For instance, a Confluent Cloud user will see:
admin Perform administrative tasks for the current organization. api-key Manage the API keys. audit-log Manage audit log configuration. cloud-signup Sign up for Confluent Cloud. completion Print shell completion code. connect Manage Kafka Connect. context Manage CLI configuration contexts. environment Manage and select Confluent Cloud environments. help Help about any command iam Manage RBAC and IAM permissions. kafka Manage Apache Kafka. ksql Manage ksqlDB. local Manage a local Confluent Platform development environment. login Log in to Confluent Cloud or Confluent Platform. logout Log out of Confluent Cloud. price See Confluent Cloud pricing information. prompt Add Confluent CLI context to your terminal prompt. schema-registry Manage Schema Registry. shell Start an interactive shell. update Update the Confluent CLI. version Show version of the Confluent CLI.
A Confluent Platform user will see a slightly different list of commands:
audit-log Manage audit log configuration. cloud-signup Sign up for Confluent Cloud. cluster Retrieve metadata about Confluent Platform clusters. completion Print shell completion code. connect Manage Kafka Connect. context Manage CLI configuration contexts. help Help about any command iam Manage RBAC, ACL and IAM permissions. kafka Manage Apache Kafka. ksql Manage ksqlDB. local Manage a local Confluent Platform development environment. login Log in to Confluent Cloud or Confluent Platform. logout Log out of Confluent Platform. prompt Add Confluent CLI context to your terminal prompt. schema-registry Manage Schema Registry. secret Manage secrets for Confluent Platform. shell Start an interactive shell. update Update the Confluent CLI. version Show version of the Confluent CLI.
A context represents a user’s login state, whether that be with Confluent Cloud or Confluent Platform, and other associated settings that persist between command executions.
Being able to quickly switch contexts is so important to our hybrid users that, in the new update, confluent context has been promoted to a top-level command. Run confluent context list to see the list of available contexts:
$ confluent context list
If a context name is too tedious to type, it can be renamed:
$ confluent context update --name cloud
Then, switching to a new context is as easy as:
$ confluent context use cloud
Tip: If you switch between contexts frequently, you can use confluent prompt<c/code> to show the current context directly in the terminal! You can set this up by following the instructions in confluent prompt -h.
(confluent|cloud) $ confluent ...
Scenario: You are a Confluent Platform user who has an on-prem cluster. You are interested in migrating your data to Confluent Cloud. You will link the on-prem cluster to a cloud cluster, mirror a topic over the cluster link, and test that the setup works by producing and consuming across the cluster link.
Prerequisites:
$ confluent login --save Enter your Confluent Cloud credentials: Username: name@example.com Password: ********
$ confluent context update --name cloud +------------+---------------------------+ | Name | cloud | | Platform | confluent.cloud | | Credential | username-name@example.com | +------------+---------------------------+
$ confluent kafka cluster create my-cluster --type dedicated --cloud aws --region us-west-2 --cku 1 It may take up to 1 hour for the Kafka cluster to be ready. The organization admin will receive an email once the dedicated cluster is provisioned. +--------------+----------------------------------------------------------+ | Id | lkc-123456 | | Name | my-cluster | | Type | DEDICATED | | Ingress | 50 | | Egress | 150 | | Storage | Infinite | | Provider | aws | | Availability | single-zone | | Region | us-west-2 | | Status | PROVISIONING | | Endpoint | SASL_SSL://pkc-12345.us-west-2.aws.stag.cpdev.cloud:9092 | | ApiEndpoint | | | RestEndpoint | https://pkc-12345.us-west-2.aws.stag.cpdev.cloud:443 | | ClusterSize | 1 | +--------------+----------------------------------------------------------+$ export CC_CLUSTER_ID=lkc-123456 $ export CC_URL=pkc-12345.us-west-2.aws-confluent.cloud
$ confluent kafka cluster use $CC_CLUSTER_ID Set Kafka cluster "lkc-123456" as the active cluster for environment "env-12345".
$ confluent api-key create --resource $CC_CLUSTER_ID It may take a couple of minutes for the API key to be ready. Save the API key and secret. The secret is not retrievable later. +---------+----------------------------------------------------------+ | API Key | AAAAAAAAAAAAAAAA | | Secret | BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB | +---------+----------------------------------------------------------+$ export CC_API_KEY=AAAAAAAAAAAAAAAA $ export CC_API_SECRET=BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
$ confluent api-key use $CC_API_KEY --resource $CC_CLUSTER_ID Set API Key "AAAAAAAAAAAAAAAA" as the active API key for "lkc-123456".
password.encoder.secret=<encoder-secret>
$ export CP_URL=https://example.com:8090$ confluent login --url $CP_URL Enter your Confluent credentials: Username: name@example.com Password: *********
$ confluent context update --name platform +------------+----------------+ | Name | platform | | Platform | localhost:8090 | | Credential | username-name | +------------+----------------+
Tip: Now that we’re dealing with multiple contexts, it may be useful to display the current context directly in the terminal with confluent prompt:
$ export PS1='$(confluent prompt) '$PS1(confluent|platform) $ confluent kafka broker list --url $CP_URL/kafka Cluster ID | Broker ID | Host | Port
-------------------------+-----------+---------------------------------------+------- 0000000000000000000000 | 1 | ip-0-0-0-0.us-west-2.compute.internal | 9092(confluent|platform) $ export CP_CLUSTER_ID=0000000000000000000000
(confluent|platform) $ confluent kafka topic create my-topic --replication-factor 1 --url $CP_URL/kafka Created topic "my-topic".
link.mode=DESTINATION connection.mode=INBOUND
(confluent|platform) $ confluent context use cloud(confluent|cloud) $ confluent kafka link create my-link --source-bootstrap-server 0.0.0.0 --source-cluster-id $CP_CLUSTER_ID --config-file clusterlink-dst.config
link.mode=SOURCE connection.mode=OUTBOUNDbootstrap.servers=<CC_URL>:9092 ssl.endpoint.identification.algorithm=https security.protocol=SASL_SSL sasl.mechanism=PLAIN sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='<CC_API_KEY>' password='<CC_API_SECRET>';
local.sasl.mechanism=OAUTHBEARER local.security.protocol=SASL_SSL local.sasl.login.callback.handler.class=io.confluent.kafka.clients.plugins.auth.token.TokenUserLoginCallbackHandler local.sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required username='<CP_USERNAME>' password='<CP_PASSWORD>' metadataServerUrls='';
local.ssl.key.password=<KEY_PASSWORD> local.ssl.truststore.location=<PATH_TO_TRUSTSTORE> local.ssl.truststore.password=<TRUSTSTORE_PASSWORD> local.ssl.keystore.location=<PATH_TO_KEYSTORE> local.ssl.keystore.password=<KEYSTORE_PASSWORD>
(confluent|cloud) $ confluent context use platform(confluent|platform) $ confluent kafka link create my-link --destination-bootstrap-server $CC_URL:9092 --destination-cluster-id $CC_CLUSTER_ID --config-file clusterlink-src.config --url $CP_URL/kafka Created cluster link "my-link".
(confluent|platform) $ confluent context use cloud(confluent|cloud) $ confluent kafka mirror create my-topic --link my-link Created mirror topic "my-topic".
(confluent|cloud) $ confluent kafka topic consume my-topic --from-beginning Starting Kafka Consumer. Use Ctrl-C to exit.
(confluent|cloud) $ confluent context use platform(confluent|platform) $ ssh <username>@example.com "sudo cat /var/ssl/private/kafka_broker.crt" > client.crt (confluent|platform) $ ssh <username>@example.com "sudo cat /var/ssl/private/kafka_broker.key" > client.key
(confluent|platform) $ confluent kafka topic produce my-topic --bootstrap $CP_URL --cert-location client.crt --key-location client.key Starting Kafka Producer. Use Ctrl-C or Ctrl-D to exit. 🎉
We can see this data mirrored in Confluent Cloud!
Starting Kafka Consumer. Use Ctrl-C to exit. 🎉
The move to the unified Confluent CLI means that the Confluent Cloud CLI, ccloud, will be deprecated. On May 9, 2022, ccloud will be sunset. To guide users through the process of updating to v2, we’ve placed helpful messages in the latest minor version of ccloud, and the full list of breaking changes can be found in the documentation.
We introduced Confluent CLI v2, discussed the new features included, and demonstrated an example of linking a Confluent Platform cluster to a Confluent Cloud cluster for the purposes of mirroring data, using only the new unified CLI.
Looking back to the initial release of Confluent CLI v1, we’ve come a long way. This major version update for the CLI represents the first fully unified product at Confluent, capable of handling cloud and on-prem workflows from a single interface. We’re excited to keep developing the CLI to support all of the new features we have planned at Confluent!
The CLI team would like to hear from you! If you have any feedback, feature requests, bug reports, reach out on the Confluent Forum, send us an email at cli-team@confluent.io, or file a ticket through Confluent Support. We look forward to hearing from you!
We covered so much at Current 2024, from the 138 breakout sessions, lightning talks, and meetups on the expo floor to what happened on the main stage. If you heard any snippets or saw quotes from the Day 2 keynote, then you already know what I told the room: We are all data streaming engineers now.
We’re excited to announce Early Access for Confluent for VS Code. This Visual Studio integration streamlines workflows, accelerates development, and enhances real-time data processing, all in a unified environment. This post shows how to get started, and also lists opportunities to get involved.