Show Me How: Build Streaming Data Pipelines for Real-Time Data Warehousing | Register Today
With Confluent Platform 5.3, we are actively embracing the rising DevOps movement by introducing CP-Ansible, our very own open source Ansible playbooks for deployment of Apache Kafka® and the Confluent Platform.
Since its release seven years ago, Ansible has become the most popular automation tool for configuration management.
Unlike other configuration management tools, it allows for a much simpler deployment model due to its agentless nature. By using passwordless SSH, it’s able to communicate with all of Confluent Platform’s supported operating systems in a clean, frictionless way.Ansible uses YAML, which has a clean and simple syntax that serves as an easy starting point while also being incredibly flexible. It allows the state of the infrastructure to be described in a very readable way using files that have a logical structure.
all: vars: ansible_connection: ssh ansible_user: ec2-user ansible_become: true sasl_protocol: plain
As an open source project written in Python, Ansible is constantly being improved by its community, and new modules are being added regularly. This leads to even greater stability and flexibility as the community continues to grow.
With the 5.3 release of CP-Ansible, we’ve added many capabilities to our playbooks to help simplify setup and configuration of the Confluent Platform while utilizing best practices.
To get started, first read the hosts you have defined in your inventory file and make sure that the correct hostnames are specified in each component’s configuration in order to define the correct communication paths. This includes making sure that the brokers have the right hostnames to communicate with each other, as well as with Apache ZooKeeper™, Confluent Schema Registry, and any additional components in the cluster. This allows you to scale across N hosts and to know that you always have the correct configuration.
We then apply our best practices to both the host and the Confluent Platform component. For example:
You can also customize your installation by overriding key variables within the playbook. In the case of overriding the ID on a broker, you would modify the
hosts_example.yml as follows:
For a more complete list of variables that you can override, review the
confluent.variables role for global variables. For component-specific variables, you can review
confluent./defaults/main.yml. Both are located in the roles directory. You can override any of these variables via your
hosts_example.yml, similar to the
broker_id example above.
In addition to the above best practices and ability to customize, CP-Ansible greatly simplifies security deployment through:
CP-Ansible playbooks allow you to specify which security protocol you wish to deploy with. By having the prerequisites for the given security protocol in place, the playbooks will deploy across any given number of components/hosts specified.
In this tutorial, we will deploy a cluster with SASL PLAIN security using CP-Ansible. The minimum installation requirements can be found in the documentation.
1. Clone the CP-Ansible repository to the host you wish to deploy from:
git clone firstname.lastname@example.org:confluentinc/cp-ansible.git
cp hosts_example.yml hosts.yml
hosts.yml using the editor of your choice. In this instance, we are using
4. Set the
ansible-user: property to be a user who has sudo access on the hosts you wish to deploy to:
all: vars: ansible_connection: ssh ansible_user: ansible_become: true
5. Uncomment this variable to enable
sasl_plain and make sure it aligns with the
# sasl_protocol: plain
6. Under each component name, modify the
hosts: section to reflect the hostnames or IP addresses of the hosts you wish to deploy to:
zookeeper: hosts: Ip-10-0-0-34.eu-west-2.compute.internal:
7. Save your edits to the
8. Execute the following command:
ansible-playbook --private-key= -i hosts.yml all.yml
9. You will now see the playbooks execute:
10. When completed successfully, you should see output similar to the following:
To validate your installation, access the Confluent Control Center interface by opening your web browser and navigate to the host and port where Control Center is running. Control Center runs at
http://localhost:9021/ by default.
When you first open Control Center, the homepage for your clusters appears. The homepage provides a global overview of all clusters managed by the Control Center instance. Each cluster tile displays its running status, Kafka overview statistics, and connected services.
You should see something similar to the image above showing one healthy cluster and a list of components. You can drill into individual clusters by clicking on the cluster name for more details.
We highly value your contributions as CP-Ansible should reflect the needs of the Apache Kafka and Confluent communities. These include code contributions, but just as importantly, feature requests and bug reports.
You can contribute code, bug reports, and feature requests to the public repository by using the contributions guide.
If you have a commercial support contract with Confluent, feel free to open a support case with our world-class support team.
We have many exciting new capabilities coming to CP-Ansible in the short and medium terms, including but not limited to:
Until then, I encourage you to learn more by listening to this episode of the Streaming Audio podcast.
And, if you haven’t already, you can download the Confluent Platform and easily spin up an installation using the playbooks and templates from CP-Ansible.
Use the Confluent CLI and API to create Stream Designer pipelines from SQL source code.
This post details how to minimize internal messaging within Confluent platform clusters. Service mesh and containerized applications have popularized the idea of control and data planes. This post applies it to the Confluent platform clusters and highlights its use in Confluent Cloud.