Extensive out-of-the-box functionality, a large user community, and up-to-date, cloud-native features make Spring and its libraries a strong option for anchoring your Apache Kafka® and Confluent Cloud based microservices architecture. Spring takes care of boilerplate system responsibilities—letting you focus on your business logic—while serverless Apache Kafka on Confluent Cloud provides your transport and storage layers.
For an entire course on using Spring and its libraries with the Apache Kafka ecosystem, make sure to visit Confluent Developer.
Spring’s opinionated approach can significantly reduce your development time and allow you to easily collaborate with other developers on your team. Moreover, its support for multiple profiles means that you are able to provide different configuration parameters based on your environment (for example, development vs. QA). Adding the Spring for Apache Kafka project to your Spring implementation provides Kafka-specific capabilities, and its features are modeled on common Spring patterns, so you are likely to find them familiar.
SPRING101for $101 of free Confluent Cloud usage.
KafkaTemplate class in the Spring for Apache Kafka project was designed to be similar to
JMSTemplate. It’s relatively simple to use, particularly if you are already familiar with Kafka producers—which it wraps. To proceed, you create a
ProducerFactory bean, providing a configuration map, then use that to make a
KafkaTemplate, which ultimately provides you with numerous convenience methods. (Note that Kafka is asynchronous by default, so Spring provides a facility for handling synchronicity.)
In Spring, the components that communicate with messaging systems have access to message-driven POJO functionality, in which regular POJOs can serve as asynchronous message listeners. Thus, consuming Kafka messages in Spring is accomplished by simply annotating a bean with
KafkaListener, which will cause the Spring framework to instantiate a
MessageListenerContainer that will take care of parallelization, configuration, retries, offsets, and other elements needed by your Kafka application. Offloading this work allows you to focus on your primary logic.
KafkaListenerannotation and specify topics that you’d like to subscribe to. Then check for the existence of your consumer on Confluent Cloud.
Manual topic creation is straightforward in Confluent Cloud, but you may wish to create topics programmatically, for example, if you are working with regular expressions in topic names or creating a large number of topics. You can accomplish this in Spring with the
TopicBuilder class in conjunction with the
KafkaAdmin class, which wraps Kafka’s Admin API.
TopicBuilder methods allow you to provide standard configuration parameters relating to, for example, the number of partitions and number of replicas—but also relating to some parameters that aren’t available by API—such as one covering the compression type used in the topic.
TopicBuilderclass in your Spring code, then verify it on Confluent Cloud.
EnableKafkaStreams annotation, some configuration parameters, as well as a
StreamsBuilder to your Spring code will let you access the Kafka Streams APIs. Spring’s wrapper over Kafka Streams is quite thin and Spring handles lifecycles, which lets you primarily focus on business logic (as you have probably come to expect by now after learning about the other Spring features). Additionally, Spring for Apache Kafka wraps JMX metrics from Kafka Streams and makes them available through the Micrometer framework.
StreamsBuilderand adding a
KStreamto it, then adding a KTable for processing your data. Next, convert your
KStream, and set up a new topic with
TopicBuilder. Then view your data on Confluent Cloud, modify your data for the console, and finally create a REST service for sharing your data.
When you create a Confluent Cloud cluster, you have the option to create a Schema Registry. This establishes it on Confluent Cloud, but to connect to it from your Spring Boot application, you’ll need its URL as well as some credentials. Your options for formats are Avro, JSON Schema, and Protobuf, and you can use multiple schemas in one application (Confluent provides SerDes for all three).
Leveraging Spring can enable you to quickly start developing sophisticated Apache Kafka based systems. Although Spring is opinionated by default (which, as mentioned above, saves time and simplifies communication among developers), keep in mind that it does provide the ability to extend or customize certain options, should you need something not in the out-of-the-box version.
To learn more about combining Spring and Apache Kafka:
SPRING101for $101 of free usage
Viktor Gamov is a principal developer advocate at Kong, a cloud connectivity company. Viktor has comprehensive expertise in building distributed systems and cloud-native architectures using open source technologies. He enjoys helping architects and developers design and develop low-latency, scalable, and highly available systems. He is a professional conference speaker on distributed systems, streaming data, JVM, and DevOps topics, and he is a regular at events like JavaOne, Devoxx, OSCON, and QCon. He co-authored O’Reilly’s “Enterprise Web Development” and co-hosts the DevRelRad.io podcast.