Kafka Streams API
Stream processing designed for developers
The Streams API in Kafka is an open-source solution that you can integrate into your application to build and execute powerful stream processing functions. If you use Kafka for stream data transport, the Streams API in Kafka can immediately add stream processing capabilities to your application without the burden of adding an entirely separate distributed processing cluster for another stream processing framework. Whether you have an IoT application, a monitoring function, a complex continuous query, or you are tracking inventory changes, the Streams API in Kafka enables you to build your application with ease.
The Streams API in Kafka is included with the Apache Kafka release v 0.10 as well as Confluent Enterprise v3.0.Download Confluent Platform
Stream processing API's are very powerful tools. With this immense capability however, comes some complexity; they need their own dedicated cluster of machines, and often rely on a distributed database to perform look-ups and aggregations. Also, your application likely will be required to handle re-processing of old data separately.
The outcome? A lot of moving parts that need to be kept in synchronization with each other to make the application work.
Meet the Streams API in Kafka.Read our blog on the Streams API in Kafka
Microservices are often built upon a Kafka-based data stream. Instead of a large monolithic application, microservices deliver small, decoupled processes that execute a constrained feature set against a stream. These are ideal for the Streams API because they work on real-time streams, depend on reliable message delivery, and might not warrant the cost of deploying a separate stream processing framework.
One of the best ways to take advantage of stream data is to compare or analyze data in different streams against each other, or to join streams together to create more meaningful information. Continuous queries are used to automate real-time intelligence at scale across an organization. The Streams API allows functions of this scale to be implemented with a low infrastructure footprint and entirely within the Kafka environment.
Continuous transformations modify or aggregate the data in a stream. These humble but critical processes are low level services that prepare data for real-time analysis. Their small scale is well-suited to a lightweight solution like the Streams API in Kafka, which supports both stateful and stateless processing.
The endless search for potentially meaningful anomalies in organizational data is moving into a real-time environment. If data pipelines exist, implementing event-triggered processes against them using the Streams API in Kafka is a simple extension of the streaming infrastructure Kafka already provides.
Have someone from Confluent contact you.Contact Us