Confluent KSQL

Streaming SQL for Apache Kafka

Confluent KSQL is the streaming SQL engine that enables real-time data processing against Apache Kafka®. It provides an easy-to-use, yet powerful interactive SQL interface for stream processing on Kafka, without the need to write code in a programming language such as Java or Python. KSQL is scalable, elastic, fault-tolerant, and it supports a wide range of streaming operations, including data filtering, transformations, aggregations, joins, windowing, and sessionization.

Download Confluent
Get started with KSQL
Streaming SQL for Apache Kafka

KSQL: Query Your Streams Without Writing Code
Enjoy real-time, fault-tolerant stream processing against Kafka today.

Get up and running with these helpful resources

WATCH THE ONLINE TALK:
STREAMING SQL FOR APACHE KAFKA

Learn how to build real-time streaming applications with KSQL. This talk explains the KSQL engine architecture, and how to design and deploy interactive, continuous queries for streaming ETL and real-time analytics.

Watch Video
WATCH THE ONLINE TALK: <br/>STREAMING SQL FOR APACHE KAFKA
<<<<<<< HEAD ======= >>>>>>> master
<<<<<<< HEAD =======

Level Up your KSQL

Whether you are brand new to KSQL or ready to take it to production, now you can dive deep on core KSQL concepts, streams and tables, enriching unbounded data and data aggregations, scalability and security configurations, and more.

  • KSQL Introduction

    Get an introduction to the concept of stream processing with Apache Kafka and KSQL.

    Watch Now
  • KSQL and Core Kafka

    Learn about relating KSQL to clients, choosing the right API and how KSQL uses Kafka topics.

    Watch Now
  • KSQL Use Cases

    KSQL use cases include data exploration, arbitrary filtering, streaming ETL and more.

    Watch Now
  • Installing and Running KSQL

    Find out how to get KSQL, start the KSQL server and CLI, along with other syntax basics.

    Watch Now
  • KSQL Streams and Tables

    Distinguish a STREAM from a TABLE, and discover how streaming queries are unbounded.

    Watch Video
  • Reading Kafka Data from KSQL

    Explore Kafka topic data. Create a STREAM or TABLE. Identify fields, metadata and formats.

    Watch Video
  • Streaming and Unbounded Data

    Stream queries, read topics, discover persistent and non-persistent queries and more.

    Watch Video
  • Enriching data with KSQL

    Use scalar functions, change field types, filter and merge data and rekey streams with KSQL.

    Watch Video
  • Aggregations in KSQL

    Review various aggregate functions (e.g., MAX, MIN), windowing and late-arriving data.

    Watch Video
  • Taking KSQL to Production

    Build a streaming ETL pipeline, scale processing, secure KSQL and monitor KSQL performance.

    Watch Now
  • INSERT INTO in KSQL

    A brief tutorial on how to use INSERT INTO in KSQL.

    Watch Now
  • STRUCT in KSQL

    A brief tutorial on how to use STRUCT (Nested Data) in KSQL.

    Watch Now

Use Cases and Examples

01

Streaming ETL

Apache Kafka is a popular choice for powering data pipelines. KSQL makes it simple to transform data within the pipeline, readying messages to cleanly land in another system.

CREATE STREAM vip_actions AS 
SELECT
userid, page, action FROM clickstream c LEFT JOIN users u ON c.userid = u.user_id
WHERE u.level = 'Platinum';

02

Anomaly Detection

KSQL is a good fit for identifying patterns or anomalies on real-time data. By processing the stream as data arrives you can identify and properly surface out of the ordinary events with millisecond latency.

CREATE TABLE possible_fraud AS
SELECT
card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 SECONDS)
GROUP BY
card_number
HAVING count(*) > 3;

03

Monitoring

Kafka’s ability to provide scalable ordered messages with stream processing make it a common solution for log data monitoring and alerting. KSQL lends a familiar syntax for tracking, understanding, and managing alerts.

CREATE TABLE error_counts AS 
SELECT
error_code, count(*)
FROM monitoring_stream
WINDOW TUMBLING (SIZE 1 MINUTE)
WHERE
type = 'ERROR'
GROUP BY error_code;

>>>>>>> master

We use cookies to understand how you use our site and to improve your experience. Click here to learn more or change your cookie settings. By continuing to browse, you agree to our use of cookies.