Partitioning

Changing and Optimizing Partition Count

KSQL automatically creates Kafka topics to hold its results, but you may still have to think about topic sizing. In cases where you don’t want to use the cluster-wide default number of partitions, you can specify the number of partitions you want to use.

Directions

In any KSQL query that will create a new Kafka topic, simply add the PARTITIONS parameter to the WITH clause. The new topic will now have that number of partitions.

1. Consider a Kafka topic called clickstream-raw. We don’t even need to know how many partitions it has, but we do know we need a new topic that has five partitions.

2. Create a STREAM from that partition:

CREATE STREAM clickstream-raw WITH (KAFKA_TOPIC=’clickstream-raw’,VALUE_FORMAT=’JSON’);


3. Create a new stream with the desired number of partitions. The newly created topic will be partitioned the way we specify.

CREATE STREAM clickstream_new WITH (PARTITIONS=5) AS SELECT * from clickstream_raw;
< Back to the Stream Processing Cookbook

We use cookies to understand how you use our site and to improve your experience. Click here to learn more or change your cookie settings. By continuing to browse, you agree to our use of cookies.