Author: Manveer Chawla

Kafka is your event backbone, not your inference runtime. This guide breaks down three patterns for running AI alongside Kafka (external API, embedded, sidecar), when to use each, and how to handle topic design, dead-letter queues, idempotency, and LLM cost control.

Batch ETL feeds AI models data that's hours old. That causes context drift in RAG, training-serving skew in fraud detection, and broken operational AI. This guide covers the Ingest, Process, Serve architecture using Kafka and Flink to keep embeddings, features, and context fresh in milliseconds.

Unstructured data (PDFs, scans, images) breaks every assumption built for structured pipelines. This guide walks through a four-stage streaming architecture for turning messy binary blobs into RAG-ready chunks and embeddings, with patterns for rate limits, cost control, and fault tolerance.

Stream processing and real-time OLAP solve different problems, but vendor marketing makes them sound the same. This guide breaks down when to use Flink vs ClickHouse/Pinot, what to precompute vs query on the fly, and how Kafka connects both layers into one architecture.

As businesses increasingly rely on Apache Kafka® for mission-critical applications, resiliency becomes non-negotiable. Any unplanned downtime and breaches can result in lost revenue, reputation damage, fines or audits, reduced CSAT, […]

A preview of Confluent Tiered Storage is now available in Confluent Platform 5.4, enabling operators to add an additional storage tier for data in Confluent Platform. If you’re curious about […]