New in Confluent Cloud: Making Data, Pipelines, and Ops Accessible for AI-Ready Streaming

Your teams want to build production-grade artificial intelligence (AI) on streaming data—but doing so has typically required specialized tooling, fragmented workflows, and a steep learning curve that walls off AI from the data engineers, developers, and operators who power the rest of your data platform. This quarter’s release, coming to you from Current London, fundamentally changes this dynamic by making AI-ready streaming accessible across the data, pipelines, and ops your teams already manage. We’re unveiling the following features: 

• dbt adapter for Confluent Cloud for Apache Flink® and Materialized Tables to bring stream processing into the SQL-based workflows your data engineers already use

• A managed Model Context Protocol (MCP) server and Confluent Agent Skills, which put Confluent best practices directly into the AI coding agents that developers rely on

• General Availability (GA) of Streaming Agents and Real-Time Context Engine to power event-driven AI with enterprise-grade reliability 

At the heart of these innovations is the usage of schema IDs within Kafka headers, providing the essential foundation for governed data. By embedding schema information directly into the message metadata, we’ve made it possible to schematize every topic, ensuring that data remains consistent and discoverable as it moves through your pipelines. This shift allows your teams to treat streaming data as a first-class citizen, turning raw events into high-quality, AI-ready assets that maintain strict governance without sacrificing real-time speed.

Expanding Confluent Cloud for Apache Flink® Accessibility for Developers and Data Engineers

To build a modern data architecture, stream processing can’t be a specialized silo. It needs to be accessible to both the data engineers who manage the pipelines and build your business logic and the application developers who build the stateful, event-driven services that power the business. This launch bridges that gap: We’re providing declarative, SQL-based workflows for data engineers to build pipelines at scale and code-driven capabilities for application developers to build complex event-driven services. Our new features bring Flink directly into the tools and languages each team uses every day.

dbt Adapter for Confluent Cloud for Apache Flink® and Materialized Tables

Data engineers have standardized on dbt for SQL-based transformation because it brings software engineering best practices—version control, testing, and continuous integration (CI)/continuous deployment (CD)—to the data warehouse. Until now, bringing those same practices to Flink SQL required manual scripts or copy/pasting code into a console.

We’re removing that friction with the dbt adapter for Confluent Cloud for Apache Flink®. This free, open source plugin enables data engineering teams to define streaming pipelines as dbt models, test them, generate documentation, and deploy streaming pipelines to Flink compute pools using the same dbt workflow for managing models on data warehouses. It supports unit testing with mock data today, and data quality tests for live outputs are coming soon. Alongside the adapter, we’ve shipped the confluent-sql driver, a standalone driver that opens Flink to the broader Python ecosystem—including orchestration tools like Airflow, data analysis in Pandas, and AI frameworks. For teams migrating batch workloads from legacy platforms, the dbt adapter is the key to reusing existing project patterns and skills to go live in days rather than months.

Check out the documentation to start deploying Flink SQL with dbt.

To further simplify managing the life cycle of these pipelines, we’re introducing Materialized Tables, now generally available. Historically, making a simple change to a Flink statement—like adding a column—involved a stop-and-recreate process that required manual offset management to prevent data loss. Materialized Tables represents a shift from managing ephemeral statements to using persistent, database-like objects that automate offset bookkeeping and job orchestration through a single SQL statement. Using the new CREATE OR ALTER command, you can evolve your query in place while Flink orchestrates the complex mechanics of offset bookkeeping and job migration under the hood. This marks the end of manual migration cycles and the beginning of an easier experience for data engineers when operating Flink pipelines in production. Learn more in the documentation.

Process Table Functions

Process table functions (PTFs), currently in Early Access (EA), allow developers to write custom, stateful stream processing logic in Java and deploy it to Confluent Cloud as a SQL-callable function. This is a game-changer for building sophisticated application logic—such as custom windowing, state machines, session tracking, and deduplication logic—directly on Confluent Cloud for Apache Flink®. For developers already using the Table API to programmatically build Flink pipelines, PTFs provide the missing link to express complex application logic against events. By leveraging managed state and timers, you can build stateful microservices directly in Flink, benefiting from serverless scaling without leaving the Table API's declarative framework. Get started with the documentation.

External Connectivity for User-Defined Functions

To further extend what your applications can do, we’ve released external connectivity for user-defined functions (UDFs), now in EA. This capability allows your streaming services to interact with the broader world without requiring a separate microservices layer. Developers can easily enrich their streams by calling external REST APIs, databases, or AI services directly from Flink. This is critical for building real-time AI agents that must fetch external context or trigger actions in third-party systems as events arrive. To ensure that these connections are production-ready, sensitive credentials are managed as centralized Confluent Cloud resources, providing a secure, governed way to bridge your streaming applications with your entire technical stack.

Unifying Batch and Streaming With Snapshot Queries

We introduced Snapshot Queries last year in EA, and we’re excited to announce they’ll be in GA in June. Historically, organizations have faced a fragmented reality: Streaming workloads handled real-time data effectively, but accessing historical data for analysis required adopting an entirely separate system built on batch processing. These workloads are typically split across multiple tools, leading to fragmented developer workflows. 

Snapshot Queries eliminates that friction by allowing you to run one-time SQL queries that return a complete, point-in-time view of your data using union reads. This unique capability combines deep historical records stored in Tableflow with the latest real-time events in Apache Kafka®. Because these queries use Tableflow data (Apache IcebergTM/Apache ParquetTM format), they’re 50x–100x faster than scanning raw Kafka streams. For the first time, you can unify batch and streaming workloads on a single platform using the same Flink SQL, the same compute pools, and the same billing model. Whether you’re exploring large datasets for anomalies, running on-demand compliance audits, or backfilling historical data, Snapshot Queries provides the final piece of a truly unified data framework.

Building a Secure, Natural-Language AI Interface With a Managed MCP Server and Confluent Agent Skills

In Q1, we introduced support for the open source MCP server for Confluent Cloud so that any MCP client could tap into Kafka, connectors, Schema Registry, metrics, and more using a standard protocol.

In Q2, we’re taking a major step forward with a managed MCP server for Confluent Cloud, now generally available, along with new Confluent Agent Skills.

Managed MCP server for Confluent Cloud: The managed MCP server turns Confluent into a first‑class tool provider for MCP‑based agents. It exposes tools for Kafka, connectors, Schema Registry, metrics, and more—all discoverable by MCP clients. Agents can explore and debug Confluent resources via natural language: Discover and consume data, debug issues with connectors or things like back pressure, and more. Fully managed and hosted by Confluent, it comes with built‑in scalability, availability, and a four nines service level agreement (SLA). Learn more from the docs here.

Confluent Agent Skills: AI coding agents are rapidly becoming a primary interface for developers. To meet them where they work, we’re introducing Confluent Agent Skills—curated skills that encode Confluent best practices directly into AI agents. This cuts time to production and improves the quality of Confluent deployments. Key features include:

• Open source skills (https://github.com/confluentinc/agent-skills): Apache 2.0‑licensed skills that guide developers through common Confluent workflows (e.g., setting up change data capture [CDC] pipelines, configuring producers/consumers, adopting Schema Registry) and that are usable with any agent that supports the Agent Skills Specification

• Governed life cycle: A three‑tier governance model (AI‑assisted authoring, continuous integration [CI] validation, and human subject matter expert review) to ensure that skills remain high quality, accurate, and aligned with best practices

For teams standardizing on AI coding agents (e.g., Claude Code, Cursor, GitHub Copilot), this means developers can get context‑aware guidance on Confluent directly from their agents, backed by the same platform and governance they rely on in production. Learn more from the Agent Skills blog post.

Expanded Local MCP Server for Apache Kafka® and Confluent Platform for Hybrid Development

We’re also expanding functionality in the open source local MCP server for Apache Kafka and Confluent Platform so that teams can develop and test locally using the same tool and skill definitions they’ll use in production. They can seamlessly move between local and cloud by pointing agents at either the local or managed MCP server without rewriting tools. Finally, they can keep sensitive or experimental tools local while still adopting a standard MCP‑based architecture.

This hybrid approach lets platform teams adopt MCP in a way that meets security and rollout requirements.

New Capabilities in Confluent Intelligence

Last year, we introduced Confluent Intelligence, a fully managed service for building real-time, context‑rich, and trustworthy AI systems on a unified data streaming platform that brings together Kafka and Flink. This quarter, we’re introducing key features and updates that make it easier for agents to act on the live state of your business.

Real-Time Context Engine

We’re excited to announce that Real-Time Context Engine is now generally available, delivering continuously refreshed structured context to power any AI system. It transforms data into structured, low-latency context for any AI agent or application to consume instantly through MCP—all fully managed in Confluent Cloud. With the GA release, the engine has evolved from primary-key lookups to a sophisticated query layer with full support for filters, ranges, and compound queries. By supporting all major schema types and scaling automatically, it removes the need for separate operational databases, ensuring that the hardest part of production AI—context engineering—is handled natively in your data streams.

Streaming Agents

Streaming Agents is also now generally available, bringing production-ready, event-driven agents directly into your Flink and Kafka pipelines. Rather than relying on batch snapshots, the agents continuously monitor live business signals to take autonomous action with enterprise-grade reliability and a 99.99% SLA. This release also includes an agent reflection pattern that lets agents iteratively critique and refine their own outputs before emitting a single, trusted event into the stream. Start building your own event-driven agents in minutes with the Quickstart and documentation.

Agent Management Console

The Agent Management Console is a centralized, UI-driven control plane that provides visual oversight for your Streaming Agents. Instead of digging through SQL jobs, developers and platform teams can see exactly how an agent is wired—viewing its inputs, outputs, prompts, models, and context tables in a single experience. This brings the same operational rigor to AI that you already apply to core microservices.

The console accelerates iteration by allowing teams to create and refine prompts or data wiring through a guided interface rather than just code. By providing tools to test agents with sample inputs and inspect live logs, it bridges the gap between AI developers and platform operators, making it easier than ever to deploy and monitor autonomous agents with confidence.

Expanding Governance With Schema IDs in Kafka Headers

We’re officially introducing support for schema IDs in Kafka headers, a feature that finally decouples your data governance from your payload format. This allows teams to move from “dumb pipes” to a smart data plane without the risk of breaking legacy systems or requiring complex wire-format changes. By moving the 5-byte schema unique identifiers out of the message body and into the metadata layer, you can now schematize existing topics in minutes, ensuring that every event is well structured and immediately usable for downstream applications.

This update solves the migration headache that historically plagued Kafka teams trying to introduce schemas onto topics that never had them. Because the new Confluent deserializers employ a header-first lookup strategy, you can upgrade your producers and consumers independently and at your own pace. Legacy consumers can continue reading raw, schemaless payloads while "smart" consumers validate data against the Schema Registry using the globally unique identifier (GUID) in the header; the payload bytes stay untouched, so nothing downstream breaks. This creates a zero-downtime path to governance, letting you enforce strict data contracts across your organization without the friction of synchronized service restarts or a big-bang cutover.

Beyond simplifying operations, schematizing your streams at the header level turns Kafka into the intelligent backbone for your entire AI and analytics estate. Well-governed topics are now lakehouse-ready by default, allowing tools like Tableflow and Flink to ingest validated, high-quality data for your analytics and AI use cases. Whether you’re feeding real-time applications or training machine learning models, schema IDs in headers ensure that your data arrives contextualized and trustworthy. It’s a foundational shift that proves you don’t have to choose between moving fast and moving with governance.

Simplifying Client Migrations With Kafka Copy Paste

Earlier this year, we introduced Kafka Copy Paste (KCP), the open source CLI designed to automate migration to Confluent Cloud, including cluster discovery and cost assessment, infrastructure provisioning, and data migration. Today, we’re extending KCP to solve for the final mile of the journey: client migrations. KCP now integrates with Confluent Cloud Gateway, a cloud-native Kafka proxy that acts as an intelligent routing layer between your clients and your Kafka clusters. It routes Kafka protocol messages, rewrites metadata and broker addresses, and presents virtualized endpoints to clients. This enables KCP to transparently redirect traffic from the source Kafka cluster to the destination Confluent Cloud cluster without any client-side changes.

Clients make a single, non-urgent change—updating their bootstrap URL to point at the Gateway—which can be done at any time with no immediate pressure or downtime. From there, platform teams use KCP to organize topics and clients into logical migration groups, enabling migration of specific domains or services independently to minimize the blast radius. KCP also exposes a command that monitors replication lag, allowing you to know exactly when your source cluster is fully replicated to the destination. When you're ready, a single command fences traffic, flips the routing from the source cluster to the destination cluster, and then resumes traffic—ensuring that consumers pick up exactly where they left off with zero manual offset management.

Gateway also handles authentication translation in the background. Clients can continue using their existing credentials while the Gateway automatically maps them to Confluent Cloud API keys. This works for any Kafka-compliant source, including Apache Kafka, hosted Kafka, and Confluent Platform.

Explore KCP on GitHub and check out the deep-dive blog post to walk through the key steps of client migration with KCP. 

Additional New Features and Updates

Tableflow Updates

We’re releasing several Tableflow enhancements so that you can run topics‑to‑tables pipelines with confidence at enterprise scale:

• User-defined namespaces (generally available): Replace opaque, system-generated IDs with meaningful, business-aligned names such as "finance" and "analytics." This ensures that your streaming tables integrate seamlessly with existing governance and discovery structures in external catalogs such as AWS Glue, Databricks Unity Catalog, and Snowflake Open Catalog.

• Dead letter queue (DLQ) support for JSON schema (generally available): Protect the integrity of your production pipelines by isolating malformed JSON records into dedicated DLQ topics. This ensures that your primary tables remain clean and trusted while preserving problematic events for later inspection and faster debugging.

• Client-side field level encryption (CSFLE) support (generally available): Security-conscious organizations can now land CSFLE-encrypted Kafka topics directly into analytics-ready tables. Tableflow natively decrypts fields using customer-managed keys during materialization, ensuring that sensitive workloads are first-class citizens in your data lake.

• Observability and metrics (generally available): Gain deep visibility into your pipelines with granular metrics for offset lag, event/processing freshness, and sync health per destination. These new metrics, paired with structured life cycle logs and actionable error messages, integrate seamlessly with tools such as Splunk and OpenTelemetry to power independent debugging and monitoring.

• Data time to live (TTL) (Limited Availability): Control storage costs for petabyte-scale workloads by implementing Kafka-style, time-based retention policies. Data TTL automatically purges expired records based on timestamps, allowing you to right-size your storage footprint and stay in compliance with internal and legal retention requirements.

We’re also excited to announce tiered pricing for Tableflow topics to make it easier for teams to get started with Tableflow. This new pricing eliminates cost uncertainty with transparent tiers that automatically lower your effective rate as your usage grows. With incremental costs reaching as low as $0.01 per topic-hour, you can confidently scale your most demanding workloads. Learn more about how Tableflow can lower your total cost of ownership (TCO) in the blog post and stay tuned for detailed updates on the Tableflow pricing page later this month. Contact your account team for more details.

Connector Updates

New Fully Managed Connectors

We’re introducing several new fully managed connectors for Confluent Cloud: 

Custom Single Message Transforms (SMTs) on Google Cloud

We’re extending support for custom SMTs to Google Cloud in addition to AWS and Azure, enabling you to upload and execute your own custom SMT code on fully managed connectors for clusters using private networking setups. 

Egress PrivateLink Support for Custom Connectors on AWS

We're introducing Egress PrivateLink connectivity to provide secure, one-way private networking for custom connectors. By configuring dedicated gateways and access points, administrators can seamlessly route traffic to external systems across Enterprise and Dedicated clusters on AWS while maintaining a rigorous security posture.

Secrets Manager Support for Fully Managed Connectors 

To eliminate credential sprawl and strengthen your security posture, we're introducing secret manager support for fully managed connectors. Currently available for AWS Secrets Manager and Azure Key Vault with Dedicated clusters, this feature allows you to centralize credential management directly in your existing cloud-native secret stores. By doing so, you can automate secret rotation and life cycle management, ensuring that your Kafka Connect pipelines stay running smoothly while credentials change safely behind the scenes.

Tiered Pricing for Kafka Connect Tasks

We’re also excited to announce tiered pricing for Kafka Connect tasks, with lower rates automatically unlocked as your connector footprint expands. With incremental discounts reaching up to 80% at scale, you can now deploy even the most complex software-as-a-service (SaaS), database, and event-stream integrations with maximum cost efficiency. Check out the Kafka Connect pricing page in the upcoming weeks or reach out to your account team to learn more.

Security Updates

Global API Keys

Confluent Cloud is tackling the headache of key sprawl by introducing Global API Keys. You can now leverage a consolidated credential to access the entire ecosystem of Confluent Cloud services instead of juggling separate keys for individual products like Kafka clusters, Schema Registry, and managed connectors. This centralization ensures that developers and administrators can interact with the platform without the friction of managing fragmented, resource-specific keys. Beyond simplifying daily workflows and development testing, Global API Keys also enable seamless client transitions across clusters without the need to change authentication. They provide a scalable, secure foundation for managing even the most complex streaming footprints.

BYOK for Freight Clusters

Bring Your Own Key (BYOK) support for Freight clusters is coming soon, making it possible for organizations to take advantage of the cost savings of this elastic cluster type while maintaining their security and compliance postures. This is on top of existing support for self-managed keys on Enterprise and Dedicated clusters on AWS, Azure, and Google Cloud.

Observability and Metrics Updates

We’re making it easier to troubleshoot and optimize your streaming workloads with a suite of new observability updates:

• The new client limit metric reports exactly which principal is responsible for breaching a throughput quota and the specific reason, collapsing hours of manual log correlation into minutes. 

• To tackle performance bottlenecks, our UI now visualizes consumer group rebalance duration and frequency, helping you identify consumer group instability at a glance. 

• We’ve also introduced a connection attempts metric that reports the total count of connection attempts per interval, allowing you to spot connection storms before they impact cluster throughput. 

• Beyond the API, the Confluent Cloud Console features a refreshed cluster monitoring page that provides deep visibility into both elastic and Dedicated clusters. For elastic clusters, the new elastic Confluent unit for Kafka (eCKU) usage chart allows you to visualize real-time capacity and the specific dimensions—such as partitions or request rates—driving your usage.

To learn more about these new updates, check out the blog post.

Confluent Cloud for Government is FedRAMP Moderate Authorized

Confluent Cloud for Government (CCG) is now available on the FedRAMP Marketplace, with FedRAMP Moderate authorization achieved through the competitive FedRAMP 20x Pilot program. This milestone means federal, state, local, and tribal government agencies—as well as commercial organizations supporting them—can now deploy enterprise-grade data streaming infrastructure in days, not months, and be confident that it meets rigorous federal security standards.

Confluent Private Cloud Centralized Policy Enforcement (Private Early Access)

We’re excited to announce the Private EA of Confluent Confluent Private Cloud (CPC) centralized policy enforcement, with general availability planned later this year. In many Kafka environments, hundreds of teams manage their own schemas and authentication practices, creating inconsistent encryption, governance gaps, and significant operational overhead. Legacy applications and third-party integrations further complicate enforcing modern security and data quality standards at the source.

CPC centralized policy enforcement brings Gateway field-level encryption/decryption, payload encryption/decryption, and deep schema validation directly into the Gateway. By centralizing these controls, you can enforce consistent security, compliance, and data quality policies without requiring code changes across individual applications. This reduces coordination overhead, simplifies upgrades and audits, and helps ensure that only validated, compliant data enters the streaming ecosystem—even for legacy and highly regulated environments.

Start Building With New Confluent Cloud Features

If you’re new to Confluent, sign up for a free trial of Confluent Cloud and create your first cluster to explore the new features. New sign-ups receive $400 to spend within Confluent Cloud during their first 30 days. Use the code CLOUDBLOG60 for an additional $60 worth of free usage.

The preceding outlines our general product direction and is not a commitment to deliver any material, code, or functionality. The development, release, timing, and pricing of any features or functionalities described may change. Customers should make their purchase decisions based on services, features, and functions that are currently available.

Confluent and associated marks are trademarks or registered trademarks of Confluent, Inc.

Apache®, Apache Kafka®, Apache Flink®, Flink®, Apache IcebergTM, IcebergTM, Apache ParquetTM, ParquetTM, and the respective logos are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by the Apache Software Foundation is implied by using these marks. All other trademarks are the property of their respective owners.