Managed Model Inference Brings AI into Real-Time Data Streams

Verfasst von

AI models are becoming increasingly powerful. But intelligence alone isn't enough. What matters is whether that intelligence can drive decisions in real time, when they matter most. With Managed Model Inference, we're closing the gap between data and AI-powered decision-making. We are extending Confluent's shift-left approach to AI, bringing AI models closer to the data source and making inference available directly in the stream, through a single SQL call, with zero configuration.

We bring AI to the data.

AI with batch ETL means slow time to action and stale context for decision-making.
Managed Model Inference shifts AI left, bringing Inference to real-time data streams.

Consider timing. A suspicious transaction is worth stopping while the payment is still in flight. Minutes later, it is a loss to investigate, and an hour later, the opportunity to act may be gone. The model can be equally accurate in all three cases. What changed is when it answered. The same is true in manufacturing, where drift has to be caught before it becomes downtime; in retail, where a shelf has to be restocked before it empties; and in energy, where unusual load has to be spotted as it happens.

When mission-critical actions lose value with every passing minute, inference belongs in the data stream, not after it.

Managed Model Inference for LLMs and embedding models is now generally available in Confluent Cloud. Forecasting, anomaly detection, and text analysis models are in early access.

Where the Friction Comes From

Bringing AI inference into real-time data streams often requires significant configuration and integration work. Models typically live somewhere else: behind a provider's API, in a separate service, or in an external machine learning environment. Making these models available within your streaming applications introduces two major challenges.

The first is delay: When inference happens in a separate processing pipeline, data may need to move between systems before a prediction can be made and acted upon. Those additional steps can delay decisions that depend on real-time data.

The second is configuration and operational complexity: Before you can run inference, you may need to set up provider accounts, manage API keys, configure endpoints, establish connectivity, and go through security reviews. Depending on your architecture, you may also need additional infrastructure to deploy, operate, and maintain.

What if calling a model from a data stream were as simple as calling any other SQL function, right where your data already lives?

Introducing Managed Model Inference

We are announcing Managed Model Inference for Confluent Cloud for Apache Flink®, and it answers exactly that: The model runs on the stream, and the setup is gone.

Inference simply means running a trained model on new input to get an output: text, a vector or a forecast. With Managed Model Inference, you run it from Flink SQL against supported models hosted by cloud service providers, with no credentials or configuration to set up. You pick a supported model and call it by name, the way you call any other SQL function, so the model's output arrives with the event and not after it. There is no provider account to open, no API key to store, and no connection, endpoint or serving infrastructure to run.

Run SHOW MODELS in a SQL workspace to see which models your region offers, then call one by name:

AI_COMPLETE(`system.anthropic.claude-sonnet-4-6`, 'your prompt')

Managed Model Inference sits alongside External Model Inference, which has been generally available since March 2025. With External Model Inference, you connect to a model yourself, using your own provider credentials and configuration. Use it for models outside the managed list or for your own endpoint, and use Managed Model Inference for everything else. Together they make up Model Inference in Confluent Cloud.

External Model Inference

Managed Model Inference

Run inference from Flink SQL against a model you connect to yourself, using your own provider credentials and configuration.

Run inference from Flink SQL against supported models hosted by cloud service providers (CSPs), with no credentials or configuration to set up.

- You create the connection and CREATE MODEL

- You bring your own account and API keys

- You pay the provider directly

- Call a supported model by name

- No connection, credential, or endpoint management

- Usage appears on your Confluent bill

Three Kinds of Managed Models, and More to Come

Managed Model Inference currently supports three categories of models. LLMs and embedding models, which add judgment and meaning to events, are generally available today. Two additional categories are in Early Access: forecasting and anomaly detection, which predict what comes next and identify unusual patterns, and text analysis, which detects sensitive information (PII) and analyzes sentiment.

LLMs and Embeddings (Generally Available)

AI_COMPLETE puts an LLM in the stream. Prompt it on every event to classify, summarize, extract or judge, in plain Flink SQL. AI_EMBEDDING turns text into vectors, so semantic search and retrieval-augmented generation can run from the same pipeline.

The models come from the three major clouds: Claude, Amazon Nova and Titan embeddings on Amazon Bedrock, GPT-5.4 and text-embedding-3 on Azure OpenAI, and Gemini and Gemini Embedding on Google Vertex AI. The SQL is the same on every cloud. Only the model name changes.

When you want a reusable model, for example for a Streaming Agent, register it with CREATE MODEL and set 'provider' to 'confluent'.

See it in action: calling a model by name. In this short demo, SHOW MODELS lists the models available to you, and a single AI_COMPLETE call asks Claude Sonnet 4.6 to judge fraud risk. As a bank's user activity streams in (purchases, logins and account changes), the model reads each user's recent activity and returns a one-line evaluation of how likely it is to be fraud, one user at a time. CREATE MODEL then registers the same model so a Streaming Agent can use it. The agent scores each user and takes action on high risk, such as freezing an account or flagging a transaction, and a dashboard shows the alerts as they come in.

Forecasting and Anomaly Detection (Early Access)

We're also bringing time-series inference directly into Flink SQL, with AI_FORECAST to predict future values and AI_DETECT_ANOMALIES to identify unexpected patterns in your data.

Behind these functions are time-series foundation models, which bring an important advantage: zero-shot inference. These models are pretrained on diverse time-series datasets, so they can generate predictions on data they've never seen before, without requiring you to train a separate model for each dataset.

We're starting with IBM Granite time-series models, which are designed for efficient inference, including lightweight models that run on CPUs. Confluent manages the serving infrastructure, so you can focus on applying these models to your data.

Our inference platform is model-agnostic by design. Different models excel at different tasks, and we want to give developers the flexibility to choose the best fit without changing their streaming applications.

Here's a quick look at the models we're supporting:

  • IBM Granite TTM: Lightweight, efficient forecasting across many time series.

  • IBM Granite FlowState: Forecasting for irregular or mixed-frequency data.

  • IBM Granite PatchTST-FM: Probabilistic forecasting that captures uncertainty.

  • IBM Granite TSPulse: Purpose-built for anomaly detection.

  • Google TimesFM: General-purpose time-series forecasting.

There's no single model that performs best on every dataset. That's why we're making it easy to experiment: For supported forecasting models, switching models requires changing just one parameter, while the rest of your SQL query stays the same.

Confluent is model-agnostic: Choose the model that matches how your data behaves, and run it from Flink SQL.

Text Analysis (Early Access)

Text analysis covers functions such as PII detection and redaction, and sentiment analysis, with models such as Piiranha for PII and DeBERTa for sentiment. It follows the same pattern as the other two kinds: a specialized model called through one SQL function.

What Managed Model Inference Costs

For LLMs and embeddings, you pay the cloud provider's token price with no Confluent markup. Usage is metered in input and output tokens, and it appears as its own line on your Confluent invoice, separate from Flink compute. The statement that makes the call is billed at normal Flink pricing. The pricing page lists the rate for every model.

To keep spend visible, organization admins can set a monthly budget in USD with the Confluent CLI, with alerts at 50%, 90% and 100% by email, and by Slack, Microsoft Teams and webhook as well. Budgets send alerts and do not cap spend, so inference keeps running past 100%.

See It All Work Together: Fraud Detection Demo

Here is a fraud detection workflow built entirely in Flink SQL. It brings together two of the three kinds of managed models: a time series model to detect, and LLMs to explain and act.

The demo watches ten simulated customers shopping and logging in, each with their own normal level of spend. About one session in twenty is fraud. Flink aggregates each customer's transactions, logins and account changes into an activity profile.

Detect

AI_DETECT_ANOMALIES runs on IBM Granite TTM, a time series model hosted by Confluent, so there is no model server to deploy or operate. Partitioning by user gives each customer their own baseline. For one customer that baseline is around $70, so a transaction of more than $2,000 is flagged.

Explain

An anomaly says that something is unusual, but not why. One AI_COMPLETE call to a managed model, Claude Haiku in this demo, turns the flagged activity into a finding: a large unexplained purchase from an unusual location, most likely a compromised account.

Act

Claude Sonnet 4.6 is registered once as a reusable model with CREATE MODEL. A Streaming Agent created with CREATE AGENT uses it, along with tools to freeze the account, flag a transaction or notify the customer. AI_RUN_AGENT runs the agent continuously on assessed anomalies as new events arrive, so there is no schedule to maintain.

A dashboard shows all three model steps together: Granite TTM comparing spend with expected behavior, Haiku explaining the anomaly, and the Sonnet-powered agent reasoning and acting on it. In one case, a customer changed their email address shortly before a large purchase, a classic account takeover pattern. The event was caught, explained and acted on, with an account freeze and a notification to the customer.

Getting Started with Managed Models

Managed Model Inference for LLMs and embeddings is generally available today. To run your first model, follow these three steps. The Managed Model Inference docs have more.

Step 1: In a Confluent Cloud environment on AWS, Azure or Google Cloud, create a Flink compute pool in a supported region.

Step 2: Open a SQL workspace and run SHOW MODELS to see the models available in your region.

Step 3: Call a model, for example:

AI_COMPLETE(`system.anthropic.claude-sonnet-4-6`, 'your prompt')

Try forecasting, anomaly detection, and text analysis (Early Access) by requesting access here.

A few limits to know about: Managed models are available in select regions on each cloud, and we are expanding regional coverage. Only the built-in model list is supported. For anything else, use External Model Inference. The Early Access models run on Confluent Cloud on AWS, in us-east-1, us-east-2 and us-west-1, with more clouds and regions on the roadmap. 

For a technical deep dive on the time series models, check out our Hugging Face deep dive and the IBM product blog.


Apache®, Apache Kafka®, Apache Flink®, Flink®, and the respective logos are trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by the Apache Software Foundation is implied by using these marks. All other trademarks are the property of their respective owners.

  • Bilge Zeren Aksu is a Senior Product Manager at Confluent, focused on AI capabilities under Confluent Intelligence.

Ist dieser Blog-Beitrag interessant? Jetzt teilen