Project Metamorphosis: Unveiling the next-gen event streaming platformLearn More
July | Infinite

Infinite retention for Apache Kafka® with Confluent

  • Transform Kafka into a central nervous system for all events
  • Infinitely retain events without pre-provisioning storage
  • Reduce storage costs up to 70% in Confluent Platform

Infinite resources

July 1

Infinite Storage in Confluent Cloud for Apache Kafka

Learn what unlimited storage unlocks in Confluent Cloud

Aug 4

Infinite Retention in Confluent Cloud

See how to set up infinite storage and retention in Confluent Cloud

Why infinite matters

Transform Kafka into your central nervous system for all events

In traditional data architectures, storage systems focus on accumulating past records while messaging services are waiting to process future events. On top of this data layer, there may be hundreds of in-house systems, SaaS applications, or microservices both generating and consuming data. Data integrity and correctness must be maintained across all these systems, which is a huge operational burden and business risk as organizations and apps must make quick decisions on what they believe is the most accurate and up to date version of the data while retaining what is needed for regulatory compliance. Confluent's event streaming solutions allow you to:

  • With no limits on storage, retain all historical and current events in Kafka as long as you want
    • Set retention at the topic level to be as long as you want without scaling additional compute resources to handle the storage
    • Reduce tech debt and downtime risk with a central source of truth for all events across your entire ecosystem
  • Democratize access to your data with all events in one place and eliminate the need for complex data engineering
    • Maintain data compatibility with Schema Registry and enforce data integrity with Schema Validation to build an infinite and immutable central source of truth for all streaming events
    • Keep the flexibility to use your programming language of choice and transform infinite amounts of data in Kafka using clients such as C, .NET, Java, GO, Python and more
  • Reduce business risk by setting retention policies that help maintain regulatory compliance and providing a persistent log of all events for compliance audits
Infinitely retain events without ever worrying about storage

Provisioning massive Kafka clusters and retaining the right amount of data in the cloud with Apache Kafka is complex. Setting it up properly to handle GBps throughputs and petabytes of data can turn into months of work that push delivery timelines delaying business go-to-market. Similarly, businesses that don't need GBps scale or petabytes of storage at the beginning but face unexpected spikes or storage needs risk causing downtime or degrading cluster performance, which can lead to customer frustration. Confluent Cloud allows you to:

  • Never worry about Kafka storage again in Confluent Cloud with elastically scalable storage that automatically grows with your traffic
    • Avoid downtime and data loss by auto-scaling storage based on retention policies and traffic
  • Only pay for storage used with Confluent Cloud instead of paying for over provisioned storage volumes attached to Kafka brokers
    • Only pay for retained data, not pre-provisioned storage capacity
    • Decouple storage and compute to avoid unnecessary compute costs for high storage use cases
Reduce storage costs up to 70% with tiered storage

Storing historical data on self-managed Kafka clusters quickly increases your infrastructure costs because you have to over provision storage to prevent downtime and deploy larger clusters to avoid hitting storage limitations on brokers. It becomes cost-prohibitive to retain all the data you need in Kafka, causing organizations to glue together disjointed architectures that end up costing more money and siphoning off more engineering resources. Confluent Platform allows you to:

  • Send long term storage to cost-efficient object storage, such as S3 and GCS, using Tiered Storage in Confluent Platform
    • Expand cluster storage in Confluent Platform without any additional resource provisioning when you offload data to 3rd party object storage
  • Reduce partition rebalancing time for self-managed clusters from hours to seconds using Tiered Storage to minimize the size of your partitions on brokers

Benefits

Scalable Storage
Infinite Retention
Data Compatibility
Confluent Cloud

Decouple compute and storage in Kafka to elastically scale storage

Infinitely retain events in Kafka only paying for storage used

Enforce consistent data formats for app development

Confluent Platform

Decouple compute and storage in Kafka to elastically scale storage

Infinitely retain events in Kafka by offloading data to object storage

Programmatically enforce data schemas at the broker-level

Features

  • July

    Infinite retention

    Store data in Basic and Standard clusters on AWS for as long as it is needed

  • Available

    Schema Registry

    Enforce consistent data formats to enable app development compatibility with Avro, Protobuf & JSON

  • July

    Schema validation

    Enable centralized policy enforcement and data governance within your event streaming platform

  • Available

    Multi-language development

    Enable all developers to leverage Kafka throughout the organization

  • Preview

    Tiered storage

    Retain infinite data in Kafka by offloading older topic data to cost-effective object storage

  • Available

    Schema Registry

    Enforce consistent data formats to enable app development compatibility with Avro, Protobuf & JSON

  • Available

    Schema validation for Schema Registry

    Programmatically validate and enforce schemas at the broker-level

  • Available

    Multi-language development

    Enable all developers to leverage Kafka throughout the organization

Try it out

Cloud
Fully managed service

Deploy in minutes. Pay as you go. Try a serverless Kafka experience.

Platform
Self-managed software

Experience the power of our enterprise-ready platform through our free download.

*Receive $200 off your bill each calendar month for the first three months

More Project Metamorphosis releases

Cost Effective

How do you keep the costs of running Kafka low and your best people focused on critical projects driving competitive advantage and revenue?

Read More

Elastic Scaling

How do you quickly scale Kafka to keep mission-critical apps running with no lag or downtime - and without over-provisioning expensive resources?

Read More

Sign Up Now

Start your 3-month trial. Get up to $200 off on each of your first 3 Confluent Cloud monthly bills

New signups only.

By clicking “sign up” above you understand we will process your personal information in accordance with our Privacy Policy.

By clicking "sign up" above you agree to the Terms of Service and to receive occasional marketing emails from Confluent. You also understand that we will process your personal information in accordance with our Privacy Policy.

Get Confluent Cloud

Get up to $200 off on each of your first 3 Confluent Cloud monthly bills


Choose one sign-up option below

Marketplaces

  • AWS
  • Azure
  • Google Cloud

  • Billed through your Cloud provider*
  • Stream only on 1 cloud
*Billing admin role needed

Marketplaces

  • Billed through your Cloud provider*
  • Stream only on 1 cloud
  • Billing admin role needed

*Billing admin role needed

Confluent


  • Pay with a credit card
  • Stream across multiple clouds

Confluent

  • Pay with a credit card
  • Stream across multiple clouds

By clicking “sign up” above you understand we will process your personal information in accordance with our Privacy Policy.

By clicking "sign up" above you agree to the Terms of Service and to receive occasional marketing emails from Confluent. You also understand that we will process your personal information in accordance with our Privacy Policy.

Free Forever on a Single Kafka Broker
i

The software will allow unlimited-time usage of commercial features on a single Kafka broker. Upon adding a second broker, a 30-day timer will automatically start on commercial features, which cannot be reset by moving back to one broker.

Select Deployment Type
Manual Deployment
  • tar
  • zip
  • deb
  • rpm
  • docker
or
Auto Deployment
  • kubernetes
  • ansible

By clicking "download free" above you understand we will process your personal information in accordance with our Privacy Policy.

By clicking "download free" above, you agree to the Confluent License Agreement and to receive occasional marketing emails from Confluent. You also agree that your personal data will be processed in accordance with our Privacy Policy.

This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising, and analytics partners.