Confluent
Disaster Recovery for Multi-Datacenter Apache Kafka Deployments
Confluent Platform

Disaster Recovery for Multi-Datacenter Apache Kafka Deployments

Yeva Byzek

Datacenter downtime and data loss can result in businesses losing a vast amount of revenue or entirely halting operations. To minimize the downtime and data loss resulting from a disaster, enterprises create business continuity plans and disaster recovery strategies.

A disaster recovery plan often requires multi-datacenter Apache Kafka® deployments where datacenters are geographically dispersed. If disaster strikes—catastrophic hardware failure, software failure, power outage, denial of service attack, or any other event that causes one datacenter to completely fail—Kafka continues running in another datacenter until service is restored. Here is a Confluent multi-datacenter reference architecture:

Disaster Recovery

The details of your design will vary depending on your business requirements. You may be considering an active-passive design (one-way data replication between Kafka clusters), active-active design (two-way data replication between Kafka clusters), client applications that read from just their local cluster or both local and remote clusters, service discovery mechanisms to enable automated failovers, geo locality offerings, etc.

Confluent Replicator is the key to any of these multi-datacenter designs. It manages multiple Kafka deployments and provides a centralized configuration of cross-datacenter replication. It reads data from the origin cluster and writes that data to the destination cluster. As topic metadata or partition count changes in the origin cluster, it replicates the changes in the destination cluster. New topics are automatically detected and replicated to the destination cluster.

In our white paper “Disaster Recovery for Multi-Datacenter Apache Kafka Deployments“, we discuss multi-datacenter designs and building blocks:

  • Data replication
  • Timestamp preservation
  • Preventing cyclic repetition of topics
  • Resetting consumer offsets
  • Centralized schema management

This white paper is a practical guide for configuring multiple Kafka clusters so that if a disaster scenario strikes, you have a working plan for failover, failback, and ultimately successful recovery. Please download the white paper to follow these recommendations to strengthen your disaster recovery plan.

Additional Resources

 

Subscribe to the Confluent Blog

Subscribe

More Articles Like This

Streaming Applications
Yeva Byzek

Getting Your Feet Wet with Stream Processing – Part 2: Testing Your Streaming Application

Yeva Byzek .

Part 1 of this blog series introduced a self-paced tutorial for developers who are just getting started with stream processing. The hands-on tutorial introduced the basics of the Kafka Streams ...

Design a multi-datacenter solution
Yeva Byzek

3 Ways to Prepare for Disaster Recovery in Multi-Datacenter Apache Kafka Deployments

Yeva Byzek .

Imagine: Disaster strikes—catastrophic hardware failure, software failure, power outage, denial of service attack or some other event causes one datacenter with an Apache Kafka® cluster to completely fail. Yet Kafka ...

Streaming in the Clouds: Where to Start
Priya Shivakumar

Streaming in the Clouds: Where to Start

Priya Shivakumar .

Only a few years ago, when someone said they had a “cloud-first strategy,” you knew exactly who their new preferred vendor was. These days, however, the story is a lot ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Try Confluent Platform

Download Now

We use cookies to understand how you use our site and to improve your experience. Click here to learn more or change your cookie settings. By continuing to browse, you agree to our use of cookies.