CockroachDB’s resilience isn’t magic; it’s a deliberate design choice that allows you to achieve aggressive Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO) by leveraging its distributed nature and replication capabilities, but only if you understand how to orchestrate it.
Let’s see what that looks like in practice. Imagine a cluster distributed across three AWS availability zones. We’ve got three nodes: node1 in us-east-1a, node2 in us-east-1b, and node3 in us-east-1c. Each node runs a CockroachDB instance and is part of a replication group for our critical orders table.
-- Show replication factor for a table
SHOW ZONE CONFIGURATION FOR TABLE orders;
This might output:
range_max_bytes | range_min_bytes | gc_percent | num_replicas | lease_max_stale_age | lease_renewal_interval | follower_reads | schema_change_policy | num_voters
-----------------+-----------------+------------+--------------+---------------------+------------------------+--------------+----------------------+-------------
536870912 | 67108864 | 0.25 | 3 | 10s | 5s | false | 2023-10-27T10:00:00Z | 3
(1 row)
The num_replicas: 3 here means that for every "range" (CockroachDB’s internal unit of data partitioning), there are three copies of that data spread across the cluster. The num_voters: 3 is key: all three replicas participate in consensus for writes. This is your primary defense against data loss (RPO) and a critical component for availability (RTO).
Now, let’s simulate a disaster: us-east-1a goes offline. CockroachDB’s Raft consensus protocol is designed to tolerate the loss of a minority of replicas. With num_voters: 3, we can tolerate the loss of one replica. When node1 in us-east-1a fails, the remaining two replicas on node2 and node3 can still achieve a quorum (2 out of 3). Writes can continue, and reads can still be served. Your RPO is effectively zero for data written before the failure, and your RTO is minimal because the cluster remains operational.
The real magic for disaster recovery scenarios, however, comes from how you prepare for larger outages, like an entire region failing. To handle this, you’d set up a multi-region cluster. Let’s say we have a primary cluster in us-east-1 and a secondary cluster in us-west-2.
# cockroach.yaml for node in us-east-1
clusterID: my-prod-cluster
locality: "aws:us-east-1:us-east-1a"
listen-addr: "0.0.0.0:26257"
http-addr: "0.0.0.0:8080"
join: "node1.us-east-1.example.com:26257,node2.us-east-1.example.com:26257"
max-sql-memory: "4Gi"
# ... other config ...
---
# cockroach.yaml for node in us-west-2
clusterID: my-prod-cluster
locality: "aws:us-west-2:us-west-2a"
listen-addr: "0.0.0.0:26257"
http-addr: "0.0.0.0:8080"
join: "node1.us-west-2.example.com:26257,node2.us-west-2.example.com:26257"
max-sql-memory: "4Gi"
# ... other config ...
You’d then configure replication zones to span these regions. A common pattern for DR is to have a global zone with num_replicas: 3 or 5 and a us-east-1 specific zone with num_replicas: 3 and default_replication_target: aws:us-east-1. For DR, you’d add a us-west-2 zone:
-- Create a US East zone
CREATE ZONE CONFIGURATION for US_EAST_1_ZONE
CONSTRAINED BY { constraints: { 'cockroach.region': 'us-east-1' } }
PRIMARY REGION "us-east-1"
REGIONS "us-east-1"
NUM_REPLICAS = 3;
-- Create a US West zone for DR
CREATE ZONE CONFIGURATION for US_WEST_2_ZONE
CONSTRAINED BY { constraints: { 'cockroach.region': 'us-west-2' } }
PRIMARY REGION "us-west-2"
REGIONS "us-west-2"
NUM_REPLICAS = 3;
-- Apply to your critical table
ALTER TABLE orders EXPERIMENT using 'zone_config_for_replication'
CONFIGURE ZONE FOR TABLE orders
(
PRIMARY_REGION = "us-east-1",
REGIONS = ["us-east-1", "us-west-2"],
NUM_REPLICAS = 5, -- Increased for cross-region resilience
DEFAULT_REPLICATION_TARGET = "aws:us-east-1",
CONSTRAINT_AWARE_RANGE_DISTRIBUTION = "enabled"
);
With NUM_REPLICAS = 5 and REGIONS = ["us-east-1", "us-west-2"], CockroachDB will try to place 3 replicas in us-east-1 and 2 in us-west-2 (or vice-versa, depending on the primary region and node availability). This means that even if the entire us-east-1 region is lost, the remaining 2 replicas in us-west-2 can still form a quorum, and your database remains available. Your RPO is still near zero, and your RTO is the time it takes for the remaining replicas to elect a leader and for your application to re-route traffic, which can be minutes.
The most surprising thing about CockroachDB’s disaster recovery is that the application often needs to be aware of the multi-region setup for optimal failover. While CockroachDB itself can remain available, your application’s connection strings and load balancing might need to be dynamic to immediately target the healthy region after a failure.
For instance, if your application connects to cockroachdb.example.com, you’ll want this DNS record to point to the available region. This is typically achieved using a DNS provider that supports health checks and automatic failover based on the health of your database endpoints in each region. When us-east-1 becomes unavailable, the DNS provider should detect this and shift traffic to us-west-2.
To achieve the fastest RTO, you’d typically run your application with a multi-region connection string or use a load balancer that can intelligently route traffic. For example, an application might have a primary connection to us-east-1 and a secondary connection to us-west-2. If the primary fails, the application code (or an external orchestrator) switches to the secondary. CockroachDB’s default_replication_target in the zone configuration helps the database place data, but the application’s routing is key for end-to-end RTO.
The next step in mastering CockroachDB resilience is understanding how to use CREATE ZONE CONFIGURATION with SUBZONES for granular control over data placement and replication across different regions or cloud providers.