Multi-region deployments in CockroachDB can achieve incredible availability and disaster recovery, but they fundamentally trade geographic proximity for consistency, and that trade-off manifests as latency.
Let’s see this in action. Imagine a simple SELECT statement hitting a row. In a single-region cluster, this might take 1-2ms. Now, spread that data across us-east-1, eu-west-1, and ap-southeast-2. A SELECT for a row in us-east-1 from a client in us-east-1 will still be fast. But what if that client is in eu-west-1? The request has to travel across the Atlantic, hit the CockroachDB node, read the data, and travel back. That’s a minimum of 100ms, just for network round-trip time (RTT).
CockroachDB uses a distributed consensus protocol (Raft) to ensure data consistency across regions. When you write data, that write needs to be acknowledged by a majority of replicas in the leaseholder’s region (by default, this is the region where the leaseholder for that data resides). If your leaseholder is in us-east-1 and your client is in eu-west-1, that write has to:
- Travel from
eu-west-1tous-east-1. - Be processed by the leaseholder in
us-east-1. - Be replicated to a majority of Raft followers (which might be in
us-east-1,eu-west-1, orap-southeast-2). This involves network hops between regions. - The leaseholder acknowledges the write.
- The acknowledgment travels back to the client in
eu-west-1.
This round trip, involving multiple cross-region hops for consensus, is the core of multi-region write latency. Reads are generally faster if they can hit a local replica, but if the data isn’t local or if the read requires a strong consistency guarantee that forces a cross-region read, latency increases.
The primary lever you have to manage this is the default_transaction_رى (or transaction_رى at the session level) setting, and how you configure your database schema and application.
default_transaction_رى: This setting determines how CockroachDB handles transaction retries and consistency. The default isSERIALIZABLE, which provides the strongest consistency but can incur higher latency in multi-region setups because it might require more coordination. Setting it toREAD_COMMITTEDcan significantly reduce read latency if your application can tolerate potentially stale reads. However,READ_COMMITTEDdoesn’t offer the same guarantees against race conditions.- Leaseholder Rebalancing: By default, CockroachDB tries to place leaseholders (the node that coordinates reads and writes for a given piece of data) in the region where the data was most recently written. You can influence this by setting
ALTER RANGE ... CONFIGURE ZONE ...to pin leaseholders to specific regions or to distribute them more evenly. For example,ALTER RANGE default CONFIGURE ZONE USING num_voters=3, num_replicas=5, constraints='{"+region=us-east-1": 2, "+region=eu-west-1": 2, "+region=ap-southeast-2": 1}'attempts to distribute voters across regions. This can help reduce latency for reads originating in a specific region if you can ensure the leaseholder for that data is also in that region. - Application Design: The most impactful change often comes from your application. Can you partition your data such that most transactions stay within a single region? For instance, if you have users primarily in Europe, can you ensure their data and the leaseholders for that data are predominantly in
eu-west-1? This often involves usingREGIONAL BY TABLEorREGIONAL BY STATEMENTtable locality settings.REGIONAL BY TABLEplaces all replicas for a table in a single region, determined by thesql.regional_by_table_default_regioncluster setting, effectively making that table region-local for reads and writes if the client is also in that region. - Geo-Partitioning: For very large datasets, you can geo-partition tables. This allows you to specify rules for where data resides based on column values. For example, you could have all European customer data in
eu-west-1and all North American customer data inus-east-1. This is powerful but requires careful planning of your data model and query patterns. - Read Quiescence: In some extreme multi-region scenarios, if you have a critical read that must be fast and must be consistent, you might consider temporarily quiescing writes to a region or routing traffic to a specific region where the data is known to be fresh and the leaseholder is local. This is a more advanced technique for very specific, high-priority operations.
The underlying mechanism that allows for these trade-offs is CockroachDB’s ability to replicate data across regions and use Raft consensus. When you configure REGIONAL BY TABLE for a table and set sql.regional_by_table_default_region to us-east-1, CockroachDB ensures that all replicas for that table are located within us-east-1. A read originating from us-east-1 for data in this table will hit a local replica with minimal network latency. Writes to this table, coordinated by the leaseholder also in us-east-1, will also experience lower latency compared to a scenario where the leaseholder is in a different region. This is because the Raft consensus group for that table’s data is contained entirely within us-east-1, avoiding cross-region network hops for the consensus process itself.
The next challenge is understanding how to manage distributed transactions that span multiple regional tables or involve operations requiring inter-region coordination.