CockroachDB nodes can be decommissioned gracefully, but the process is about more than just shutting down a process; it’s about ensuring the cluster’s resilience and data integrity throughout the operation.
Let’s see a cluster in action, then break it down. Imagine we have a three-node cluster, node1, node2, and node3. We want to remove node3.
First, we check the cluster health.
# On any node in the cluster
./cockroach node status --format=json
This might show:
[
{
"desc": {
"nodeID": 1,
"address": "node1:26257",
"locality": {
"value": "region=us-east-1,zone=us-east-1a"
},
"sqlAddr": "node1:26257",
"grpcAddr": "node1:26257",
"httpAddr": "node1:8080",
"startedAt": "2023-10-27T10:00:00Z",
"buildInfo": {
"goVersion": "go1.21.0",
"fillbv": "v22.2.7",
"platform": "linux amd64",
"tag": "v22.2.7"
},
"isLive": true,
"cloudProvider": "aws"
},
"stats": {
"replicas": 150,
"ranges": 150,
"queries": 10000,
"bytes": 100000000
}
},
{
"desc": {
"nodeID": 2,
"address": "node2:26257",
"locality": {
"value": "region=us-east-1,zone=us-east-1b"
},
"sqlAddr": "node2:26257",
"grpcAddr": "node2:26257",
"httpAddr": "node2:8080",
"startedAt": "2023-10-27T10:00:05Z",
"buildInfo": {
"goVersion": "go1.21.0",
"fillbv": "v22.2.7",
"platform": "linux amd64",
"tag": "v22.2.7"
},
"isLive": true,
"cloudProvider": "aws"
},
"stats": {
"replicas": 150,
"ranges": 150,
"queries": 11000,
"bytes": 101000000
}
},
{
"desc": {
"nodeID": 3,
"address": "node3:26257",
"locality": {
"value": "region=us-east-1,zone=us-east-1c"
},
"sqlAddr": "node3:26257",
"grpcAddr": "node3:26257",
"httpAddr": "node3:8080",
"startedAt": "2023-10-27T10:00:10Z",
"buildInfo": {
"goVersion": "go1.21.0",
"fillbv": "v22.2.7",
"platform": "linux amd64",
"tag": "v22.2.7"
},
"isLive": true,
"cloudProvider": "aws"
},
"stats": {
"replicas": 150,
"ranges": 150,
"queries": 9500,
"bytes": 99000000
}
}
]
The key is to gracefully "drain" the node, meaning we tell it to stop accepting new connections and to migrate its data replicas to other nodes. This is orchestrated by setting the node’s join-flag to 0.
The problem this solves is preventing data loss and maintaining cluster availability during node removal. CockroachDB, being a distributed database, relies on replication for fault tolerance. Simply shutting down a node without preparation can leave the cluster in a state where it cannot form quorum for certain data ranges, or worse, lose data if that node was the last replica for some ranges.
The internal mechanism involves several steps:
- Draining: The node is instructed to stop accepting new client connections and to cease participating in new Raft consensus rounds.
- Replication Migration: Existing replicas on the draining node are identified. CockroachDB’s replication manager then initiates the process of creating new replicas on other healthy nodes and removing the old ones from the draining node. This is a background process, and it’s crucial to let it complete before physically removing the node.
- Decommissioning: Once all replicas have been migrated, the node is effectively empty and can be safely removed from the cluster’s metadata.
To initiate the draining process, you use the cockroach node decommission command. You’ll need the nodeID of the node you wish to decommission. In our example, this is nodeID: 3.
# On any node in the cluster
./cockroach node decommission 3 --insecure
The --insecure flag is for development/testing; in production, you’d use --certs-dir=/path/to/certs.
After running this, the node’s status will change. You can monitor this with cockroach node status. You’ll see a flag indicating it’s being decommissioned. This is the critical phase where data is migrated. The cluster’s replication layer will work to ensure that for every range that had a replica on node 3, there are now enough replicas on other nodes (node 1 and node 2 in this case) to maintain the replication factor.
The cluster’s replication factor is a fundamental setting. For example, if your cluster is configured with a replication factor of 3 (the default), then each piece of data is stored on three different nodes. When you decommission one node, the cluster needs to create a new replica on a remaining node to bring the total number of replicas for that data back up to 3.
How to verify completion:
You can continuously monitor the node status command. The isLive field for node 3 will eventually change from true to false as it shuts down. More importantly, the stats for replicas and ranges on the remaining nodes (1 and 2) should increase to compensate for the data that was on node 3, and the stats for node 3 will eventually disappear entirely from the node status output. You can also check the cluster’s "Range Report" in the DB Console (Admin UI) to see replica distribution.
Once the node is fully drained and its replicas are migrated, you can stop the cockroach process on that node and then physically remove it.
# On node3
pkill cockroach
And then, you can remove the node from the cluster’s registry if it doesn’t disappear automatically. This is typically done by running a cockroach node deregister <nodeID> command from one of the remaining active nodes.
# On node1 or node2
./cockroach node deregister 3 --insecure
The most surprising true thing about this process is that CockroachDB doesn’t simply "remove" a node’s data when you decommission it. Instead, it actively rebalances the data and ensures that the cluster’s replication factor is maintained by creating new replicas on the remaining nodes before the old node is fully out of the picture. This proactive migration is what guarantees no data loss.
After successfully decommissioning a node, the next challenge you’ll face is understanding how to adjust the cluster’s replication factor if the removal of a node changes your availability requirements.