The CAP theorem doesn’t actually say you must choose between Consistency and Availability; it states that in the presence of a network partition, you cannot simultaneously guarantee both.
Let’s see this in action. Imagine a simple distributed key-value store. We have two nodes, Node A and Node B, and a client.
Client -> Node A (data: { "key": "value1" })
Client -> Node B (data: { "key": "value1" })
Now, imagine a network partition occurs, meaning Node A and Node B can’t talk to each other.
Scenario 1: Client writes to Node A
Client -> Node A (write: { "key": "new_value" })
At this point:
- Node A has:
{ "key": "new_value" } - Node B has:
{ "key": "value1" }
If a client then tries to read from Node B, it will get the stale value1. This is a failure of Consistency (all nodes don’t see the same data at the same time). However, Node B was still Available to serve the read request. If the client had written to Node A and Node A was available, it would have served that write.
Scenario 2: Client reads from Node B
Client -> Node B (read: "key")
Node B is Available and serves value1. However, if the client previously wrote new_value to Node A (and Node A is now partitioned away), Node B doesn’t know about it. This is a failure of Consistency. If the client had been able to reach Node A, it might have gotten the newer value.
Scenario 3: The system chooses to be Consistent
If the system must be consistent, when the partition occurs and a client tries to write to Node A, Node A would have to refuse the write or block until it can confirm Node B is back online and synchronized.
Client -> Node A (write: { "key": "new_value" })
// Node A cannot confirm with Node B due to partition.
// Node A must either:
// 1. Refuse the write (becomes unavailable for this operation).
// 2. Block indefinitely until partition heals (effectively unavailable).
In this case, the system prioritizes Consistency by sacrificing Availability. No client can successfully write to Node A during the partition.
Scenario 4: The system chooses to be Available
If the system prioritizes Availability, when the client writes to Node A, Node A accepts the write and responds immediately.
Client -> Node A (write: { "key": "new_value" })
// Node A accepts the write and responds.
Node A is Available. However, since Node B is partitioned, it doesn’t receive the update. If a client then reads from Node B, it will get the old value1. This is a sacrifice of Consistency.
The theorem’s core insight is that when a network partition (P) happens, you must choose between C and A. You can have CA (if no partition), CP (Consistent and Partition Tolerant), or AP (Available and Partition Tolerant). You cannot have CAP (Consistent, Available, and Partition Tolerant) simultaneously in a distributed system.
Think about how this plays out in real-world systems. A distributed SQL database like CockroachDB or Google Spanner often leans towards CP. If a network split occurs between data centers, they might choose to make the affected region unavailable for writes or reads rather than serve potentially inconsistent data. They ensure that once a write is acknowledged, it is durable and visible to all subsequent reads, even if it means temporarily halting operations in a partitioned segment.
Conversely, systems like Cassandra or DynamoDB often lean towards AP. During a partition, they will allow writes to proceed on both sides. Reads might return older data from one side until the partition heals and data can be reconciled. The trade-off is that clients might read stale data, but the system remains operational. The reconciliation process (often using mechanisms like read repair or anti-entropy protocols) happens in the background once the partition is resolved.
Many systems don’t strictly adhere to CP or AP but rather implement tunable consistency levels. For example, in a system like Cassandra, you can specify the consistency level for reads and writes. A QUORUM read might require a majority of nodes to respond, offering a stronger guarantee than a ONE read (which only needs one node), but still not the strong consistency of a globally synchronous system. This allows developers to choose the specific trade-off for their application’s needs.
The most surprising part for many is how network partitions are far more common and insidious than outright node failures. A slow network link, a misconfigured firewall, or a router issue can all effectively create a partition, cutting off communication between parts of your distributed system. This means the CAP theorem isn’t a hypothetical scenario for rare catastrophic failures; it’s a constant consideration for any distributed system that spans multiple machines or network segments.
The next logical step after understanding CAP is exploring how different distributed consensus algorithms (like Paxos or Raft) provide mechanisms to achieve strong consistency when possible and how systems manage eventual consistency when Availability is prioritized.