etcd’s watch API doesn’t just tell you that something changed; it tells you precisely what changed, down to the key-value pair and its revision number.
Let’s see it in action. Imagine you have a simple key in etcd:
# Put a key
ETCDCTL_API=3 etcdctl put /mykey "initial_value"
Now, we want to watch for any changes to this key. We’ll use the etcdctl watch command. The crucial part is specifying the key and the --prefix flag if you want to watch all keys under a certain path. For a single key, you just specify the key.
# Watch for changes on /mykey
ETCDCTL_API=3 etcdctl watch /mykey
When you run this, your terminal will hang, waiting for events. Now, open another terminal and make a change:
# Update the key
ETCDCTL_API=3 etcdctl put /mykey "updated_value"
Back in your watching terminal, you’ll see output similar to this:
PUT,key="/mykey",value="updated_value",mod_revision="5",length=13
This output is incredibly rich. It tells you the operation (PUT), the exact key that changed (/mykey), the new value (updated_value), the revision number at which this change occurred (mod_revision="5"), and the size of the value (length=13).
What if you want to watch for deletions?
# Delete the key
ETCDCTL_API=3 etcdctl del /mykey
And on your watching terminal:
DELETE,key="/mykey",mod_revision="6",remaining_keys="0"
Notice how the output clearly indicates a DELETE operation. The remaining_keys="0" is specific to delete operations on a key that might have been a prefix itself.
The watch API is the backbone of how distributed systems like Kubernetes manage state. When a pod’s status changes, or a new service is registered, etcd emits watch events. Controllers in Kubernetes (like the deployment controller or the service controller) are constantly watching etcd for these specific events. When an event for a key they care about arrives, they react. For example, a deployment controller watches for changes to Deployment objects in etcd. If it sees a PUT event on a Deployment key, it knows it needs to reconcile the desired state (defined in the Deployment object) with the actual state of the cluster, perhaps by creating or deleting ReplicaSets.
Internally, etcd maintains a persistent connection for each client that initiates a watch. This connection is used to stream events as they happen. When a write operation occurs, etcd checks if any active watches are interested in that particular key or prefix. If so, it serializes the relevant information and sends it over the established watch stream. The mod_revision is key here because it allows clients to understand the order of operations and to resume watches from a specific point in history if their connection is interrupted.
A common misconception is that watches are a heavy operation. While they do consume resources, etcd is designed for high-throughput watch operations. The efficiency comes from etcd’s Raft consensus algorithm, which ensures that all members agree on the order of operations, and its storage engine (mvcc) that efficiently tracks key versions. The watch API leverages this by allowing clients to specify a start_revision. If a watch is established with a start_revision that is older than the current etcd cluster revision, etcd will replay the intervening events up to the current revision before starting to stream new events. This is crucial for clients that might have been offline and need to catch up on state.
When you use etcdctl watch with the --rev flag, you’re telling etcd to start watching from a specific historical revision. This is powerful for debugging or for clients that need to ensure they process all changes since a known good state. For instance, if you know your cluster was in a consistent state at revision 1000, you can start a watch from rev=1000 to see all changes that have occurred since then, even if your watch client was down at the time.
The next logical step after understanding how to watch for individual key changes is to learn how to aggregate these events or filter them based on more complex criteria, such as watching only for specific types of modifications or only for keys that match a certain pattern beyond a simple prefix.