The most surprising thing about upgrading AKS node images without downtime is that it’s not about replacing nodes, but about persuading the cluster to start using new nodes while the old ones gracefully shut down.
Let’s see this in action. Imagine you have a deployment of nginx.
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.25.3
ports:
- containerPort: 80
This deployment is running on nodes with an older image. We want to upgrade to a newer node image, say ubuntu:22.04-20240422.1.
Here’s the core mechanism: AKS uses a rolling upgrade strategy for node pools. When you initiate an upgrade, AKS doesn’t touch your running pods directly. Instead, it provisions new nodes with the updated image, adds them to the node pool, and then, one by one, cordons and drains the old nodes. Cordoning marks a node as unschedulable, preventing new pods from landing on it. Draining evicts existing pods, allowing them to be rescheduled onto the new, healthy nodes.
The magic lies in Kubernetes’ built-in resilience. Your Deployment object, with its replicas: 3 and selector: matchLabels: app: nginx, tells the control plane: "I always want 3 pods with the label app: nginx running." As an old node is drained, its pods are terminated. The Kubernetes scheduler then sees that the desired replica count isn’t met and schedules those pods onto the available new nodes. If you’re using a StatefulSet, the process is similar but respects pod ordering and unique identities. For DaemonSets, the behavior is slightly different; new nodes will get the daemon pods, and old nodes will have them removed as they drain.
To upgrade a node pool, you typically use the Azure CLI. First, you need to identify your node pool.
az aks nodepool list --resource-group myResourceGroup --cluster-name myAKSCluster -o table
Let’s say you see agentpool with a NodeImageVersion that’s not the latest. To upgrade it to the latest supported Ubuntu image:
az aks nodepool upgrade --resource-group myResourceGroup --cluster-name myAKSCluster --name agentpool --node-image-version $(az aks nodepool get-upgrades --resource-group myResourceGroup --cluster-name myAKSCluster --nodepool-name agentpool -o json | jq -r '.recommendedUpgrade')
This command fetches the recommended upgrade version and applies it to your specified agentpool. AKS then orchestrates the rolling upgrade. You can monitor the progress:
az aks nodepool show --resource-group myResourceGroup --cluster-name myAKSCluster --name agentpool --query provisionState
You’ll see this transition from Upgrading back to Succeeded. During the upgrade, you can also observe pods being terminated and recreated using kubectl get pods -o wide and kubectl get nodes. You’ll notice pods disappearing from old nodes and reappearing on new ones.
The key configuration levers here are the maxUnavailable and maxSurge parameters on your Deployment or StatefulSet (though for node pool upgrades, AKS manages this implicitly by upgrading one node at a time by default). maxUnavailable defines how many pods can be unavailable during the upgrade, and maxSurge defines how many extra pods can be created above the desired count. For node pool upgrades, AKS’s default behavior is to upgrade one node at a time, ensuring maxUnavailable is effectively 1 for that node’s pods, and then it adds a new node to compensate, keeping your service available.
What most people overlook is that the node image upgrade itself is distinct from the Kubernetes version upgrade. You can upgrade node images independently of the control plane’s Kubernetes version, and vice-versa. This allows for more granular patching and security updates without requiring a full cluster Kubernetes version jump. The node image contains the OS, kubelet, container runtime, and other essential components that run on the node itself.
Once your node images are upgraded, the next logical step is to consider upgrading the Kubernetes version of your control plane and nodes to leverage new features and security patches.