The most surprising thing about updating EKS worker node AMIs without downtime is that it’s not about updating the existing nodes, but about replacing them with new nodes running the new AMI.
Let’s see this in action. Imagine we have a running EKS cluster and we want to update our worker nodes from Amazon Linux 2 AMI ami-0a117fc07f873f778 to a newer version, say ami-0d987214576230946.
First, we need to create a new Auto Scaling Group (ASG) and Launch Template (LT) that uses the new AMI.
Here’s how you’d create a Launch Template. Notice the ImageId is our new AMI:
aws ec2 create-launch-template --launch-template-name my-new-worker-template --launch-template-data \
'{"ImageId": "ami-0d987214576230946", "InstanceType": "t3.medium", "NetworkInterfaces": [{"DeviceIndex": 0, "AssociatePublicIpAddress": false, "Groups": ["sg-0123456789abcdef0"], "SubnetId": "subnet-0123456789abcdef0"}], "TagSpecifications": [{"ResourceType": "instance", "Tags": [{"Key": "eks:cluster-name", "Value": "my-eks-cluster"}, {"Key": "eks:nodegroup-name", "Value": "my-nodegroup"}]}]}'
Next, we create a new ASG pointing to this new Launch Template. Crucially, this ASG should not have a desired capacity that overlaps with your existing ASG’s capacity. We’ll start it with a desired capacity of 0 and scale it up gradually.
aws autoscaling create-auto-scaling-group --auto-scaling-group-name my-new-worker-asg --launch-template \
LaunchTemplateName=my-new-worker-template,Version=1 --min-size 0 --max-size 5 --desired-capacity 0 \
--vpc-zone-identifier "subnet-0123456789abcdef0,subnet-1234567890abcdef1" --tags \
'ResourceId=my-new-worker-asg,ResourceType=auto-scaling-group,Key=eks:cluster-name,Value=my-eks-cluster' \
'ResourceId=my-new-worker-asg,ResourceType=auto-scaling-group,Key=eks:nodegroup-name,Value=my-nodegroup'
Now, you have a new ASG ready to launch nodes with the updated AMI, but it’s not running any instances yet. Your existing nodes are still running the old AMI.
The core of the "no downtime" strategy is a rolling replacement. You’ll manually increase the desired-capacity of your new ASG by one, and simultaneously decrease the desired-capacity of your old ASG by one.
As the new ASG launches an instance with the new AMI, EKS will register it. Kubernetes will then schedule pods onto this new node. Once the new node is healthy and ready, you drain the old node. Draining evicts pods gracefully, allowing them to reschedule onto other available nodes (including the new ones you just added). After the old node is drained, you can terminate it (or let the old ASG scale down).
You repeat this process: add one to new ASG desired capacity, subtract one from old ASG desired capacity, wait for the new node to be ready, drain an old node, terminate it. You do this one node at a time, or in small batches, until all your old nodes are replaced.
The key is that Kubernetes never loses capacity. When you add a new node, pods eventually move to it. When you drain an old node, its pods are rescheduled before it’s taken offline.
What most people don’t realize is that the eks:cluster-name and eks:nodegroup-name tags on your ASG and EC2 instances are how EKS identifies which nodes belong to which managed nodegroup. When you create new resources, you must copy these tags over. If you forget, EKS won’t recognize the new nodes as part of your managed nodegroup, and they won’t be managed correctly (e.g., kubelet configuration won’t be applied, and kubectl get nodes might show them with a NotReady status).
Once all old nodes are replaced, you delete the old ASG and its associated Launch Template.
The next thing you’ll likely encounter is managing updates for your Kubernetes version itself, which involves a similar, but more involved, rolling replacement process for the control plane and nodes.