containerd’s default behavior is to trust any image it can pull, which is a massive security hole waiting to be exploited.
Let’s see what that looks like in practice. We’ll use cosign, a popular tool for signing and verifying container images, to demonstrate.
First, we need an image and a signature. We’ll use a simple alpine image and sign it with cosign.
# Make sure you have cosign installed: https://github.com/sigstore/cosign#installation
export IMAGE_NAME="ghcr.io/your-dockerhub-username/alpine-signed:latest" # Replace with your registry/username
export IMAGE_DIGEST=$(docker pull alpine:latest | grep "Digest:" | awk '{print $2}')
export IMAGE_URL="${IMAGE_NAME}@${IMAGE_DIGEST}"
# Sign the image (this will prompt for a password if you're using a password-protected key)
cosign sign --key cosign.key $IMAGE_URL
# Push the image and its signature
docker push $IMAGE_NAME
docker push --annotation "org.opencontainers.image.sig" "cosign" $IMAGE_NAME # This pushes the signature as an annotation
Now, let’s configure containerd to enforce signature verification. This is done via the config.toml file, usually located at /etc/containerd/config.toml.
We need to add a plugins.cri.containerd.runtimes.runc.options.ImageDecryption section and configure the signaturePolicy.
# In /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
ContainerdSnapshotters = ["overlayfs"]
ImageDecryption = true # Enable image decryption if you use encrypted images
[plugins."io.containerd.grpc.v1.cri".containerd.seccomp.profiles]
# ... other profiles ...
[plugins."io.containerd.grpc.v1.cri".containerd.assets]
# ... assets configuration ...
[plugins."io.containerd.grpc.v1.cri".containerd.registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".containerd.registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io"]
# THIS IS THE IMPORTANT PART FOR SIGNATURE VERIFICATION
[plugins."io.containerd.grpc.v1.cri".containerd.policy]
[plugins."io.containerd.grpc.v1.cri".containerd.policy.v1]
[[plugins."io.containerd.grpc.v1.cri".containerd.policy.v1.signature_policy]]
# Match all images
all = true
# Require the 'cosign' signature type
required = ["cosign"]
After saving config.toml, restart containerd:
sudo systemctl restart containerd
Now, if you try to pull an image that hasn’t been signed with cosign (or has been signed by someone you don’t trust), containerd will reject it.
Let’s test this. If you try to pull alpine:latest (which we didn’t sign with cosign in this example, only our signed version), you’ll get an error.
# This will fail if your policy requires cosign signatures for all images
ctr images pull alpine:latest
You’ll see an error similar to:
failed to pull image "docker.io/library/alpine:latest": failed to resolve reference "docker.io/library/alpine:latest": failed to authorize: failed to fetch OCI descriptor: failed to fetch signature: manifest invalid: failed to verify signature: cosign: public key not found
This error message tells us that containerd did check for signatures because of our policy, but it couldn’t find a valid cosign signature for the image it was trying to pull. This is exactly what we want!
To successfully pull our signed image, containerd needs to trust the public key used to sign it. This is configured in containerd’s policy. We can tell containerd to trust a specific key by adding it to the policy.
# In /etc/containerd/config.toml, within the signature_policy section:
[[plugins."io.containerd.grpc.v1.cri".containerd.policy.v1.signature_policy.identity]]
# Match our specific image
dockerpolicy = "ghcr.io/your-dockerhub-username/alpine-signed" # Replace with your image name prefix
# Specify the trusted public key (base64 encoded)
# You can get this from `cosign public-key --key cosign.pub`
# Example:
public_key = "-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAw..." # Replace with your actual public key
After updating config.toml and restarting containerd again, you should now be able to pull your signed image:
ctr images pull ghcr.io/your-dockerhub-username/alpine-signed:latest
This process allows you to enforce a strict supply chain for your container images, ensuring that only images signed by trusted parties can be deployed. The cosign tool integrates seamlessly with containerd’s policy engine, providing a robust mechanism for verifying image integrity and authenticity.
The surprising part is that containerd doesn’t just verify signatures; it enforces a policy that dictates which signatures are acceptable. You can have multiple policies for different image registries or even individual images, specifying which signing identities (keys) are trusted for each.
Let’s look at how containerd actually finds these signatures. When you pull an image, containerd doesn’t just look at the image manifest itself. It queries the registry for any associated "signatures" or "attestations" that match the signing methods specified in your policy. For cosign, this typically means looking for signed manifests with specific annotations (like org.opencontainers.image.sig) or referring to separate signature blobs in the registry. If the required signatures are found and they successfully verify against the trusted public keys in your policy, the image is allowed. If not, the pull fails.
When containerd pulls an image, it first fetches the image manifest. Then, based on the signaturePolicy in your config.toml, it looks for associated signatures. If required = ["cosign"] is set, containerd will specifically search for signatures in a format cosign understands. This involves querying the registry for any artifacts linked to the image manifest that are identified as cosign signatures. If containerd finds signatures, it then attempts to verify them against the public_key entries defined in the identity section of your policy that match the image’s registry path. If all required signatures are present and valid according to the policy, the image is considered trusted and pulled successfully. Otherwise, the pull operation is aborted with an error.
The next step is to integrate this into your CI/CD pipeline, automating the signing of images before they are pushed to your registry.