containerd is failing to connect to your private registry over TLS because it doesn’t trust the registry’s certificate.
Common Causes and Fixes
-
Registry Certificate Not Trusted by
containerdHost:- Diagnosis: Check if the registry’s certificate (or its issuing CA certificate) is present in
containerd’s trusted certificate store. On Linux, this is typically/etc/containerd/certs.d/<registry_host>:<port>/ca.crt. - Fix:
- Copy the registry’s CA certificate (or the end-entity certificate if it’s self-signed) to the appropriate directory. For example, if your registry is
myregistry.local:5000and the CA cert isca.crton your local machine:sudo mkdir -p /etc/containerd/certs.d/myregistry.local:5000 sudo cp ca.crt /etc/containerd/certs.d/myregistry.local:5000/ca.crt - Restart
containerd:sudo systemctl restart containerd
- Copy the registry’s CA certificate (or the end-entity certificate if it’s self-signed) to the appropriate directory. For example, if your registry is
- Why it works:
containerdchecks the TLS certificate presented by the registry against a list of trusted Certificate Authorities. By adding the registry’s CA (or the certificate itself for self-signed cases) to this specific directory structure, you’re tellingcontainerdto trust it.
- Diagnosis: Check if the registry’s certificate (or its issuing CA certificate) is present in
-
Incorrect Certificate File Permissions:
- Diagnosis: The
containerdprocess might not have read permissions for the certificate file. - Fix: Ensure the certificate file is readable by the user
containerdruns as (oftenroot):sudo chmod 644 /etc/containerd/certs.d/myregistry.local:5000/ca.crt - Why it works: Standard file permissions prevent
containerdfrom accessing the certificate, even if it’s in the right place. Correcting permissions allowscontainerdto read the file.
- Diagnosis: The
-
Certificate Not Present in System-Wide Trust Store (Less Common for
containerdSpecific):- Diagnosis: While
containerduses its owncerts.ddirectory, some environments might rely on the system’s CA bundle. - Fix: Add the CA certificate to the system’s trust store. On Debian/Ubuntu:
On RHEL/CentOS/Fedora:sudo cp ca.crt /usr/local/share/ca-certificates/myregistry.crt sudo update-ca-certificates
Then restartsudo cp ca.crt /etc/pki/ca-trust/source/anchors/myregistry.crt sudo update-ca-trust extractcontainerd. - Why it works: This makes the certificate trusted by the operating system, and
containerdmight pick it up as a fallback or if its configuration is set to use system CAs.
- Diagnosis: While
-
Incorrect Registry Hostname/Port in
certs.dPath:- Diagnosis: The directory name must exactly match the registry hostname and port as configured in
containerd’sconfig.tomlor as used indocker pullcommands. - Fix: Verify
config.toml(usually/etc/containerd/config.toml) and your pull commands. If your registry isregistry.example.com:5443and you configured it asregistry.example.comorregistry.example.com:5000, the path will be wrong. Correct the directory name:# Example: If registry is registry.example.com:5443 sudo mv /etc/containerd/certs.d/registry.example.com:5000 /etc/containerd/certs.d/registry.example.com:5443 sudo cp ca.crt /etc/containerd/certs.d/registry.example.com:5443/ca.crt sudo systemctl restart containerd - Why it works:
containerduses the hostname and port from the registry address to look up the corresponding CA certificate in itscerts.ddirectory. A mismatch means it won’t find the correct trust anchor.
- Diagnosis: The directory name must exactly match the registry hostname and port as configured in
-
Self-Signed Certificate Issues:
- Diagnosis: If the registry uses a self-signed certificate (not issued by a public or private CA), the certificate itself must be placed in the
ca.crtfile, not just the CA. - Fix: For self-signed certificates, copy the actual certificate file (e.g.,
registry.pem) into the directory:sudo mkdir -p /etc/containerd/certs.d/myregistry.local:5000 sudo cp registry.pem /etc/containerd/certs.d/myregistry.local:5000/ca.crt sudo systemctl restart containerd - Why it works: When a certificate is self-signed, it acts as its own root of trust.
containerdneeds to be explicitly told to trust this specific certificate.
- Diagnosis: If the registry uses a self-signed certificate (not issued by a public or private CA), the certificate itself must be placed in the
-
Expired Certificate:
- Diagnosis: The registry’s TLS certificate has expired.
- Fix: Obtain a new certificate for your registry that is not expired. Update the certificate files on the registry server and, if necessary, update the
ca.crton thecontainerdhosts if the new certificate is signed by a different CA. - Why it works: TLS relies on valid, non-expired certificates for security. An expired certificate is invalid and will be rejected by any TLS client.
-
Intermediate CA Certificate Missing:
- Diagnosis: The registry’s certificate is signed by an intermediate CA, and that intermediate CA certificate is not included in the certificate chain presented by the registry, nor is it trusted by
containerd. - Fix: Ensure the registry server is configured to send the full certificate chain. If that’s not possible, you may need to bundle the intermediate CA certificate with the end-entity certificate on the registry, or add the intermediate CA’s certificate to the
ca.crtfile incontainerd’scerts.ddirectory (concatenating multiple CA certs if needed).# Concatenate your primary CA and the intermediate CA cat my_primary_ca.crt my_intermediate_ca.crt > bundled_ca.crt sudo cp bundled_ca.crt /etc/containerd/certs.d/myregistry.local:5000/ca.crt sudo systemctl restart containerd - Why it works: TLS clients verify a certificate by tracing its signature back to a trusted root CA. If an intermediate CA is missing from the chain presented by the server, the client cannot complete this validation chain unless it already trusts the intermediate CA directly.
- Diagnosis: The registry’s certificate is signed by an intermediate CA, and that intermediate CA certificate is not included in the certificate chain presented by the registry, nor is it trusted by
After fixing certificate trust issues, you might encounter unauthorized errors if the registry’s authentication configuration is incorrect or missing credentials.