Articles

Diagnose and Fix Slow Image Pulls in containerd (2026)

April 16, 2026 5 min read

The problem is that containerd is failing to pull container images in a timely manner, often timing out or taking excessively long, which directly impacts application deployment and startup times.

Common Causes and Fixes for Slow Image Pulls

  1. Network Throughput/Latency: The most frequent culprit is a saturated or high-latency network link between your containerd host and the container registry.

    • Diagnosis:
      • Run iperf3 between your containerd host and a server with similar network characteristics to your registry, or directly to a known good network endpoint within your cloud provider’s network.
      • Check ping and traceroute to your registry’s hostname (e.g., gcr.io, docker.io) from the containerd host.
    • Fix:
      • If iperf3 shows low throughput, investigate network configuration. This might involve upgrading your instance type, checking VPC/subnet configurations, ensuring no network ACLs or security groups are throttling traffic, or optimizing routing.
      • If ping shows high latency, it’s likely a physical/geographic issue. Consider deploying your containerd hosts closer to the registry or using a registry closer to your hosts.
    • Why it works: containerd needs to download image layers, which are essentially large files. If the pipe is narrow or the round trip is long, this process will naturally be slow.

  2. Registry Rate Limiting: Public registries (like Docker Hub) and even private ones can impose rate limits on how many images or layers you can pull per IP address or per account in a given time frame.

    • Diagnosis:
      • Check the containerd logs (journalctl -u containerd -f) for messages indicating "rate limit exceeded," "unauthorized," or HTTP status codes like 429 Too Many Requests.
      • If using Docker Hub, check your account’s pull statistics on their website.
    • Fix:
      • For Docker Hub: Authenticate containerd with Docker Hub using a valid account. Add an [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"] section to /etc/containerd/config.toml with your registry mirror if you have one, or ensure your dockerconfigjson is correctly configured in the daemon.json.
      • For private registries: Review your registry’s documentation for rate limit policies and consider upgrading your plan or using a dedicated endpoint.
      • Example config.toml snippet for authentication:
        [plugins."io.containerd.grpc.v1.cri".registry.auth]
  [plugins."io.containerd.grpc.v1.cri".registry.auth.configs]
    ["docker.io".auth]
      # Base64 encoded "username:password" or "token"
      # Example: echo -n "myuser:mypass" | base64
      # Or for token: echo -n ":mytoken" | base64
      "auth" = "dXNlcm5hbWU6cGFzc3dvcmQ="
    • Why it works: Authentication often grants higher rate limits, and using mirrors can distribute the load.

  3. DNS Resolution Issues: Slow or intermittent DNS lookups can cause delays when containerd initially tries to connect to the registry.

    • Diagnosis:
      • Run dig <registry-hostname> (e.g., dig index.docker.io) from the containerd host. Measure the time it takes for the query to resolve.
      • Check /etc/resolv.conf on the containerd host for your DNS server configuration.
    • Fix:
      • Ensure your DNS servers are responsive and geographically close to your containerd host.
      • If using cloud provider DNS, verify its health. Consider using a local caching DNS resolver (like systemd-resolved or dnsmasq) on the host.
      • Example config.toml snippet to force DNS usage:
        [plugins."io.containerd.grpc.v1.cri".registry.resolver]
  [plugins."io.containerd.grpc.v1.cri".registry.resolver.host."docker.io"]
    # Use specific DNS server for this host
    # "dns_server" = "8.8.8.8:53"
    • Why it works: Faster DNS means containerd establishes the connection to the registry endpoint quicker.

  4. containerd Configuration (Content Store Size/Location): If containerd’s content store (where image layers are cached) is on a slow disk or is very full, it can slow down operations.

    • Diagnosis:
      • Check the containerd configuration file (/etc/containerd/config.toml) for the root directory, which often defaults to /var/lib/containerd.
      • Use df -h to check disk space and iostat or iotop to monitor I/O performance on the filesystem where the content store resides.
    • Fix:
      • If the disk is slow (e.g., an HDD), migrate the content store to a faster disk (SSD/NVMe). This involves stopping containerd, moving the /var/lib/containerd/content directory, updating config.toml to point to the new location, and restarting containerd.
      • If the disk is nearly full, prune unused images and layers: ctr image prune or containerd snapshot prune --all.
      • Example config.toml change:
        root = "/mnt/fast_ssd/containerd"
    • Why it works: Faster I/O for reading and writing image layers directly speeds up the pull process.

  5. MTU Mismatch: In complex network environments (like Kubernetes clusters with overlay networks), an MTU mismatch between nodes and the registry can cause packet fragmentation, leading to severe performance degradation or outright failure.

    • Diagnosis:
      • Check the MTU of your host’s network interfaces (ip a).
      • Check the MTU of your Kubernetes CNI interface (e.g., Calico, Flannel).
      • Run ping -M do -s <packet_size> <registry-hostname> to find the largest packet size that can be sent without fragmentation. Start with a large size (e.g., 1472) and decrease.
    • Fix:
      • Ensure MTU settings are consistent across your network path. For many CNI plugins, this involves updating their configuration and potentially restarting pods or nodes.
      • For example, with Flannel using VXLAN, you might set flanneld --mtu=1450 or similar. With Calico, it’s often configured in the calico-config ConfigMap.
      • Alternatively, configure containerd to use a smaller MTU for its connections if possible, though this is less common.
    • Why it works: Consistent MTU prevents unnecessary packet fragmentation and reassembly, which is computationally expensive and can be dropped by intermediate network devices.

  6. Registry API/Server Issues: Occasionally, the container registry itself might be experiencing performance problems or outages.

    • Diagnosis:
      • Check the status page for your container registry provider (e.g., Docker Hub Status, Google Container Registry Status).
      • Try pulling a known small, public image from a different registry (e.g., alpine from Docker Hub vs. busybox from Quay.io) to see if the issue is specific to one registry.
    • Fix:
      • Wait for the registry provider to resolve their issues.
      • If possible, configure containerd to use a different registry mirror or a different registry entirely for your images.
      • Example config.toml for mirrors:
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
  endpoint = ["https://your-private-registry.com"]
    • Why it works: Offloading pulls to a different, functional registry or mirror bypasses the problematic endpoint.

After fixing these issues, the next error you might encounter is related to image signature verification failures if your registry requires it and the signing process or key management is misconfigured.

Want structured learning?

Take the full Containerd course →