The Network Not Ready: CNI Not Initialized error in containerd means the container runtime couldn’t set up network interfaces for your containers because the Container Network Interface (CNI) plugin failed to load or execute.
Common Causes and Fixes
1. CNI Configuration Files Missing or Incorrectly Formatted:
containerd relies on CNI configuration files (usually in /etc/cni/net.d/) to know which network plugins to use and how to configure them. If these files are missing, empty, or malformed (e.g., invalid JSON), the CNI plugin will fail.
- Diagnosis: Check for files in
/etc/cni/net.d/.
If the directory is empty or contains non-JSON files, or if existing files are not valid JSON, this is your problem. You can validate JSON withls -l /etc/cni/net.d/jq:jq . /etc/cni/net.d/your-cni-config.conf - Fix: Ensure you have at least one valid CNI configuration file. For a basic setup using
bridgeandhost-local, a common configuration looks like this:/etc/cni/net.d/10-my-cni.conf
This configuration tells the CNI plugin to create a{ "cniVersion": "0.3.1", "name": "mynet", "type": "bridge", "bridge": "cnibr0", "isGateway": true, "ipMasq": true, "ipam": { "type": "host-local", "subnet": "10.244.0.0/16", "routes": [ { "dst": "0.0.0.0/0" } ] } }bridgenetwork namedcnibr0, assign IPs from the10.244.0.0/16subnet, and set up IP masquerading. - Why it works: This provides containerd with the necessary instructions to load and use the appropriate CNI plugins for network setup.
2. CNI Binaries Not Found or Executable:
The CNI plugins themselves are executables (e.g., bridge, host-local, loopback). These need to be present in a directory listed in the PATH environment variable for the CNI process, and they must have execute permissions.
- Diagnosis: Check if CNI binaries are in
/opt/cni/bin/and are executable.
Look for files likels -l /opt/cni/bin/bridge,host-local,loopback, etc. Ensure they havexin their permissions (e.g.,-rwxr-xr-x). - Fix: Install the CNI plugins package for your distribution or ensure they are correctly placed and executable. For example, on systems using
apt:
If they are already installed but not executable:sudo apt-get update sudo apt-get install containerd.io cni-pluginssudo chmod +x /opt/cni/bin/* - Why it works: This ensures that when containerd’s CNI plugin invokes a network setup command (like
bridge), the executable binary is found and can be run.
3. Incorrect containerd Configuration for CNI:
The containerd configuration file (/etc/containerd/config.toml) specifies how containerd should interact with CNI. If the disabled_plugins list incorrectly includes CNI or if the root directory for CNI is misconfigured, it can cause this error.
- Diagnosis: Examine
/etc/containerd/config.toml.
Look for acat /etc/containerd/config.toml[plugins]section and specifically[plugins."io.containerd.grpc.v1.cri"]. Ensurecniis not listed underdisabled_plugins. Also, verify therootpath for CNI is correctly set if it’s explicitly defined. - Fix: If
cniis indisabled_plugins, remove it. A typical configuration relevant to CNI would look something like this withinconfig.toml:
After editing, restart containerd:[plugins."io.containerd.grpc.v1.cri"] sandbox_image = "k8s.gcr.io/pause:3.5" [plugins."io.containerd.grpc.v1.cri".cni] bin_dir = "/opt/cni/bin" conf_dir = "/etc/cni/net.d"sudo systemctl restart containerd - Why it works: This ensures containerd is configured to use the CNI plugins and knows where to find their binaries and configuration files.
4. Insufficient File Descriptors or System Limits: When many containers are started, or if there are issues with resource limits, the CNI plugin might fail due to running out of available file descriptors or hitting other system-level constraints.
- Diagnosis: Check system logs for messages related to file descriptor limits or
ulimit.
Also, check the system’s current limits:sudo journalctl -u containerd -fulimit -n - Fix: Increase the file descriptor limit for the
containerdprocess. This is often done by editing/etc/systemd/system/containerd.service.d/override.conf(or similar override file) and adding:
Then reload systemd and restart containerd:[Service] LimitNOFILE=1048576sudo systemctl daemon-reload sudo systemctl restart containerd - Why it works: Providing more file descriptors allows the CNI plugin to manage the network interfaces and associated kernel resources for a larger number of containers without hitting OS limits.
5. Corrupted CNI State or IPAM Data:
The host-local IPAM plugin (often used by default) stores its IP allocation data in /var/lib/cni/networks/. If this directory or its contents become corrupted, the IPAM plugin can fail, leading to CNI initialization errors.
- Diagnosis: Inspect the contents of
/var/lib/cni/networks/.
Look for malformed files or directories that seem out of place.ls -lR /var/lib/cni/networks/ - Fix: If you suspect corruption and can afford to reset IP allocations (e.g., in a development environment or if you know no critical containers are running), you can remove the stale state. Caution: This will cause IP conflicts if containers are still running and relying on these allocations.
Then restart containerd.sudo rm -rf /var/lib/cni/networks/* - Why it works: This clears out any potentially corrupted IP address allocation records, allowing the
host-localIPAM plugin to start fresh and re-allocate addresses.
6. Kernel Module Issues (e.g., overlay or bridge):
CNI plugins often rely on specific kernel modules being loaded, such as overlay for storage drivers or bridge for network bridging. If these modules are not loaded or are malfunctioning, CNI operations can fail.
- Diagnosis: Check if necessary kernel modules are loaded.
lsmod | grep overlay lsmod | grep bridge - Fix: Load the required modules manually.
To ensure they load on boot, you might need to add them tosudo modprobe overlay sudo modprobe bridge/etc/modules-load.d/containerd.conf:overlay bridge - Why it works: Ensures the underlying kernel functionality required by CNI plugins (like network bridging or specific storage drivers) is available.
After resolving these issues, you might encounter a failed to load Kubelet configuration error if you are using containerd with Kubernetes and the Kubelet configuration itself was dependent on the CNI being initialized correctly.