Docker’s overlay2 storage driver is failing because the underlying XFS filesystem was created with an older feature flag that Docker’s modern requirements cannot tolerate.
Common Causes and Fixes:
-
XFS
ftypesetting is0:- Diagnosis: Check the
ftypesetting for the filesystem where Docker stores its data. This is typically/var/lib/docker.
You’re looking forsudo xfs_info /var/lib/docker | grep ftypeftype=0. - Fix: Recreate the XFS filesystem with
ftype=1. This is a destructive operation, so back up any critical data first.
The# Unmount the filesystem (e.g., if /var/lib/docker is on a separate partition) sudo umount /var/lib/docker # Format with ftype=1 sudo mkfs.xfs -f -i=attr /dev/your_xfs_partition # Mount it back sudo mount /dev/your_xfs_partition /var/lib/dockerftype=1setting enables extended file metadata, which overlay2 relies on for efficient inode handling and performance. This is the default and recommended setting for modern XFS. - Why it works:
ftype=1allocates more space per inode by default, allowing for more extended attributes to be stored directly within the inode. Overlay2 uses these extended attributes for tracking file metadata efficiently.ftype=0(the older default) forces extended attributes to be stored in a separate, less efficient data fork, which overlay2 struggles with.
- Diagnosis: Check the
-
Docker Daemon Configuration Incorrectly Specifying
overlay2:- Diagnosis: Inspect the Docker daemon’s configuration file, typically
/etc/docker/daemon.json.
Look for acat /etc/docker/daemon.jsonstorage-driverkey. - Fix: Ensure the
storage-driveris set tooverlay2and remove any explicitoverlaydriver setting if present.
Restart the Docker daemon after making changes.{ "storage-driver": "overlay2" }
This ensures Docker attempts to use the correct, modern storage driver.sudo systemctl restart docker
- Diagnosis: Inspect the Docker daemon’s configuration file, typically
-
Corrupted Docker Data Directory:
- Diagnosis: While less common for this specific error, a corrupted
/var/lib/dockerdirectory could lead to unexpected behavior, including issues with the storage driver. Check for any unusual file permissions or missing files within/var/lib/docker. - Fix: This is a last resort as it will remove all existing images, containers, and volumes. Stop Docker, clear the data directory, and restart.
This forces Docker to reinitialize its data directory, creating fresh metadata and potentially resolving underlying corruption.sudo systemctl stop docker sudo rm -rf /var/lib/docker/* sudo systemctl start docker
- Diagnosis: While less common for this specific error, a corrupted
-
Kernel Module Issues for OverlayFS:
- Diagnosis: Verify that the
overlaykernel module is loaded and functioning correctly.
Look for any errors related to the overlay module inlsmod | grep overlay sudo modprobe overlay dmesg | grep overlaydmesg. - Fix: Ensure the module is loaded. If it’s not, you might need to load it manually or ensure it’s configured to load at boot.
This ensures the necessary kernel module is available for theecho overlay | sudo tee /etc/modules-load.d/overlay.conf sudo systemctl restart dockeroverlay2storage driver to use.
- Diagnosis: Verify that the
-
Incorrect Mount Options for
/var/lib/docker:- Diagnosis: Check the mount options for the filesystem hosting
/var/lib/docker.
Look for options that might interfere with extended attributes or file metadata.mount | grep /var/lib/docker - Fix: Ensure the filesystem is mounted with standard options that allow for extended attributes. If you have custom
noexecor other restrictive options that aren’t necessary, consider removing them from/etc/fstaband remounting.
The# Example of a typical mount for XFS /dev/sda1 on /var/lib/docker type xfs (rw,relatime,attr2,inode64,noquota)attr2andinode64options are standard and indicate extended attribute support.
- Diagnosis: Check the mount options for the filesystem hosting
-
SELinux Context Issues:
- Diagnosis: SELinux can sometimes interfere with Docker’s operations by restricting access to files and directories. Check the SELinux audit log for any denials related to Docker.
sudo ausearch -m avc -ts recent | grep docker - Fix: Restore the default SELinux contexts for the Docker directory.
This ensures that SELinux policies are correctly applied to the Docker data directory, allowing it to function as expected.sudo restorecon -Rv /var/lib/docker sudo systemctl restart docker
- Diagnosis: SELinux can sometimes interfere with Docker’s operations by restricting access to files and directories. Check the SELinux audit log for any denials related to Docker.
After resolving the ftype=0 issue and ensuring the XFS filesystem is correctly formatted with ftype=1, the next error you might encounter is related to missing aufs or btrfs kernel modules if you previously tried to force those drivers.