Fix Docker DevMapper Thin Pool Running Out of Data Space (2026)

The devicemapper storage driver in Docker has failed because its underlying thin pool, which stores all container and image data, has run out of allocated space.

Common Causes and Fixes

Here’s a breakdown of why this happens and how to fix it, ordered by likelihood:

Accumulated Dangling Images and Build Cache: Docker, by default, keeps all downloaded images and intermediate build layers. Over time, these can consume a huge amount of space.
- Diagnosis: Run docker system df. This command shows Docker’s disk usage, including images, containers, and build cache. Look for a high "Reclaimable" value.
- Fix: Execute docker system prune -a --volumes. This command aggressively cleans up all unused images (not just dangling ones), stopped containers, and build cache. The --volumes flag also removes unused anonymous volumes, which can be a significant space hog.
- Why it works: This removes unreferenced data that Docker still holds onto, freeing up space within the thin pool.
Large, Uncleaned Container Volumes: Even if containers are stopped, their associated volumes can persist and grow.
- Diagnosis: Run docker volume ls to list all volumes. Then, for each volume, use docker inspect <volume_name> | grep Mountpoint to find its actual location on the host. Manually check the size of these directories (e.g., /var/lib/docker/volumes/<volume_name>/_data).
- Fix: Identify large, unused volumes and remove them with docker volume rm <volume_name>. If a volume is still in use by a stopped container, you’ll need to stop and remove the container first (docker stop <container_id> then docker rm <container_id>) before removing the volume.
- Why it works: Volumes are often where persistent application data resides. If this data is no longer needed, explicitly removing the volume frees up the space it occupied.
Large Container Log Files: By default, Docker logs can grow indefinitely. If a container is logging excessively or has a runaway process, its logs can fill up the disk.
- Diagnosis: Check the size of the log files within /var/lib/docker/containers/<container_id>/<container_id>-json.log. You can find container IDs with docker ps -a -q.
- Fix:
  - Immediate Cleanup: Manually truncate large log files: echo "" > /var/lib/docker/containers/<container_id>/<container_id>-json.log.
  - Preventative Measure: Configure log rotation for containers. In your daemon.json (usually located at /etc/docker/daemon.json), add or modify the log-driver and log-opts:
```
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}
```
    Then, restart the Docker daemon: sudo systemctl restart docker. This will ensure logs are rotated after reaching 10MB, keeping only the last 3 files.
- Why it works: Log rotation limits the continuous growth of log files by capping their size and number, preventing them from consuming all available space.
Under-Provisioned Thin Pool Size: The devicemapper thin pool is backed by a specific block device or file. If this backing storage was too small to begin with, you’ll hit this limit.
- Diagnosis:
  - Check the current thin pool usage: sudo dmsetup status docker-thinpool (or similar, the name might vary). This will show you the total size and usage percentage.
  - Identify the backing device: sudo dmsetup info docker-thinpool. Look for the Block device field.
  - Check the size of the backing device: sudo blockdev --getsize64 /dev/mapper/<backing_device_name> or ls -lh /path/to/backing_file.
- Fix: This is the most involved fix. You’ll need to stop Docker, detach the thin pool, resize the backing storage (either by resizing the logical volume or replacing the backing file with a larger one), and then reattach and restart Docker.
  - Example for a logical volume:
    - Stop Docker: sudo systemctl stop docker
    - Resize the LV: sudo lvresize -L +50G /dev/vg_docker/lv_docker_pool (adjust size and path as needed)
    - Start Docker: sudo systemctl start docker
  - Example for a backing file:
    - Stop Docker: sudo systemctl stop docker
    - Create a larger backing file: dd if=/dev/zero of=/var/lib/docker/devicemapper/thinpool.new bs=1G count=100 (creates a 100GB file)
    - Configure Docker to use the new file (requires editing daemon.json to point to dm.thinpool.direct-lvm.data_file or similar, depending on your setup).
    - Start Docker: sudo systemctl start docker
- Why it works: Directly increases the total capacity of the storage pool that Docker uses.
Stale Snapshot Data: devicemapper uses snapshots for containers and images. Sometimes, due to unclean shutdowns or specific operations, stale snapshot metadata can accumulate and occupy space that isn’t accounted for by active containers.
- Diagnosis: This is harder to diagnose directly from user-facing tools. High dmsetup status usage without a clear corresponding number of active containers or large volumes is a hint.
- Fix: A docker system prune -a (as in cause 1) is often sufficient to clean up some of this. More aggressively, a full Docker data directory cleanup might be necessary, which involves stopping Docker, moving/deleting /var/lib/docker, and restarting Docker. Caution: This will remove ALL images, containers, and volumes.
- Why it works: A clean restart forces devicemapper to rebuild its internal state, potentially discarding orphaned or corrupted snapshot metadata.
Host System Disk Full: While the error specifically mentions the thin pool, it’s backed by host disk space. If the underlying host partition is completely full, the thin pool cannot allocate more.
- Diagnosis: Run df -h on the host machine. Check the usage of the partition where /var/lib/docker resides.
- Fix: Free up space on the host partition by removing unnecessary files, cleaning package caches (sudo apt autoremove, sudo yum autoremove), or moving data to a different location.
- Why it works: Provides the physical disk space that the devicemapper thin pool needs to expand into.

The next error you’ll likely encounter after fixing this is related to image pulls failing due to network issues or a full aufs (or other storage driver) layer cache if that also becomes a problem.