Docker’s default logging driver, json-file, doesn’t automatically clean up old logs, which can quickly fill up your disk if you’re not careful.
# First, let's see what's actually using up space
du -sh /var/lib/docker/containers/*
This command will show you the disk usage for each container’s log directory. You’ll likely see some very large numbers for containers that have been running for a while.
The solution is to configure the json-file logging driver to automatically rotate and limit the size of log files. This is done on a per-container basis using docker run options or within a docker-compose.yml file.
Here are the common causes and their fixes:
1. No Log Rotation Configured (Most Common)
- Diagnosis: You’ll see large log files in
/var/lib/docker/containers/<container_id>/<container_id>-json.log. - Fix (docker run):
docker run \ --log-opt max-size=10m \ --log-opt max-file=3 \ your_image_name - Fix (docker-compose.yml):
services: your_service_name: image: your_image_name logging: driver: "json-file" options: max-size: "10m" max-file: "3" - Why it works:
max-size=10mtells Docker to create a new log file once the current one reaches 10 megabytes.max-file=3tells Docker to keep only the 3 most recent log files, automatically deleting the oldest ones.
2. Incorrect max-size Units
- Diagnosis: Logs are still growing excessively, but you think you’ve set
max-size. The file sizes might be larger than expected. - Fix (docker run):
docker run \ --log-opt max-size=100m \ --log-opt max-file=5 \ your_image_name - Fix (docker-compose.yml):
services: your_service_name: image: your_image_name logging: driver: "json-file" options: max-size: "100m" # Using 'm' for megabytes, 'g' for gigabytes, 'k' for kilobytes max-file: "5" - Why it works: The
max-sizeoption accepts suffixes likek,m, andgfor kilobytes, megabytes, and gigabytes, respectively. If you omit the suffix, it defaults to bytes, which is rarely useful. Ensure you’re using10m(10 megabytes) or100k(100 kilobytes) and not just10or100.
3. max-file Too High or Not Set
- Diagnosis: Log files are rotating, but you still have too many of them accumulating, eventually filling the disk.
- Fix (docker run):
docker run \ --log-opt max-size=5m \ --log-opt max-file=2 \ your_image_name - Fix (docker-compose.yml):
services: your_service_name: image: your_image_name logging: driver: "json-file" options: max-size: "5m" max-file: "2" # Keeps only 2 rotated files (plus the current one) - Why it works:
max-filedirectly controls the number of log files kept. Setting it to a low number like2or3ensures that older logs are pruned aggressively. The current log file is always present, somax-file=2means you’ll have the current file and 2 rotated files, for a total of 3 log files at any given time.
4. Applying Settings to Running Containers
- Diagnosis: You’ve updated your
docker-compose.ymlor addeddocker runoptions, but existing containers aren’t using the new settings. - Fix: You must recreate the containers for the new logging options to take effect.
- For
docker-compose:docker-compose up -d --force-recreate --no-deps your_service_name - For standalone
docker run: Stop, remove, and re-run the container with the new options.docker stop <container_name_or_id> docker rm <container_name_or_id> docker run \ --log-opt max-size=10m \ --log-opt max-file=3 \ your_image_name
- For
- Why it works: Logging driver configurations are typically set when a container is created. Modifying the configuration in
docker-compose.ymlor command-line flags doesn’t retroactively apply to already existing containers. Recreating the container ensures the new configuration is applied from scratch.
5. Using the syslog or journald Driver Instead
- Diagnosis: You’re experiencing disk fill issues, but you’re not using the
json-filedriver. The problem might be with the remote logging setup. - Fix: While
syslogandjournalddrivers send logs elsewhere, they can still accumulate local buffers or cause issues if the remote target is misconfigured. If you want local rotation, revert tojson-fileand configure it as above. If you intend to usesyslogorjournald, ensure yoursyslogdaemon orjournaldconfiguration is also set up for rotation and size limits, or that the remote endpoint is healthy.- Example
docker-compose.ymlforsyslog:services: your_service_name: image: your_image_name logging: driver: "syslog" options: syslog-address: "udp://127.0.0.1:514"
- Example
- Why it works: If you’re not using
json-file, the problem lies in how your chosen logging driver handles logs. Forsyslogorjournald, the responsibility shifts from Docker to the system’s logging infrastructure. If disk fill is still an issue, it means that infrastructure is also not properly managing log sizes.
6. Host System’s journald Configuration (if journald is your logging driver)
- Diagnosis: If you’ve configured Docker to use
journaldas its logging driver (e.g.,--log-driver=journald), and your host system’sjournaldis not configured for rotation, it can also fill up disk space. - Fix: Edit
/etc/systemd/journald.confon your Docker host. Uncomment and setSystemMaxUseandMaxFileSec.
After editing, restart the# /etc/systemd/journald.conf [Journal] #Compress=yes Storage=auto #Seal=yes #SplitMode=login #KeepFreeSpace= #MaxFileSec=1month #MaxRetentionSec= #MaxWalSize= #MaxFileArchive= #SystemMaxUse=500M # Limit total journal size to 500MB #SystemMaxFileSize= #RuntimeMaxUse= #RuntimeMaxFileSize=systemd-journaldservice:sudo systemctl restart systemd-journald - Why it works:
journaldis the systemd journal service, which acts as a central logging repository for many Linux systems. When Docker logs tojournald, it’s essentially sending its logs to this service.SystemMaxUselimits the total disk spacejournaldcan consume, andMaxFileSec(orMaxRetentionSec) can prune older entries based on time.
After applying these settings, you should see your log files staying within the configured limits, preventing your disks from filling up. The next error you might encounter is related to container health checks failing if logs are too sparse or if the application itself is crashing repeatedly.