Docker’s default logging driver, json-file, doesn’t automatically clean up old logs, which can quickly fill up your disk if you’re not careful.

# First, let's see what's actually using up space
du -sh /var/lib/docker/containers/*

This command will show you the disk usage for each container’s log directory. You’ll likely see some very large numbers for containers that have been running for a while.

The solution is to configure the json-file logging driver to automatically rotate and limit the size of log files. This is done on a per-container basis using docker run options or within a docker-compose.yml file.

Here are the common causes and their fixes:

1. No Log Rotation Configured (Most Common)

  • Diagnosis: You’ll see large log files in /var/lib/docker/containers/<container_id>/<container_id>-json.log.
  • Fix (docker run):
    docker run \
      --log-opt max-size=10m \
      --log-opt max-file=3 \
      your_image_name
    
  • Fix (docker-compose.yml):
    services:
      your_service_name:
        image: your_image_name
        logging:
          driver: "json-file"
          options:
            max-size: "10m"
            max-file: "3"
    
  • Why it works: max-size=10m tells Docker to create a new log file once the current one reaches 10 megabytes. max-file=3 tells Docker to keep only the 3 most recent log files, automatically deleting the oldest ones.

2. Incorrect max-size Units

  • Diagnosis: Logs are still growing excessively, but you think you’ve set max-size. The file sizes might be larger than expected.
  • Fix (docker run):
    docker run \
      --log-opt max-size=100m \
      --log-opt max-file=5 \
      your_image_name
    
  • Fix (docker-compose.yml):
    services:
      your_service_name:
        image: your_image_name
        logging:
          driver: "json-file"
          options:
            max-size: "100m" # Using 'm' for megabytes, 'g' for gigabytes, 'k' for kilobytes
            max-file: "5"
    
  • Why it works: The max-size option accepts suffixes like k, m, and g for kilobytes, megabytes, and gigabytes, respectively. If you omit the suffix, it defaults to bytes, which is rarely useful. Ensure you’re using 10m (10 megabytes) or 100k (100 kilobytes) and not just 10 or 100.

3. max-file Too High or Not Set

  • Diagnosis: Log files are rotating, but you still have too many of them accumulating, eventually filling the disk.
  • Fix (docker run):
    docker run \
      --log-opt max-size=5m \
      --log-opt max-file=2 \
      your_image_name
    
  • Fix (docker-compose.yml):
    services:
      your_service_name:
        image: your_image_name
        logging:
          driver: "json-file"
          options:
            max-size: "5m"
            max-file: "2" # Keeps only 2 rotated files (plus the current one)
    
  • Why it works: max-file directly controls the number of log files kept. Setting it to a low number like 2 or 3 ensures that older logs are pruned aggressively. The current log file is always present, so max-file=2 means you’ll have the current file and 2 rotated files, for a total of 3 log files at any given time.

4. Applying Settings to Running Containers

  • Diagnosis: You’ve updated your docker-compose.yml or added docker run options, but existing containers aren’t using the new settings.
  • Fix: You must recreate the containers for the new logging options to take effect.
    • For docker-compose:
      docker-compose up -d --force-recreate --no-deps your_service_name
      
    • For standalone docker run: Stop, remove, and re-run the container with the new options.
      docker stop <container_name_or_id>
      docker rm <container_name_or_id>
      docker run \
        --log-opt max-size=10m \
        --log-opt max-file=3 \
        your_image_name
      
  • Why it works: Logging driver configurations are typically set when a container is created. Modifying the configuration in docker-compose.yml or command-line flags doesn’t retroactively apply to already existing containers. Recreating the container ensures the new configuration is applied from scratch.

5. Using the syslog or journald Driver Instead

  • Diagnosis: You’re experiencing disk fill issues, but you’re not using the json-file driver. The problem might be with the remote logging setup.
  • Fix: While syslog and journald drivers send logs elsewhere, they can still accumulate local buffers or cause issues if the remote target is misconfigured. If you want local rotation, revert to json-file and configure it as above. If you intend to use syslog or journald, ensure your syslog daemon or journald configuration is also set up for rotation and size limits, or that the remote endpoint is healthy.
    • Example docker-compose.yml for syslog:
      services:
        your_service_name:
          image: your_image_name
          logging:
            driver: "syslog"
            options:
              syslog-address: "udp://127.0.0.1:514"
      
  • Why it works: If you’re not using json-file, the problem lies in how your chosen logging driver handles logs. For syslog or journald, the responsibility shifts from Docker to the system’s logging infrastructure. If disk fill is still an issue, it means that infrastructure is also not properly managing log sizes.

6. Host System’s journald Configuration (if journald is your logging driver)

  • Diagnosis: If you’ve configured Docker to use journald as its logging driver (e.g., --log-driver=journald), and your host system’s journald is not configured for rotation, it can also fill up disk space.
  • Fix: Edit /etc/systemd/journald.conf on your Docker host. Uncomment and set SystemMaxUse and MaxFileSec.
    # /etc/systemd/journald.conf
    [Journal]
    #Compress=yes
    Storage=auto
    #Seal=yes
    #SplitMode=login
    #KeepFreeSpace=
    #MaxFileSec=1month
    #MaxRetentionSec=
    #MaxWalSize=
    #MaxFileArchive=
    #SystemMaxUse=500M  # Limit total journal size to 500MB
    #SystemMaxFileSize=
    #RuntimeMaxUse=
    #RuntimeMaxFileSize=
    
    After editing, restart the systemd-journald service:
    sudo systemctl restart systemd-journald
    
  • Why it works: journald is the systemd journal service, which acts as a central logging repository for many Linux systems. When Docker logs to journald, it’s essentially sending its logs to this service. SystemMaxUse limits the total disk space journald can consume, and MaxFileSec (or MaxRetentionSec) can prune older entries based on time.

After applying these settings, you should see your log files staying within the configured limits, preventing your disks from filling up. The next error you might encounter is related to container health checks failing if logs are too sparse or if the application itself is crashing repeatedly.

Want structured learning?

Take the full Docker course →