Docker build cache isn’t working, causing painfully slow rebuilds because the builder thinks it needs to re-execute every instruction when it doesn’t.

Common Causes and Fixes

  1. Instruction Order and Cache Invalidation: Docker caches layers based on the instruction and its arguments. If an instruction that shouldn’t change (like RUN apt-get update && apt-get install -y ...) is placed after an instruction that does change frequently (like COPY . .), the entire cache from that point onward is invalidated.

    • Diagnosis: Examine your Dockerfile. Look for instructions that are frequently re-executed.

    • Fix: Reorder your Dockerfile to place frequently changing instructions (like COPY) later, and stable instructions (like package installations) earlier.

      # BAD: COPY invalidates RUN apt-get
      COPY . .
      RUN apt-get update && apt-get install -y some-package
      
      # GOOD: apt-get runs from cache if nothing changes before it
      RUN apt-get update && apt-get install -y some-package
      COPY . .
      
    • Why it works: Docker builds layers sequentially. If a layer’s instruction and its arguments haven’t changed since the last build, Docker reuses the cached layer. By moving COPY later, it only invalidates the cache for the COPY instruction and subsequent ones, not for earlier, stable operations like package installation.

  2. ADD with Remote URLs: Using ADD to fetch files from a remote URL can cause cache misses if the URL’s content changes, even if the URL itself remains the same. Docker doesn’t cache the content of the URL independently; it caches based on the instruction.

    • Diagnosis: Check if you’re using ADD with URLs.

    • Fix: Use curl or wget within a RUN instruction instead of ADD for remote files.

      # BAD: ADD can invalidate cache on remote content change
      ADD https://example.com/file.tar.gz /tmp/
      
      # GOOD: RUN instruction allows for explicit caching control and better debugging
      RUN curl -sSL https://example.com/file.tar.gz | tar xz -C /tmp/
      
    • Why it works: RUN instructions are executed by a shell, and Docker caches the layer based on the exact command string. By using curl and piping to tar, you make the operation a single, reproducible step that Docker can cache effectively.

  3. Changed Build Context: The build context (the directory sent to the Docker daemon) includes all files and subdirectories. If any file within the build context changes, and an instruction like COPY . . or ADD . . is present, it can invalidate the cache for that COPY/ADD layer and all subsequent layers.

    • Diagnosis: Observe if small, unrelated file changes trigger long rebuilds.

    • Fix: Use .dockerignore to exclude unnecessary files and directories from the build context. Be specific about what you COPY or ADD.

      # GOOD: Only copy necessary files
      COPY app/ /app/
      
      # .dockerignore
      .git
      node_modules
      *.log
      
    • Why it works: Reducing the build context size and excluding files that don’t affect the build means fewer potential cache invalidations. Explicitly copying specific directories or files, rather than the entire context (.), makes the dependency clearer to Docker’s cache.

  4. File Timestamps: Docker’s cache invalidation for COPY and ADD instructions considers file content and timestamps. If you’re copying files that are generated or modified during the build process (e.g., using npm install and then COPY package*.json .), and the timestamps are inconsistent, it can lead to cache misses.

    • Diagnosis: Look for COPY or ADD operations on files that are also modified by subsequent RUN commands.

    • Fix: Ensure files copied into the image have consistent, predictable timestamps, or perform operations in an order that minimizes timestamp dependencies. For generated files, consider building them within the container.

      # GOOD: Copying package files first, then installing, then copying app code
      COPY package*.json ./
      RUN npm install
      COPY . .
      
    • Why it works: By copying the package.json and package-lock.json first and running npm install, you create a layer for dependencies. Only if these specific files change will the npm install layer be invalidated. Then, copying the rest of the application code invalidates subsequent layers. This is more granular than copying everything at once.

  5. BuildKit Not Enabled or Configured Incorrectly: Docker’s newer builder, BuildKit, offers more advanced caching mechanisms, including remote caching and better cache invalidation strategies. If you’re not using it or it’s misconfigured, you won’t benefit from these improvements.

    • Diagnosis: Check your Docker version (docker version). Ensure DOCKER_BUILDKIT=1 is set in your environment or buildx.buildkit=true in ~/.docker/config.json.

    • Fix: Enable BuildKit.

      • Environment Variable:
        export DOCKER_BUILDKIT=1
        docker build .
        
      • Docker Daemon Configuration (Docker Desktop): Go to Settings -> Features in development -> Enable BuildKit (experimental).
      • docker-compose.yml:
        version: '3.8'
        services:
          myapp:
            build:
              context: .
              dockerfile: Dockerfile
              args:
                BUILDKIT_INLINE_CACHE: 1 # Optional, for faster cache pulls
            # ...
        
    • Why it works: BuildKit is a more intelligent builder. It understands build graphs better, can perform more operations in parallel, and has features like secret management and improved caching that go beyond the legacy builder.

  6. docker build vs. docker buildx build: While docker buildx build uses BuildKit by default and offers more features, sometimes issues can arise if you’re mixing usage or have specific buildx configurations that interfere with caching.

    • Diagnosis: If you’re using buildx and experiencing cache issues, check your builder instances (docker buildx ls) and their configurations.

    • Fix: Ensure you’re using a consistent builder and that its cache settings are as expected. For example, to ensure you’re using the default builder:

      docker buildx use default
      docker buildx build --platform linux/amd64 -t myimage .
      
    • Why it works: buildx manages multiple builders, and each can have its own cache configuration (e.g., local vs. remote). Explicitly selecting and configuring your builder ensures you’re leveraging the caching mechanism you intend to.

After fixing these, the next error you’ll likely encounter is a "no space left on device" error if your cache has grown too large and you haven’t pruned it.

Want structured learning?

Take the full Docker course →