BuildKit can actually make your Docker builds slower if you don’t understand its fundamental performance levers.
Let’s see BuildKit in action. Imagine this simple Dockerfile:
FROM alpine:latest
RUN echo "Hello, world!" > /hello.txt
COPY app.py /app/app.py
RUN pip install -r requirements.txt
CMD ["python", "/app/app.py"]
When BuildKit runs this, it doesn’t just execute these commands sequentially. It analyzes the dependencies between them. If requirements.txt hasn’t changed, it won’t re-run pip install. If app.py hasn’t changed, it won’t copy it again. This is called caching, and it’s BuildKit’s superpower.
The core problem BuildKit solves is the inefficiency of traditional Docker builds. Before BuildKit, every RUN command created a new layer, and if anything in the Dockerfile changed before that command, the entire build from that point down would be re-executed, even if the command’s inputs (like files or dependencies) hadn’t changed. BuildKit introduces a directed acyclic graph (DAG) for build steps, allowing for parallel execution and smarter caching.
Here are 10 tips to make your BuildKit builds faster:
-
Enable BuildKit: This sounds obvious, but many environments still run with the legacy builder. Ensure
DOCKER_BUILDKIT=1is set in your environment or viaDOCKER_OPTSindaemon.json.- Diagnosis: Run
docker build --progress=plain .and observe the output. If you don’t see the[+]syntax or detailed build steps, you’re likely not using BuildKit. - Fix: For Docker Desktop, it’s usually enabled by default. For server setups, edit
/etc/docker/daemon.jsonand add:
Then restart the Docker daemon:{ "features": { "buildkit": true } }sudo systemctl restart docker. - Why it works: BuildKit’s advanced caching and parallel execution are only available when the builder is active.
- Diagnosis: Run
-
Leverage Multi-Stage Builds: Separate your build environment from your runtime environment. This drastically reduces the final image size and build time by not carrying over build tools and intermediate artifacts.
- Diagnosis: Look for large base images in your
Dockerfilethat include development dependencies (like compilers, SDKs) that aren’t needed at runtime. - Fix:
# Builder stage FROM golang:1.20 AS builder WORKDIR /app COPY . . RUN go build -o myapp # Runtime stage FROM alpine:latest COPY --from=builder /app/myapp /myapp CMD ["/myapp"] - Why it works: The final image only contains what’s explicitly copied from the builder stage, discarding all build-time dependencies and intermediate layers.
- Diagnosis: Look for large base images in your
-
Order Your
DockerfileWisely: Place instructions that change less frequently (e.g., installing base packages) earlier in theDockerfile, and instructions that change often (e.g., copying application code) later.- Diagnosis: Observe build times after making small code changes. If the build takes a long time and rebuilds many steps, your order might be suboptimal.
- Fix:
FROM ubuntu:latest # Base dependencies, change infrequently RUN apt-get update && apt-get install -y --no-install-recommends \ python3 \ python3-pip \ # ... other packages && rm -rf /var/lib/apt/lists/* # Application code, changes frequently WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY . . CMD ["python", "app.py"] - Why it works: BuildKit caches layers. If a layer hasn’t changed, it’s reused. Placing stable instructions first ensures those layers are cached and reused across builds, even if later instructions change.
-
Use Specific Image Tags: Avoid
latest. Pin your base images to specific versions (e.g.,ubuntu:22.04,python:3.10-slim).- Diagnosis: Builds are intermittently slow or break unexpectedly due to upstream base image changes.
- Fix: Change
FROM ubuntu:latesttoFROM ubuntu:22.04. - Why it works: Using
latestmeans BuildKit might pull a new base image even if yourDockerfilehasn’t changed, invalidating caches for subsequent layers. Specific tags guarantee consistent inputs.
-
Optimize
RUNCommands: Combine multipleRUNcommands where logical, especially for package installations, to reduce the number of layers and improve cache hit rates.- Diagnosis: You have many small
RUNcommands in a row, likeRUN apt-get update,RUN apt-get install foo,RUN apt-get install bar. - Fix:
# Before # RUN apt-get update # RUN apt-get install -y foo # RUN apt-get install -y bar # After RUN apt-get update && apt-get install -y --no-install-recommends \ foo \ bar \ && rm -rf /var/lib/apt/lists/* - Why it works: Each
RUNcommand creates a new layer. Combining them reduces layer count. More importantly, if any part of the combined command changes (e.g., a package list), the entire layer is invalidated. Grouping related, stable installations together minimizes this.
- Diagnosis: You have many small
-
Use
COPYInstead ofADDfor Local Files:ADDhas special behavior for remote URLs and tarball extraction, which can be less predictable and harder to cache effectively.COPYis simpler and more direct.- Diagnosis: Builds are slow or unpredictable when
ADDis used, especially with compressed files. - Fix: Replace
ADD archive.tar.gz /destination/withCOPY archive.tar.gz .followed byRUN tar -xzf archive.tar.gz -C /destination/. - Why it works:
COPYhas a more straightforward caching mechanism based on file checksums.ADD’s automatic extraction and URL fetching can lead to unexpected cache invalidations.
- Diagnosis: Builds are slow or unpredictable when
-
Clean Up Build Artifacts: Remove temporary files, package manager caches, and unneeded build tools within the same
RUNinstruction that created them.- Diagnosis: Images are larger than necessary, or build steps that should be cached are being re-run because intermediate files are left behind.
- Fix:
# Before # RUN apt-get install -y some-package # RUN rm -rf /var/lib/apt/lists/* # After RUN apt-get update && apt-get install -y --no-install-recommends some-package \ && rm -rf /var/lib/apt/lists/* - Why it works: If you install a package in one
RUNand clean up in a separateRUN, the cleanup layer doesn’t affect the caching of the installation layer. However, if the installation layer changes, the cleanup layer is also re-executed, potentially re-downloading and then removing things unnecessarily. Combining them ensures that if the installation layer is cached, the cleanup is also implicitly done for that cached state.
-
Optimize
COPYInstructions: Copy only what you need. AvoidCOPY . .if only a few files are required for a specific build step.- Diagnosis: A small change to one file in your project invalidates a
COPYinstruction that copies your entire project, causing a lengthy rebuild of subsequent steps. - Fix:
# Instead of COPY . . COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY app.py . - Why it works: BuildKit caches based on the content of what is copied. Copying only the necessary files means that if unrelated files change, the
COPYinstruction’s cache is not invalidated, preventing unnecessary rebuilds.
- Diagnosis: A small change to one file in your project invalidates a
-
Use BuildKit’s Cache Mounts (
--mount=type=cache): For package managers or dependency directories that are expensive to rebuild but whose contents can be shared across builds, use cache mounts.- Diagnosis: Repeatedly installing dependencies for languages like Node.js (
npm install,yarn install) or Python (pip install) takes a long time on every build, even though the dependencies themselves rarely change. - Fix:
In yourdocker build \ --mount=type=cache,target=/root/.cache/pip \ .Dockerfile:FROM python:3.10-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt # ... rest of your Dockerfile - Why it works: This tells BuildKit to persist the contents of
/root/.cache/pip(or other specified targets) between builds, dramatically speeding up dependency installation by avoiding re-downloading and re-compiling.
- Diagnosis: Repeatedly installing dependencies for languages like Node.js (
-
Parallelize Independent Build Steps: If you have multiple independent build tasks that don’t depend on each other, BuildKit can run them concurrently.
- Diagnosis: Your
Dockerfilehas sections that could logically run at the same time but are executed sequentially. - Fix: This is more of a structural
Dockerfileapproach. If you have multiple services to build, consider separateDockerfiles and a meta-build system, or structure yourDockerfileto have distinct, parallelizable stages. For example, building frontend assets and backend code can often happen in parallel.# Stage 1: Build backend FROM golang:1.20 AS backend-builder # ... build backend ... # Stage 2: Build frontend FROM node:18 AS frontend-builder # ... build frontend assets ... # Final stage combining results FROM alpine:latest COPY --from=backend-builder /app/backend /backend COPY --from=frontend-builder /app/dist /static CMD ["/backend"] - Why it works: BuildKit’s scheduler can identify independent execution graphs within a build and run them concurrently on available CPU resources, reducing overall wall-clock build time.
- Diagnosis: Your
The next common problem you’ll encounter is cache invalidation due to subtle changes in base images or build arguments.