BuildKit’s snapshotter is the unsung hero that makes your container builds fast and efficient, but it’s often overlooked until things go wrong.
Let’s see it in action. Imagine you’re building a Go application.
# syntax=docker/dockerfile:1
FROM golang:1.20 AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go build -o myapp .
FROM alpine:latest
COPY --from=builder /app/myapp /usr/local/bin/
CMD ["myapp"]
When you docker buildx build --platform linux/amd64 -t myapp ., BuildKit doesn’t just download each layer. It uses a snapshotter to manage intermediate build states. The first time go mod download runs, BuildKit creates a snapshot of the filesystem with the downloaded modules. If you change your Go code but not your dependencies, BuildKit can reuse that go mod download snapshot. This is the core idea: sharing and reusing filesystem states between build steps.
The snapshotter is responsible for storing and retrieving these filesystem states, often called "layers" or "snapshots." When a build step needs a filesystem (e.g., to copy files into it or to execute a command within it), BuildKit asks the snapshotter for a copy-on-write (COW) view of the required state. Changes made during the build step are written to this COW layer, leaving the original snapshot untouched. This is how BuildKit achieves layer reuse and parallelism.
The most common snapshotter is overlayfs, which is a Linux kernel feature that provides a layered filesystem. BuildKit can also use other snapshotters like native (which uses the underlying filesystem’s copy-on-write capabilities, like APFS on macOS or Btrfs/ZFS on Linux) or stargz for efficient remote layer pulling.
The primary configuration point for the snapshotter is within the BuildKit daemon’s configuration. You typically define this in ~/.config/buildkit/buildkitd.toml.
Here’s a typical configuration for overlayfs:
[worker.oci]
enabled = true
platforms = ["linux/amd64", "linux/arm64"]
[worker.oci.snapshotter]
# Use overlayfs for Linux. This is generally the default and recommended.
# Other options include "native" (if supported by the host OS) or "stargz".
"linux/amd64" = "overlayfs"
"linux/arm64" = "overlayfs"
If you were on macOS and wanted to leverage APFS’s native COW capabilities, you might configure it like this:
[worker.oci]
enabled = true
platforms = ["linux/amd64", "linux/arm64"]
[worker.oci.snapshotter]
# Use "native" for macOS to leverage APFS snapshots.
# BuildKit will automatically select the appropriate native snapshotter
# based on the OS. For Linux, this might be btrfs or zfs.
"linux/amd64" = "native"
"linux/arm64" = "native"
The key levers you control are the platforms and the corresponding snapshotter types. BuildKit uses these mappings to decide which snapshotting technology to employ for a given build architecture. Choosing the right snapshotter can significantly impact build performance, especially for large codebases or complex dependency trees. For instance, stargz can dramatically speed up builds that pull from remote registries by only downloading the necessary parts of a layer on demand.
When BuildKit encounters a build step that requires a filesystem, it queries the configured snapshotter for the appropriate platform. The snapshotter then either provides a fresh copy of the base layer or, more commonly, a copy-on-write view on top of an existing layer. Any modifications made during the build step are written to this COW layer. Upon completion of the step, this COW layer can be committed as a new immutable snapshot, ready to be used by subsequent steps or even future builds. This process is what allows BuildKit to cache intermediate build states effectively, avoiding redundant work.
The trickiest part of snapshotter configuration is understanding how it interacts with storage drivers and the underlying filesystem. While overlayfs is common, its performance can degrade if the underlying filesystem isn’t optimized for it, or if it’s running on certain virtualized environments. Sometimes, switching to a different snapshotter (like native if available) or ensuring your underlying storage is configured for optimal performance (e.g., using SSDs, appropriate mount options) is the real performance win, even if the buildkitd.toml entry remains the same.
The next problem you’ll likely encounter is managing the lifecycle and pruning of these build cache snapshots.