BuildKit’s max-parallelism isn’t just about how many build steps run at once; it directly dictates how aggressively BuildKit can saturate your CPU and network for dependency downloads and compilations.
Here’s a typical BuildKit build, showing the execution graph and concurrent operations:
{
"vertex": "docker.io/library/alpine:latest",
"children": [
{
"vertex": "base:alpine:latest",
"children": [
{
"vertex": "RUN apk add --no-cache curl",
"children": [
{
"vertex": "RUN curl -fsSL https://example.com/data.tar.gz | tar xz",
"children": [
{
"vertex": "COPY ./app /app",
"children": [
{
"vertex": "RUN make build",
"children": []
}
]
}
]
}
]
}
]
}
]
}
When you run docker buildx build . --output type=docker, BuildKit analyzes this graph. If max-parallelism is set low, say to 4, it will only execute four operations (like RUN apk add, RUN curl, COPY, or RUN make build) concurrently, even if you have 16 CPU cores and a gigabit connection. Increasing max-parallelism tells BuildKit to try and fill all available resources.
The primary problem BuildKit solves is slow, inefficient Docker image builds. By default, Docker’s builder executes instructions sequentially, even if they are independent. BuildKit, with its advanced execution engine, can identify and run independent build steps in parallel. This is crucial for complex Dockerfiles with many RUN commands, multiple COPY operations, or external resource fetches. Without proper tuning, you might be leaving significant build time on the table, especially on powerful machines or in CI/CD environments where build speed directly impacts developer velocity and cost.
Here’s how it works internally: BuildKit builds a directed acyclic graph (DAG) of your build steps. Each node in the DAG represents an operation (e.g., RUN, COPY, ADD). BuildKit then uses a scheduler to traverse this DAG. The scheduler identifies nodes that have all their dependencies met (i.e., all preceding nodes in the graph have successfully completed) and are ready for execution. The max-parallelism setting directly limits the number of these ready-to-execute nodes that the scheduler will dispatch concurrently. When a node finishes, its successors become candidates for execution, and the scheduler picks new ready nodes up to the max-parallelism limit.
The key levers you control are:
max-parallelism: This is the most direct control. It’s an integer specifying the maximum number of build operations that can run concurrently.- BuildKit Daemon Configuration: You can set
max-parallelismglobally for the BuildKit daemon, which affects all builds run by that daemon. This is often done in~/.config/buildx/buildkitd.toml. - Per-Build Command Line Flag: You can override the daemon setting for a specific build using
docker buildx build --max-parallelism <N> .... DOCKER_BUILDKIT=1: This environment variable ensures that the experimental BuildKit builder is used fordocker buildcommands if it’s not the default.
Let’s say you have a machine with 16 CPU cores and a fast SSD. Your Dockerfile has 20 independent RUN commands that involve compiling code. If max-parallelism is 4, you’re only utilizing 4 cores effectively for these compilation steps. By increasing max-parallelism to 16 (or slightly higher, considering system overhead), you instruct BuildKit to try and run all 20 RUN commands concurrently, as long as their dependencies are met. This can dramatically reduce the total build time by allowing more CPU cores to be utilized simultaneously.
You might think that simply setting max-parallelism to a very high number, like 100, is always the best approach. However, BuildKit’s scheduler is intelligent. It doesn’t just spawn threads; it manages actual build operations. If your build involves many steps that are I/O bound (like downloading packages from a slow repository) or if you have limited network bandwidth, an excessively high max-parallelism can lead to contention for these resources, potentially slowing down the build more than helping it. BuildKit’s internal heuristics try to balance CPU, disk, and network I/O, but an aggressive setting can push these resources beyond their optimal operating point. The sweet spot is often close to the number of CPU cores you have, but it’s worth experimenting.
The next concept you’ll likely encounter is optimizing the use of BuildKit’s cache, specifically understanding how to leverage cache-mounts for faster layer caching and how to purge stale cache entries.