BuildKit can run as a non-root user, but it requires specific configurations to ensure security and proper operation.
Here’s a breakdown of how to get BuildKit running without root privileges, covering the essential setup and potential pitfalls.
Understanding the Core Problem
BuildKit, by default, often relies on root privileges for several reasons:
- Container Runtime Access: It needs to interact with the Docker daemon or a compatible container runtime, which typically runs as root.
- Filesystem Operations: Creating and managing build contexts, caching layers, and output artifacts can involve operations that require elevated permissions.
- Network Access: Sometimes, building images requires privileged network access for downloading dependencies or communicating with registries.
Running BuildKit as a non-root user means you need to delegate or replicate these necessary permissions in a secure, user-specific manner.
The Non-Root BuildKit Setup
The primary way to run BuildKit without root is by using its client-server architecture and configuring the server to run under a specific user.
1. Install BuildKit
First, ensure BuildKit is installed on your system. This usually involves installing the buildkitd binary.
# Example installation for Debian/Ubuntu
apt-get update && apt-get install -y buildkit-daemon
2. Configure the BuildKit Daemon (buildkitd)
The buildkitd daemon needs to be configured to run as a non-root user. This is typically done via a configuration file, often located at /etc/buildkit/buildkitd.toml.
Here’s a sample buildkitd.toml for non-root operation:
# /etc/buildkit/buildkitd.toml
# Disable rootless mode if you're explicitly running as a user,
# but this is more for when buildkitd itself is launched by a rootless daemon.
# For manual non-root daemon startup, this isn't strictly necessary,
# but good to be aware of.
# rootless = true
# Define a specific user/group for buildkitd to run as.
# This requires that the user and group exist and have appropriate permissions.
# user = "buildkit:buildkit" # Example: if you create a 'buildkit' user/group
# If you're running buildkitd directly as a non-root user (e.g., via systemd user service),
# you might not need the 'user' directive here, as the process will inherit the user's ID.
# Configure the cache directory to be owned by the non-root user.
# This is crucial. The directory must be writable by the user running buildkitd.
[worker.oci]
# Example: pointing to a user-owned directory
cache-dir = "/home/youruser/.cache/buildkit/worker"
[worker.containerd]
# If using containerd as a backend, ensure its paths are accessible.
# This is less common for simple non-root setups.
# Configure storage driver if needed. Default is usually fine.
# Network configuration: buildkitd needs to listen on a TCP port.
# Ensure this port is accessible and not already in use by other services.
# By default, it might try to listen on 127.0.0.1:8080 or similar.
# You can explicitly set it:
# [grpc]
# address = ["tcp://127.0.0.1:8080"]
# Logging configuration
[debug]
# Log level can be set to 'debug', 'info', 'warn', 'error'
level = "info"
# If you want to use rootless Docker daemon as the backend, you need to point to its socket.
# This is a common scenario for true rootless builds.
# [rootless]
# daemon-mode = "docker"
# docker-socket = "/run/user/1000/docker.sock" # Example path for rootless Docker
# For Podman as a backend:
# [rootless]
# daemon-mode = "podman"
# podman-endpoint = "unix:///run/user/1000/podman/podman.sock" # Example path
Key Configuration Points:
cache-dir: This is arguably the most critical setting. The directory specified for caching must be owned by the user runningbuildkitd. Ifbuildkitdis run asyouruser, then/home/youruser/.cache/buildkit/workermust be owned byyouruser:youruser.userdirective: If you are startingbuildkitdas root and want it to drop privileges to a specific user (e.g., via a systemd service file), use this. If you are startingbuildkitddirectly as a non-root user (e.g.,sudo -u youruser buildkitd), then this directive is usually omitted, and the process runs with the privileges ofyouruser.- Backend Configuration (
[rootless]): If you want to leverage an existing rootless container runtime (like rootless Docker or Podman), you need to specify its socket path. This allows BuildKit to delegate container creation and execution to that runtime.
3. Create Necessary Directories and Set Permissions
Before starting buildkitd, ensure the directories it needs are set up correctly.
# As the user who will run buildkitd (e.g., 'youruser')
mkdir -p /home/youruser/.cache/buildkit/worker
chown -R youruser:youruser /home/youruser/.cache/buildkit
4. Start the BuildKit Daemon
You can start buildkitd in several ways:
-
Manually (for testing):
# As your non-root user buildkitd --config /etc/buildkit/buildkitd.tomlThis is good for initial testing. You’ll see output directly in your terminal.
-
Using
systemd(user service): This is the recommended approach for persistent, background operation. Create a service file, e.g.,~/.config/systemd/user/buildkit.service:[Unit] Description=BuildKit Daemon Documentation=https://github.com/moby/buildkit After=network.target [Service] Type=simple ExecStart=/usr/bin/buildkitd --config /etc/buildkit/buildkitd.toml Restart=on-failure RestartSec=5 # Optional: Set environment variables if buildkitd needs them # Environment="PATH=/usr/local/bin:/usr/bin:/bin" [Install] WantedBy=default.targetThen enable and start it:
systemctl --user enable buildkit.service systemctl --user start buildkit.service systemctl --user status buildkit.service
5. Configure the BuildKit Client (buildctl)
Your build clients (e.g., buildctl, Docker CLI configured to use BuildKit) need to connect to the running buildkitd instance.
By default, buildkitd might expose a gRPC API on localhost:8080. You can configure your client to use this.
-
Using
buildctl:# Set the BuildKit endpoint as an environment variable export BUILDKIT_HOST="tcp://localhost:8080" # Now run your build commands buildctl build --frontend dockerfile.v1 --local context=. --local dockerfile=. --output type=image,name=my-nonroot-image:latest -
Using Docker CLI: If you want the
docker buildcommand to use your non-root BuildKit, you need to configure the Docker CLI. Edit~/.docker/config.json:{ "features": { "buildkit": true }, "buildkit-daemon.address": "tcp://localhost:8080" }With this,
docker buildwill attempt to connect to your specifiedbuildkitdinstance.
Common Causes for Failure and How to Fix Them
-
Permission Denied for Cache Directory:
- Diagnosis: Check
buildkitdlogs (e.g.,journalctl --user -u buildkit.service -for terminal output) for errors likepermission deniedwhen accessing/path/to/cache. - Fix: Ensure the cache directory specified in
buildkitd.tomlis owned by the user runningbuildkitd.# Example: If buildkitd runs as 'youruser' chown -R youruser:youruser /home/youruser/.cache/buildkit - Why it works: BuildKit needs to write build cache layers and intermediate data. If the user running the daemon doesn’t own the cache directory, it cannot perform these essential write operations.
- Diagnosis: Check
-
Port Binding Issues:
- Diagnosis:
buildkitdlogs might showaddress already in useor errors related to binding to0.0.0.0:8080. - Fix:
- If running manually, ensure no other process is using port 8080.
- If using
systemdand it fails, checksudo netstat -tulnp | grep 8080orsudo lsof -i :8080. - Change the
grpc.addressinbuildkitd.tomlto an unused port, e.g.,address = ["tcp://127.0.0.1:8081"].
- Why it works: The gRPC server needs a unique, available network port to listen for client connections.
- Diagnosis:
-
Backend Runtime Not Found or Inaccessible (Rootless Docker/Podman):
- Diagnosis: Build errors indicating the container runtime (Docker, Podman) couldn’t be contacted or failed to create containers. Logs might mention
docker.sockorpodman.socknot found or permission denied. - Fix:
- Ensure your rootless Docker or Podman daemon is running.
- Verify the socket path in
buildkitd.toml([rootless] docker-socketorpodman-endpoint) is correct for your user’s session. - Ensure your user has permissions to access that socket file (e.g.,
chmod 666 /run/user/1000/docker.sockif necessary, though often group membership is better).
- Why it works: BuildKit, when configured for rootless backends, acts as a client to the rootless container runtime. It needs to correctly locate and communicate with that runtime’s API socket.
- Diagnosis: Build errors indicating the container runtime (Docker, Podman) couldn’t be contacted or failed to create containers. Logs might mention
-
Docker CLI Not Connecting to BuildKit:
- Diagnosis:
docker buildcommands still behave like the old builder, or you getconnection refusederrors when trying to build. - Fix: Double-check your
~/.docker/config.jsonfor the correctbuildkit-daemon.addressand ensurefeatures.buildkitistrue. Restart your Docker client or terminal session. - Why it works: The Docker CLI needs to be explicitly told to use BuildKit and where to find its API.
- Diagnosis:
-
Build Context Permissions:
- Diagnosis: Build errors like
file not foundorpermission deniedfor files within your build context whenbuildkitdis trying to read them. - Fix: Ensure the user running
buildkitd(or the user whosesystemduser service is running) has read permissions on the build context directory and its contents.# Example: If build context is in your home directory chmod -R u+rX /home/youruser/my-project/ - Why it works: The build process needs access to the source files, Dockerfile, and any other assets included in the build context.
- Diagnosis: Build errors like
The Next Hurdle: Multi-Stage Builds and Network Access
Once you have BuildKit running successfully without root, the next common challenge is managing multi-stage builds where intermediate stages might require specific network access or filesystem operations that are more restricted for non-root users. You might encounter issues with downloading dependencies in a secure way, or with cross-stage file copying if permissions aren’t handled meticulously.