Docker’s build process is notoriously slow, and when you’re doing it in CI, that slowness compounds into wasted time and money. CircleCI’s layer caching is designed to nip this in the bud by intelligently reusing previously built Docker image layers.
Here’s a typical Dockerfile:
FROM ubuntu:20.04
RUN apt-get update && apt-get install -y --no-install-recommends \
package1 \
package2 \
&& rm -rf /var/lib/apt/lists/*
COPY . /app
WORKDIR /app
RUN pip install -r requirements.txt
CMD ["python", "app.py"]
And here’s how you’d configure CircleCI to cache the Docker layers:
version: 2.1
jobs:
build_and_push_docker:
docker:
- image: cimg/base:2021.04
steps:
- checkout
- setup_remote_docker:
version: 19.03.13
- restore_cache:
keys:
- docker-{{ arch }}-{{ .Branch }}-{{ checksum "Dockerfile" }}
- docker-{{ arch }}-{{ .Branch }}
- docker-{{ arch }}
- run:
name: Build Docker Image
command: |
docker build \
--cache-from $IMAGE_NAME \
-t $IMAGE_NAME .
- run:
name: Push Docker Image
command: echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USERNAME" --password-stdin && docker push $IMAGE_NAME
- save_cache:
key: docker-{{ arch }}-{{ .Branch }}-{{ checksum "Dockerfile" }}
paths:
- /var/lib/docker
The setup_remote_docker step is crucial. It provisions a Docker daemon that your job can interact with. When you run docker build, it checks for existing layers. If it finds them in the cache, it uses them instead of re-downloading or re-executing the instructions.
The restore_cache and save_cache steps are where CircleCI’s magic happens.
-
restore_cache: This step looks for a cache matching the providedkeys. The keys are ordered by preference. CircleCI will try to find a cache that exactly matchesdocker-{{ arch }}-{{ .Branch }}-{{ checksum "Dockerfile" }}first. If it can’t find that, it will trydocker-{{ arch }}-{{ .Branch }}, and so on. Thechecksum "Dockerfile"is particularly powerful because it means the cache will only be restored if yourDockerfilehasn’t changed. -
save_cache: After your image is built and potentially pushed, this step saves the Docker daemon’s state. Thepaths: - /var/lib/dockeris the directory where Docker stores its layers. By saving this, you’re essentially saving all the built layers for future builds.
The docker build --cache-from $IMAGE_NAME flag tells Docker to look for layers in the specified image (which should be a previously pushed image from a successful build). This is a more robust way to ensure you’re pulling from a remote source if the local cache isn’t sufficient or if you’re starting a new build environment.
The real power comes from how CircleCI manages these caches. It doesn’t just dump everything. When you save_cache to /var/lib/docker, CircleCI is effectively creating a snapshot of the Docker daemon’s storage. On the next build, restore_cache pulls that snapshot down. If your Dockerfile hasn’t changed, and the COPY commands haven’t changed the files they’re copying, Docker can reuse those cached layers, skipping RUN commands that produced them. This can turn builds that took 10 minutes into ones that take 1 minute.
The trick to making this work efficiently is understanding how the cache keys interact. A more specific key like docker-{{ arch }}-{{ .Branch }}-{{ checksum "Dockerfile" }} will hit more often if your Dockerfile and branch are the same. However, if you change a dependency in requirements.txt (which changes the checksum of the Dockerfile if it’s included in the checksum calculation, or if you have a separate step that depends on it), that specific cache key will miss. The fallback keys (docker-{{ arch }}-{{ .Branch }} and docker-{{ arch }}) then allow Docker to reuse some layers, even if not all.
What most people miss is that save_cache to /var/lib/docker captures the entire Docker daemon’s state for that job. This means it’s not just caching the layers for your specific image but potentially layers from any Docker operations that occurred during that job. This can make your cache keys more effective but also potentially larger. The paths argument in save_cache is critical; without it, you’re not actually saving the Docker layers.
The next thing you’ll likely run into is cache invalidation issues where CircleCI’s cache doesn’t seem to be updating correctly, leading to stale builds.