A self-hosted CircleCI runner lets you execute your CI/CD jobs on your own servers, giving you more control over your build environment and potentially reducing costs.
Let’s see a runner in action. Imagine you have a simple Go application that needs to be built and tested.
# .circleci/config.yml
version: 2.1
jobs:
build-and-test:
docker:
- image: cimg/go:1.18
steps:
- checkout
- run:
name: Build
command: go build ./...
- run:
name: Test
command: go test ./...
workflows:
main:
jobs:
- build-and-test
Normally, CircleCI would spin up a Docker container on their infrastructure to run this. With a self-hosted runner, you’d configure CircleCI to tell your runner to pick up this job and execute it on your hardware.
Here’s how you’d set that up. First, on your own infrastructure (e.g., a VM or a bare-metal server), you’ll install the CircleCI runner. You can get the latest binary from the CircleCI documentation.
Once installed, you’ll register it with your CircleCI organization. This involves creating a runner resource in your CircleCI UI and then using the circleci runner install command with the token provided.
# On your self-hosted runner machine
circleci runner install --token <your-registration-token>
This command downloads the runner executable and sets it up as a service. You’ll also need to configure the runner to specify which jobs it can execute. This is done via resource classes. In your .circleci/config.yml, you’d modify the job to request a specific resource class that your runner is configured to handle.
# .circleci/config.yml
version: 2.1
jobs:
build-and-test:
docker:
- image: cimg/go:1.18
resource_class: my-self-hosted-runner # This is the key
steps:
- checkout
- run:
name: Build
command: go build ./...
- run:
name: Test
command: go test ./...
workflows:
main:
jobs:
- build-and-test
On your runner machine, you’d configure it to advertise the my-self-hosted-runner resource class. This is typically done in a configuration file for the runner service, often located at /etc/circleci/runner.yml or similar.
# /etc/circleci/runner.yml (example)
token: <your-registration-token>
api-url: https://circleci.com
resources:
- name: my-self-hosted-runner
cpu: 2
memory: 4Gi
disk: 50Gi
When a workflow starts and encounters the build-and-test job, CircleCI will look for an available self-hosted runner registered with your organization that advertises the my-self-hosted-runner resource class. If one is found, the job’s commands will be executed on that runner’s infrastructure.
The runner itself acts as an agent, polling CircleCI for jobs it’s capable of running. When it finds one, it downloads the necessary Docker image (if specified) or uses the host’s environment, executes the commands, and reports the status back to CircleCI. This means your build artifacts and logs still appear in the CircleCI UI, but the actual computation happens on your hardware.
The real magic is how CircleCI orchestrates this. It’s not just about running commands; it’s about securely passing job definitions, retrieving logs, and managing the lifecycle of the build. The runner client is designed to be resilient, handling network interruptions and ensuring that jobs are picked up and completed. You can even configure multiple runners under the same resource class, providing high availability and load balancing for your CI workloads.
One aspect that often trips people up is the docker executor within the job definition. When using a self-hosted runner with a docker executor, the runner machine itself must have Docker installed and running. The runner client will then use that Docker daemon to pull and run the specified images, effectively creating a Docker-in-Docker or Docker-on-Docker scenario from the perspective of the runner’s host. If you’re not using the docker executor and instead relying on the runner’s host environment (e.g., for native builds), you’d omit the docker section in your job definition.
The next hurdle you’ll likely face is managing secrets and environment variables securely when running on your own infrastructure.