You can generate your CircleCI configuration file (.circleci/config.yml) on the fly during pipeline execution.
Let’s say you have a monorepo with several distinct services, and you want to run specific integration tests for only those services that have changed in a given commit. Instead of manually maintaining a config.yml that accounts for every possible combination, you can dynamically generate it.
Here’s a simplified example of how you might do this, focusing on the core idea. Imagine you have a script, generate_config.sh, that lives in your repository.
#!/bin/bash
# This script simulates detecting changes and generating a CircleCI config.
# In a real scenario, you'd use git diff or similar tools.
# Simulate detecting changed services
CHANGED_SERVICES=""
if [[ "$1" == "service-a" ]]; then
CHANGED_SERVICES="service-a"
elif [[ "$1" == "service-b" ]]; then
CHANGED_SERVICES="service-b"
elif [[ "$1" == "all" ]]; then
CHANGED_SERVICES="service-a service-b"
fi
echo "Detected changes in: $CHANGED_SERVICES"
# Start building the YAML output
echo "version: 2.1"
echo "jobs:"
# Define a base job that can be reused
echo " build_and_test_service:"
echo " docker:"
echo " - image: cimg/node:18.17.0"
echo " steps:"
echo " - checkout"
echo " - run:"
echo " name: Install Dependencies for \${SERVICE_NAME}"
echo " command: |"
echo " echo 'Installing dependencies for \${SERVICE_NAME}'"
echo " # Simulate installing dependencies"
echo " sleep 2"
echo " - run:"
echo " name: Run Tests for \${SERVICE_NAME}"
echo " command: |"
echo " echo 'Running tests for \${SERVICE_NAME}'"
echo " # Simulate running tests"
echo " sleep 3"
# Dynamically create jobs and workflows based on detected changes
if [[ -n "$CHANGED_SERVICES" ]]; then
echo "workflows:"
echo " version: 2"
echo " build_and_test:"
echo " jobs:"
for SERVICE in $CHANGED_SERVICES; do
# Create a unique job name for each service
JOB_NAME="build_test_${SERVICE}"
echo " - build_and_test_service:"
echo " name: ${JOB_NAME}" # Explicitly name the job instance
echo " context: org-billing-context # Example context"
echo " environment:"
echo " SERVICE_NAME: ${SERVICE}"
done
else
echo "workflows:"
echo " version: 2"
echo " build_and_test:"
echo " jobs:"
echo " - run:"
echo " name: No services changed, nothing to do."
echo " command: echo 'Skipping build and test.'"
fi
Now, you’d configure your CircleCI pipeline to execute this script before it tries to parse your .circleci/config.yml. This is done using the setup step in your .circleci/config.yml if you’re using the v2.1 config or by having a separate pipeline that generates the config and then triggers another pipeline. A more common pattern is to have a "bootstrap" job in your main config.yml that generates the actual config.yml for the subsequent jobs.
Here’s a conceptual .circleci/config.yml that starts the process:
version: 2.1
orbs:
# Example: Use an orb for dynamic config generation if available,
# otherwise, use a shell script as shown below.
# dynamic-config: circleci/dynamic-config@x.y.z
jobs:
generate_dynamic_config:
docker:
- image: cimg/base:stable # A minimal image for scripting
steps:
- checkout
- run:
name: Generate config.yml
command: |
# Detect changed services (e.g., using git diff against main branch)
# For demonstration, we'll hardcode or pass a parameter.
# In a real CI, you'd do something like:
# CHANGED=$(git diff --name-only main...HEAD | grep '^services/' | cut -d'/' -f2 | sort -u)
# echo "Detected changes in: $CHANGED"
# For this example, let's assume we know service-a changed.
./scripts/generate_config.sh service-a > .circleci/generated_config.yml
- persist_to_workspace:
root: .circleci
paths:
- generated_config.yml
workflows:
version: 2
main_workflow:
jobs:
- generate_dynamic_config
- run:
name: Dynamic Build and Test
requires:
- generate_dynamic_config
filters:
branches:
ignore:
- main # Or your primary branch
# This is the crucial part: we use the generated config.yml
# CircleCI will look for this file in the workspace.
# However, CircleCI's native execution doesn't directly support
# a 'config.yml' that is *generated* and then *used* by the same pipeline run
# in this direct way. This pattern usually involves triggering a *new* pipeline.
# The standard way to achieve this is by having the *first* job
# generate the config and then trigger a *new* pipeline run with that config.
# This is often done via the CircleCI API.
# A more practical approach within a single pipeline run often involves
# a "setup" job that prepares artifacts or variables, and then subsequent
# jobs that *use* those artifacts. However, the *entire* config.yml
# structure itself is parsed upfront.
# Let's refine the idea to a more direct, albeit less common, pattern for demonstration:
# The `setup` directive in a v2.1 config file allows you to dynamically
# load configuration from a *different* file. This is usually used for
# modularity, but can be adapted.
# If you're using the `setup` directive, your primary config.yml
# would look like this:
# .circleci/config.yml (Primary)
# version: 2.1
# setup: true # This tells CircleCI to look for other config files
# orbs:
# # ... orbs
# jobs:
# generate_config_job:
# docker:
# - image: cimg/base:stable
# steps:
# - checkout
# - run:
# name: Generate config
# command: |
# ./scripts/generate_config.sh service-a > .circleci/dynamic_config.yml
# - persist_to_workspace:
# root: .circleci
# paths:
# - dynamic_config.yml
# workflows:
# version: 2
# setup_workflow:
# jobs:
# - generate_config_job
# - load_dynamic_config:
# requires:
# - generate_config_job
# # This job's configuration would be in .circleci/dynamic_config.yml
# The problem with the `setup: true` approach is that it requires
# the *main* config.yml to be very minimal and then loads others.
# It doesn't quite let you *execute* a job that *generates* the config
# and then have *that same pipeline run* use the generated config for its jobs.
# The most robust and common pattern involves:
# 1. A pipeline trigger (e.g., from a push to main) that runs a job.
# 2. This job checks out code, runs the generation script, and saves the generated config.yml as an artifact or persists it.
# 3. This job then uses the CircleCI API to trigger a *new* pipeline run, passing the generated config.yml as a parameter or uploading it.
# 4. The new pipeline run then executes the jobs defined in the dynamically generated config.
# Let's illustrate the API trigger approach conceptually.
# Your main .circleci/config.yml might look like this:
version: 2.1
jobs:
generate_and_trigger:
docker:
- image: cimg/node:18.17.0 # Or any image with 'curl' and 'jq'
steps:
- checkout
- run:
name: Generate dynamic config.yml
command: |
# Simulate detecting changes. In reality, use git diff.
# For example, to detect changes in 'service-a':
# CHANGED_SERVICES=$(git diff --name-only main...HEAD | grep '^services/service-a/' | cut -d'/' -f2 | sort -u)
# For this example, assume we know service-a changed.
./scripts/generate_config.sh service-a > .circleci/generated_config.yml
echo "Generated config:"
cat .circleci/generated_config.yml
- run:
name: Trigger new pipeline with generated config
command: |
# Replace YOUR_CIRCLECI_TOKEN with an environment variable
# or project-level context variable containing your CircleCI API token.
# Ensure this token has permissions to trigger pipelines.
CIRCLECI_API_TOKEN=$CIRCLECI_API_TOKEN
PROJECT_ID=$(jq -r '.vcs.origin.project_id' <<< "$CI_PIPELINE_TRIGGERED_BY") # Example extraction, might vary
# This part is complex and depends on your VCS (GitHub/Bitbucket) and how you trigger.
# A common way is to use the API to create a new pipeline.
# Example using GitHub API via curl (simplified):
# You'd need the repo owner, repo name, and commit SHA.
# This is highly specific to your VCS and setup.
# For demonstration, let's assume a hypothetical API call:
echo "Simulating API trigger..."
# circleci pipeline trigger --token $CIRCLECI_API_TOKEN \
# --vcs-type github --git-username your-github-user --git-repo your-repo-name \
# --branch $CIRCLE_BRANCH \
# --config .circleci/generated_config.yml # This parameter might not be directly supported like this.
# Instead, you might upload the config as a parameter or use a template.
# A more reliable approach is to store the generated config in S3 or GCS
# and then trigger a pipeline that downloads it.
# Or, use a template mechanism where the generated config is a parameter.
# Let's assume a simplified scenario where you are uploading the config
# as a parameter to a new pipeline run.
# The actual API call would look something like this (conceptual):
curl --request POST \
--url https://circleci.com/api/v2/project/github/YOUR_ORG/YOUR_REPO/pipeline \
--header "Circle-Token: $CIRCLECI_API_TOKEN" \
--header "Content-Type: application/json" \
--data '{
"branch": "'"$CIRCLE_BRANCH"'",
"parameters": {
"generated_config": "'$(cat .circleci/generated_config.yml | sed 's/"/\\"/g' | sed ':a;N;$!ba;s/\n/\\n/g')'"
}
}'
echo "Pipeline trigger simulated."
workflows:
version: 2
trigger_dynamic_pipeline:
jobs:
- generate_and_trigger:
filters:
branches:
only:
- main # Or whichever branch you want to trigger this for
And then, your actual .circleci/config.yml would be designed to receive this parameter and use it. This is where it gets complex because CircleCI parses the config.yml before executing jobs. The setup: true directive is the closest built-in mechanism, but it’s more for modularity.
A more common and flexible pattern is to use a templating engine (like Jinja2, Handlebars, or even simple shell scripting as shown) to generate the config.yml in a separate project or a dedicated pipeline run, and then use that generated file to trigger a new pipeline execution via the CircleCI API. The triggered pipeline would then execute jobs defined by the generated configuration.
The most surprising true thing about this is that CircleCI doesn’t have a direct "execute this script to define the rest of my pipeline jobs" feature within a single pipeline run. You typically generate the config and then use it to spin up a new pipeline execution.
This pattern effectively allows you to have a highly dynamic CI setup where the jobs that run are precisely tailored to the code that has changed, reducing execution time and cost. You control the logic of how the configuration is generated—which services are included, which tests are run, and how they are ordered—by modifying the generation script.
The next concept you’ll likely run into is managing secrets and environment variables across these dynamically generated jobs, ensuring they have the correct access and configurations.