CircleCI workflows are not just a way to organize your jobs; they’re the core orchestration engine that determines when and how your jobs run, offering a surprising amount of control beyond simple sequential execution.
Let’s see this in action. Imagine a typical CI setup where you want to build, test, and then deploy.
version: 2.1
jobs:
build-and-test:
docker:
- image: cimg/node:18.17.0
steps:
- checkout
- run:
name: Install Dependencies
command: npm install
- run:
name: Run Tests
command: npm test
deploy-staging:
docker:
- image: cimg/node:18.17.0
steps:
- checkout
- run:
name: Deploy to Staging
command: ./scripts/deploy-staging.sh
workflows:
version: 2
build_test_and_deploy:
jobs:
- build-and-test
- deploy-staging:
requires:
- build-and-test
In this configuration, the workflows section defines a single workflow named build_test_and_deploy. It lists two jobs: build-and-test and deploy-staging. The requires: - build-and-test line for deploy-staging is crucial. It tells CircleCI that the deploy-staging job cannot start until build-and-test has completed successfully. This is the simplest form of dependency management.
This structure allows for much more complex relationships. You can have jobs that run in parallel, jobs that only run on specific branches, and jobs that only run after a manual approval.
Consider this more advanced workflow:
version: 2.1
jobs:
build:
docker:
- image: cimg/go:1.20.5
steps:
- checkout
- run: go build -o myapp ./...
test:
docker:
- image: cimg/go:1.20.5
steps:
- checkout
- run: go test ./...
lint:
docker:
- image: cimg/go:1.20.5
steps:
- checkout
- run: golint ./...
deploy-production:
docker:
- image: cimg/go:1.20.5
steps:
- checkout
- run: ./scripts/deploy-production.sh
workflows:
version: 2
ci:
jobs:
- build
- test:
requires:
- build
- lint:
requires:
- build
- deploy-production:
requires:
- test
- lint
filters:
branches:
only: main
Here, test and lint both depend on build and can run in parallel after build finishes. The deploy-production job, however, requires both test and lint to have succeeded. Furthermore, it’s restricted to run only on the main branch using filters. This is how you build sophisticated pipelines where different stages must be met before proceeding, and where branching strategies influence execution.
The power of workflows comes from their ability to define these relationships explicitly. A job can depend on multiple other jobs, creating a directed acyclic graph (DAG) of execution. CircleCI intelligently schedules jobs that have their dependencies met, maximizing parallelism and minimizing wait times. You can also use or conditions within requires to specify that a job needs any one of a set of jobs to succeed, not necessarily all of them.
One subtle but powerful aspect of workflows is the concept of "filters." Beyond just branching, you can filter jobs based on commit messages (using regular expressions), or even trigger jobs on tags. This allows for fine-grained control over when specific actions, like deployments or notifications, occur. For instance, a filter could be set up to only deploy when a commit message includes [deploy-prod], ensuring manual intention for critical releases.
The matrix feature, when used within jobs, further expands workflow capabilities by allowing you to run a single job configuration across multiple combinations of parameters, such as different language versions or environments. These matrixed jobs then appear as individual, distinct jobs within the workflow graph, respecting their defined dependencies.
When you define a workflow, you’re essentially telling CircleCI the dependencies between your atomic units of work (jobs) and the conditions under which they should execute. This declarative approach makes complex pipelines manageable and auditable directly from your .circleci/config.yml file.
The next logical step in mastering CircleCI orchestration is understanding how to manage secrets and pass data between jobs, which often involves context and artifact usage.