You can pass parameters between CircleCI pipelines and jobs by using the workflows and jobs configuration in your .circleci/config.yml file, specifically with the parameters and context keywords.
Here’s a CircleCI config demonstrating how to pass parameters between jobs within a pipeline:
version: 2.1
parameters:
greeting_message:
type: string
default: "Hello from the pipeline!"
jobs:
say_hello:
parameters:
message:
type: string
default: "Default greeting"
docker:
- image: cimg/base:2023.07
steps:
- run: echo "<< parameters.message >>"
run_script:
parameters:
input_message:
type: string
docker:
- image: cimg/base:2023.07
steps:
- run: echo "Received: << parameters.input_message >>"
workflows:
my_workflow:
parameters:
pipeline_param:
type: string
default: "This is a pipeline default"
jobs:
- say_hello:
name: greet_world
message: "<< pipeline.parameters.pipeline_param >>" # Pass pipeline param to job param
- run_script:
name: process_greeting
requires:
- greet_world
input_message: << jobs.greet_world.output.message >> # This won't work directly as jobs don't output parameters like this
# To pass output from one job to another, you'd typically use artifacts or custom keys.
# For demonstration, let's assume 'greet_world' somehow outputs a value we can access.
# A common pattern is to use the `store_output` command in the first job
# and `add_ssh_keys` or similar for more complex data passing.
# A more realistic example of passing data:
- say_hello:
name: greet_with_custom_message
message: "Custom message for this job"
- run_script:
name: process_custom_greeting
requires:
- greet_with_custom_message
input_message: << jobs.greet_with_custom_message.parameters.message >> # Accessing job parameters passed to another job
In this example:
- Pipeline Parameters: The
parametersblock at the top level defines parameters that can be used across the entire pipeline, including within workflows. - Job Parameters: Each
jobcan define its ownparameters. These are specific to that job and can be set when the job is invoked within a workflow. - Workflow Parameters: The
workflowsblock can also have its ownparameters, which can then be passed down to the jobs within that workflow. - Passing Parameters:
- You can pass pipeline parameters to job parameters using
<< pipeline.parameters.parameter_name >>. - You can pass parameters from one job to another (if they are defined as job parameters) using
<< jobs.job_name.parameters.parameter_name >>.
- You can pass pipeline parameters to job parameters using
Crucially, directly passing arbitrary output from one job’s execution to another job’s input parameter isn’t as straightforward as accessing its defined parameters. For actual data passing (like the result of a command), you typically use:
store_outputandadd_ssh_keys(oradd_ssh_keysfor more general context): The first job canstore_outputwith a specific key, and the second job can then access this output. This is often used for small pieces of data like version numbers or build artifacts.- Artifacts: For larger data, you can store artifacts in the first job and download them in the second.
- External Storage: For significant data, use external services like S3, GCS, etc.
Let’s refine the example to show a more practical data-passing scenario using store_output:
version: 2.1
jobs:
generate_data:
docker:
- image: cimg/base:2023.07
steps:
- run:
name: Generate a dynamic value
command: |
echo "HELLO_WORLD" > output_value.txt
echo "dynamic_value=$(cat output_value.txt)" >> "$BASH_ENV"
- run: echo "Generated value is: $dynamic_value"
- persist_to_workspace:
root: .
paths:
- output_value.txt # Persist the file containing the value
- run:
name: Store the value as an output parameter
command: |
VALUE=$(cat output_value.txt)
echo "$VALUE" > "$BASH_ENV" # This is a common trick to make values available in subsequent steps within the same job
circleci-agent step publish --key "dynamic_output_key" --value "$VALUE" # Use the CLI to publish a key-value pair
consume_data:
docker:
- image: cimg/base:2023.07
steps:
- attach_workspace:
at: ~/project # Attach the workspace from the previous job
- run:
name: Access the dynamic value
command: |
# This is where you'd typically retrieve from an external source or an artifact.
# For direct parameter passing demonstration, we'll simulate retrieval.
# In a real scenario, you'd use `circleci-agent step get --key "dynamic_output_key"`
# or retrieve from persisted workspace/artifacts.
echo "Simulating retrieval of dynamic_output_key..."
# Let's assume the previous job published "dynamic_output_key" with value "HELLO_WORLD"
# In a real workflow, the 'consume_data' job would need to be configured to access this.
# This is often done via custom contexts or by passing configuration that references the output.
workflows:
main_workflow:
jobs:
- generate_data
- consume_data:
requires:
- generate_data
# To actually pass the output, you'd need a mechanism.
# The `circleci-agent step publish` command makes the value available
# via the API, which can be queried by other jobs if configured.
# A more direct way for simple values is often to use `store_artifacts`
# and then process the artifact, or use a custom context.
# For this example, we'll illustrate how to access it IF it were stored in a context.
# You would typically define a context and use `context: <your-context-name>`
# in the job definition and then `circleci-agent context store` in the previous job.
The circleci-agent step publish --key "dynamic_output_key" --value "$VALUE" command is the key here. It makes the dynamic_output_key and its value available through the CircleCI API. While other jobs can’t directly reference this dynamically published value as a parameter in the same way they reference static parameters, you can use this mechanism in conjunction with custom contexts or by having the consuming job query the API (though this is less common for simple parameter passing).
A more common pattern for passing simple, dynamic values between jobs is to:
-
generate_datajob:- Calculate the value.
echo "$VALUE" > "$BASH_ENV"(makes it available in subsequent steps of the same job).persist_to_workspaceto save a file containing the value.circleci-agent step publish --key "my_dynamic_value" --value "$VALUE"to make it available via API.
-
consume_datajob:attach_workspaceto get the file.- Read the value from the file.
- Alternatively, you could use
circleci-agent context storeingenerate_dataandcontext: <your-context-name>inconsume_datato pass values via custom contexts.
The most robust way to pass dynamic values between jobs for use as parameters in subsequent jobs is often through custom contexts.
Here’s how that would look:
1. In generate_data job (or any job that produces the value):
jobs:
generate_data:
docker:
- image: cimg/base:2023.07
steps:
- run:
name: Generate dynamic value
command: |
echo "BUILD_VERSION=1.2.3" > version.txt
VERSION=$(cat version.txt)
echo "Version is: $VERSION"
- run:
name: Store value in CircleCI Context
command: |
VERSION=$(cat version.txt)
# This command requires the CircleCI CLI to be available and authenticated.
# It writes to a context named 'my-app-context'
circleci-agent context store --context my-app-context --key BUILD_VERSION --value "$VERSION"
2. In consume_data job (the job that needs the value):
jobs:
consume_data:
docker:
- image: cimg/base:2023.07
parameters: # Define a parameter to receive the value
build_version:
type: string
default: "unknown"
steps:
- run: echo "The build version is: << parameters.build_version >>"
workflows:
main_workflow:
jobs:
- generate_data
- consume_data:
requires:
- generate_data
context: my-app-context # Specify the context to load
# Now, map the context value to the job parameter
build_version: << pipeline.parameters.BUILD_VERSION >> # This syntax is incorrect for context values
# Correct way: You cannot directly map context values to job parameters like this in the workflow definition.
# Instead, context values are typically accessed via environment variables within the job.
# Let's revise the 'consume_data' job to show this.
Revised consume_data job and workflow:
version: 2.1
jobs:
generate_data:
docker:
- image: cimg/base:2023.07
steps:
- run:
name: Generate dynamic value
command: |
echo "BUILD_VERSION=1.2.3" > version.txt
VERSION=$(cat version.txt)
echo "Version is: $VERSION"
- run:
name: Store value in CircleCI Context
command: |
VERSION=$(cat version.txt)
# Ensure the CircleCI CLI is available and authenticated.
# This command writes to a context named 'my-app-context'
circleci-agent context store --context my-app-context --key BUILD_VERSION --value "$VERSION"
consume_data:
docker:
- image: cimg/base:2023.07
steps:
- run:
name: Access value from context
command: |
# Values stored in a context become available as environment variables
# prefixed with CIRCLECI_CONTEXT_<CONTEXT_NAME_UPPERCASE>_<KEY_UPPERCASE>
echo "The build version from context is: $CIRCLECI_CONTEXT_MY_APP_CONTEXT_BUILD_VERSION"
workflows:
main_workflow:
jobs:
- generate_data
- consume_data:
requires:
- generate_data
context: my-app-context # This loads the context, making its variables available.
In this revised context example:
- The
generate_datajob usescircleci-agent context storeto saveBUILD_VERSION=1.2.3into a context namedmy-app-context. - The
consume_datajob declarescontext: my-app-context. This action makes all key-value pairs stored inmy-app-contextavailable as environment variables within that job. - The environment variable will be named
CIRCLECI_CONTEXT_MY_APP_CONTEXT_BUILD_VERSION, and its value will be1.2.3. You can then reference this environment variable directly in yourruncommands.
This method of using contexts is the most common and recommended way to pass dynamic, configuration-like values between jobs in CircleCI.