Secrets like API keys and passwords can accidentally leak into your CircleCI job logs, which are often accessible to a wider audience than intended.
Here’s what a CircleCI job log looks like when a secret is printed, and how we’re going to prevent it. Imagine this snippet from a npm install step:
> npm install --save-dev some-package
>
> npm notice
> npm notice New minor version of npm available! 6.14.15 -> 6.14.17
> npm notice Run npm install -g npm@6.14.17 to update
>
> > some-package@1.0.0 postinstall /path/to/project/node_modules/some-package
> > node ./postinstall.js
>
> Successfully installed some-package.
>
> Detected AWS credentials.
> Access Key ID: AKIAIOSFODNN7EXAMPLE
> Secret Access Key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
That last part, the Secret Access Key, is what we want to avoid.
The core mechanism CircleCI provides for handling secrets is environment variables. When you define a secret in CircleCI’s UI, it’s stored as an environment variable that’s injected into your job’s execution environment. The trick is to ensure that these variables are never accidentally printed or logged.
Use CircleCI’s Built-in Secret Management
The first and most crucial step is to never hardcode secrets directly in your configuration files (.circleci/config.yml) or your application code.
- Navigate to your project’s settings in CircleCI.
- Go to "Security" > "Environment Variables".
- Click "Add Environment Variable".
- Enter a name for your variable (e.g.,
AWS_ACCESS_KEY_ID,STRIPE_SECRET_KEY). - Paste the secret value into the "Value" field.
- Click "Add and Save".
Now, instead of writing AWS_ACCESS_KEY_ID: AKIAIOSFODNN7EXAMPLE in your config.yml, you’ll reference it as an environment variable.
Masking Secrets in config.yml
CircleCI automatically attempts to mask secrets that are passed as environment variables. However, this masking is not foolproof and depends on how the output is formatted.
If you define an environment variable in your .circleci/config.yml like this:
jobs:
build:
docker:
- image: cimg/node:16.14.2
environment:
MY_SECRET_KEY: !secret my-project-secret-key-name
steps:
- run: echo "My secret is $MY_SECRET_KEY" # This will be masked by CircleCI
CircleCI’s UI will attempt to replace AKIAIOSFODNN7EXAMPLE with *** in the job logs. This works for direct echo statements.
However, if a tool or script reads the environment variable and then prints it, the masking might fail. For example, if a postinstall.js script reads AWS_ACCESS_KEY_ID and prints it, you’ll see the raw value.
Preventing Accidental Printing in Scripts
This is where most leaks happen. Scripts, build tools, or even your application code might inadvertently print secrets.
Common Cause 1: Debugging Statements
You might have added console.log or print statements during development that are never removed.
- Diagnosis: Carefully review all scripts executed within your CircleCI jobs (e.g.,
npm scripts,bash scripts,Python scripts). Search for any commands that print environment variables or hardcoded secrets. - Fix: Remove or comment out any
console.logorprintstatements that output secrets. For example, in Node.js:// console.log(`Access Key ID: ${process.env.AWS_ACCESS_KEY_ID}`); // Remove this line console.log("AWS credentials loaded."); // Keep this safe line - Why it works: The secret is no longer explicitly sent to standard output.
Common Cause 2: Logging Libraries Some logging libraries, if misconfigured, might include sensitive environment variables in their output.
- Diagnosis: Check your application’s logging configuration. Are there any settings that tell the logger to include environment variables?
- Fix: Configure your logging library to exclude sensitive environment variables from being logged. For example, if using
winstonin Node.js, ensure you’re not serializingprocess.envdirectly into your log messages. - Why it works: The logging mechanism is prevented from ever seeing or processing the secret data for output.
Common Cause 3: Third-Party Tools Tools used in your CI pipeline might have verbose logging that can expose secrets.
- Diagnosis: Run the specific command that produced the leaked secret in a local, isolated environment. Observe its output carefully.
- Fix: Look for flags or configuration options within the tool to reduce logging verbosity or to specifically exclude secrets. For example, some AWS CLI commands have
--no-cli-pageror--quietoptions. You might also need to useset +xin bash scripts to turn off command echoing.set +x # Turn off command echoing for subsequent commands aws s3 sync my-bucket . set -x # Turn command echoing back on if needed later - Why it works: This prevents the shell itself from echoing commands that might contain secrets, and the tool’s output is less verbose.
Common Cause 4: Incorrect Environment Variable Naming
If a variable is named something generic like API_KEY and you think it’s a secret but it’s not configured as one in CircleCI, it might get printed if used in a script. Conversely, if a non-secret variable is accidentally named SECRET_KEY, it might be masked unnecessarily.
- Diagnosis: Cross-reference all environment variables used in your
config.ymland scripts with the list of variables defined in CircleCI’s UI. - Fix: Ensure that any value intended to be secret is defined in CircleCI’s UI as an environment variable. Rename variables for clarity if needed.
- Why it works: Consistent naming and proper configuration ensure CircleCI knows which variables to mask.
Common Cause 5: Dependency Installation Output Some package managers or build tools might print sensitive information during dependency installation if those dependencies themselves require secrets for their own build processes (though this is less common for typical application secrets).
- Diagnosis: If you see secrets during
npm install,pip install, or similar commands, investigate the specific package being installed. - Fix: Ensure that any dependencies requiring secrets are configured to use them via environment variables and that their installation scripts do not echo these secrets. This is often a fix within the dependency itself, or you might need to use a custom build process that injects secrets more carefully.
- Why it works: It prevents the installation process from exposing secrets that might be temporarily used or exposed by the dependency’s build steps.
Common Cause 6: Shell Debugging (set -x)
If you’ve enabled shell debugging with set -x in your .circleci/config.yml and not properly masked secrets, the shell will print the commands it’s executing, including the secrets.
- Diagnosis: Look for
set -xin yourrunsteps withinconfig.yml. - Fix: Either remove
set -xif not needed for debugging, or ensure that any lines using secrets are prefixed with+(whichset -xadds) and then potentially masked or handled carefully. The best practice is to avoidset -xin production jobs or to carefully craft commands that don’t expose secrets. A common workaround is to useecho "..." | commandor to pipe output togrep -vif you must see intermediate steps.
To fix this specific example, you would remove- run: name: Deploy command: | set -x # Debugging enabled MY_SECRET_VALUE="my-super-secret-token" # This will be printed by set -x echo "Deploying with token..." # ... actual deployment command using MY_SECRET_VALUE ... set +x # Disable debuggingMY_SECRET_VALUE="my-super-secret-token"and ensureMY_SECRET_VALUEis an environment variable injected by CircleCI.
The Next Problem: Incorrectly Masked Variables
After you’ve fixed all the leaks, you might encounter a new issue where CircleCI still shows *** for a variable that you know is not a secret. This usually happens when a non-secret variable is accidentally named to look like a secret (e.g., API_KEY, PASSWORD) and CircleCI’s heuristic masking picks it up.