Secrets like API keys and passwords can accidentally leak into your CircleCI job logs, which are often accessible to a wider audience than intended.

Here’s what a CircleCI job log looks like when a secret is printed, and how we’re going to prevent it. Imagine this snippet from a npm install step:

> npm install --save-dev some-package
>
> npm notice
> npm notice New minor version of npm available! 6.14.15 -> 6.14.17
> npm notice Run npm install -g npm@6.14.17 to update
>
> > some-package@1.0.0 postinstall /path/to/project/node_modules/some-package
> > node ./postinstall.js
>
> Successfully installed some-package.
>
> Detected AWS credentials.
> Access Key ID: AKIAIOSFODNN7EXAMPLE
> Secret Access Key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

That last part, the Secret Access Key, is what we want to avoid.

The core mechanism CircleCI provides for handling secrets is environment variables. When you define a secret in CircleCI’s UI, it’s stored as an environment variable that’s injected into your job’s execution environment. The trick is to ensure that these variables are never accidentally printed or logged.

Use CircleCI’s Built-in Secret Management

The first and most crucial step is to never hardcode secrets directly in your configuration files (.circleci/config.yml) or your application code.

  1. Navigate to your project’s settings in CircleCI.
  2. Go to "Security" > "Environment Variables".
  3. Click "Add Environment Variable".
  4. Enter a name for your variable (e.g., AWS_ACCESS_KEY_ID, STRIPE_SECRET_KEY).
  5. Paste the secret value into the "Value" field.
  6. Click "Add and Save".

Now, instead of writing AWS_ACCESS_KEY_ID: AKIAIOSFODNN7EXAMPLE in your config.yml, you’ll reference it as an environment variable.

Masking Secrets in config.yml

CircleCI automatically attempts to mask secrets that are passed as environment variables. However, this masking is not foolproof and depends on how the output is formatted.

If you define an environment variable in your .circleci/config.yml like this:

jobs:
  build:
    docker:
      - image: cimg/node:16.14.2
    environment:
      MY_SECRET_KEY: !secret my-project-secret-key-name
    steps:
      - run: echo "My secret is $MY_SECRET_KEY" # This will be masked by CircleCI

CircleCI’s UI will attempt to replace AKIAIOSFODNN7EXAMPLE with *** in the job logs. This works for direct echo statements.

However, if a tool or script reads the environment variable and then prints it, the masking might fail. For example, if a postinstall.js script reads AWS_ACCESS_KEY_ID and prints it, you’ll see the raw value.

Preventing Accidental Printing in Scripts

This is where most leaks happen. Scripts, build tools, or even your application code might inadvertently print secrets.

Common Cause 1: Debugging Statements You might have added console.log or print statements during development that are never removed.

  • Diagnosis: Carefully review all scripts executed within your CircleCI jobs (e.g., npm scripts, bash scripts, Python scripts). Search for any commands that print environment variables or hardcoded secrets.
  • Fix: Remove or comment out any console.log or print statements that output secrets. For example, in Node.js:
    // console.log(`Access Key ID: ${process.env.AWS_ACCESS_KEY_ID}`); // Remove this line
    console.log("AWS credentials loaded."); // Keep this safe line
    
  • Why it works: The secret is no longer explicitly sent to standard output.

Common Cause 2: Logging Libraries Some logging libraries, if misconfigured, might include sensitive environment variables in their output.

  • Diagnosis: Check your application’s logging configuration. Are there any settings that tell the logger to include environment variables?
  • Fix: Configure your logging library to exclude sensitive environment variables from being logged. For example, if using winston in Node.js, ensure you’re not serializing process.env directly into your log messages.
  • Why it works: The logging mechanism is prevented from ever seeing or processing the secret data for output.

Common Cause 3: Third-Party Tools Tools used in your CI pipeline might have verbose logging that can expose secrets.

  • Diagnosis: Run the specific command that produced the leaked secret in a local, isolated environment. Observe its output carefully.
  • Fix: Look for flags or configuration options within the tool to reduce logging verbosity or to specifically exclude secrets. For example, some AWS CLI commands have --no-cli-pager or --quiet options. You might also need to use set +x in bash scripts to turn off command echoing.
    set +x # Turn off command echoing for subsequent commands
    aws s3 sync my-bucket .
    set -x # Turn command echoing back on if needed later
    
  • Why it works: This prevents the shell itself from echoing commands that might contain secrets, and the tool’s output is less verbose.

Common Cause 4: Incorrect Environment Variable Naming If a variable is named something generic like API_KEY and you think it’s a secret but it’s not configured as one in CircleCI, it might get printed if used in a script. Conversely, if a non-secret variable is accidentally named SECRET_KEY, it might be masked unnecessarily.

  • Diagnosis: Cross-reference all environment variables used in your config.yml and scripts with the list of variables defined in CircleCI’s UI.
  • Fix: Ensure that any value intended to be secret is defined in CircleCI’s UI as an environment variable. Rename variables for clarity if needed.
  • Why it works: Consistent naming and proper configuration ensure CircleCI knows which variables to mask.

Common Cause 5: Dependency Installation Output Some package managers or build tools might print sensitive information during dependency installation if those dependencies themselves require secrets for their own build processes (though this is less common for typical application secrets).

  • Diagnosis: If you see secrets during npm install, pip install, or similar commands, investigate the specific package being installed.
  • Fix: Ensure that any dependencies requiring secrets are configured to use them via environment variables and that their installation scripts do not echo these secrets. This is often a fix within the dependency itself, or you might need to use a custom build process that injects secrets more carefully.
  • Why it works: It prevents the installation process from exposing secrets that might be temporarily used or exposed by the dependency’s build steps.

Common Cause 6: Shell Debugging (set -x) If you’ve enabled shell debugging with set -x in your .circleci/config.yml and not properly masked secrets, the shell will print the commands it’s executing, including the secrets.

  • Diagnosis: Look for set -x in your run steps within config.yml.
  • Fix: Either remove set -x if not needed for debugging, or ensure that any lines using secrets are prefixed with + (which set -x adds) and then potentially masked or handled carefully. The best practice is to avoid set -x in production jobs or to carefully craft commands that don’t expose secrets. A common workaround is to use echo "..." | command or to pipe output to grep -v if you must see intermediate steps.
    - run:
        name: Deploy
        command: |
          set -x # Debugging enabled
          MY_SECRET_VALUE="my-super-secret-token" # This will be printed by set -x
          echo "Deploying with token..."
          # ... actual deployment command using MY_SECRET_VALUE ...
          set +x # Disable debugging
    
    To fix this specific example, you would remove MY_SECRET_VALUE="my-super-secret-token" and ensure MY_SECRET_VALUE is an environment variable injected by CircleCI.

The Next Problem: Incorrectly Masked Variables

After you’ve fixed all the leaks, you might encounter a new issue where CircleCI still shows *** for a variable that you know is not a secret. This usually happens when a non-secret variable is accidentally named to look like a secret (e.g., API_KEY, PASSWORD) and CircleCI’s heuristic masking picks it up.

Want structured learning?

Take the full Circleci course →