You can SSH into a CircleCI job to debug failures live.
Here’s a CircleCI job running a simple Python test suite:
version: 2.1
jobs:
build:
docker:
- image: cimg/python:3.9
steps:
- checkout
- run:
name: Install dependencies
command: pip install -r requirements.txt
- run:
name: Run tests
command: pytest
And here’s the requirements.txt:
pytest
requests
And test_example.py:
import requests
def test_example_site_up():
response = requests.get("https://example.com")
assert response.status_code == 200
def test_broken_link():
response = requests.get("https://httpbin.org/status/404")
assert response.status_code == 200 # This will fail
When you run this, the test_broken_link will fail because example.com returns a 200, but httpbin.org/status/404 returns a 404, and the test asserts it should be 200.
To debug this live, you’ll need to modify the job to include SSH capabilities. This involves adding a debug section to your job configuration.
Here’s the modified job:
version: 2.1
jobs:
build:
docker:
- image: cimg/python:3.9
steps:
- checkout
- run:
name: Install dependencies
command: pip install -r requirements.txt
- run:
name: Run tests
command: pytest
- run:
name: Debug SSH
command: |
echo "SSH debugging enabled. Job will pause here."
echo "To connect, run: ssh -p 2222 root@circleci-ssh.circleci.com"
echo "Your SSH key is: $(cat ~/.ssh/id_rsa.pub)"
sleep infinity # Keep the job running indefinitely
debug:
docker:
- image: cimg/python:3.9
steps:
- checkout
- run:
name: Install dependencies
command: pip install -r requirements.txt
- run:
name: Run tests
command: pytest
- run:
name: Debug SSH
command: |
echo "SSH debugging enabled. Job will pause here."
echo "To connect, run: ssh -p 2222 root@circleci-ssh.circleci.com"
echo "Your SSH key is: $(cat ~/.ssh/id_rsa.pub)"
sleep infinity # Keep the job running indefinitely
The key addition here is the debug section. This tells CircleCI to provision an SSH-accessible environment for this job. Notice that the debug section essentially mirrors the jobs section, ensuring the same environment and steps are available. The sleep infinity command is crucial; it prevents the job from exiting after the pytest command, allowing you time to connect.
Once you push this change and the job starts, you’ll see a new "Debug" tab appear in the CircleCI UI for that specific job run. Click on it. You’ll find instructions, including a command like ssh -p 2222 root@circleci-ssh.circleci.com. This command is specific to your job run. You’ll also see your public SSH key printed out.
You need to add this public key to your CircleCI user settings. Go to your CircleCI profile, find the "SSH Keys" section, and add the public key you see in the job output.
After adding the key, execute the provided SSH command in your local terminal. You should now be connected to the running container for your CircleCI job. You can navigate the filesystem, inspect files, and even manually run commands like pytest to see the failure in real-time.
The sleep infinity command is a common pattern for debugging. It keeps the container alive and accessible via SSH. When you’re done debugging, you can simply cancel the job in the CircleCI UI, which will terminate the container and end the sleep process.
The debug section in CircleCI is distinct from the main jobs section. While they often contain identical steps, the debug block is specifically for configuring the SSH debugging environment. This separation allows you to enable debugging without altering the core logic of your job for regular runs. It’s a way to "opt-in" to a debug-enabled execution.
When you’re done, remember to remove the debug section from your configuration to avoid unnecessary resource usage and to prevent accidental debugging in production builds. The next step in debugging complex CI/CD issues often involves understanding how to use environment variables effectively to pass configuration or secrets between steps and jobs.