Your application’s health check is failing because it’s trying to connect to the database before the database is fully ready.
Here’s why this happens and how to fix it, covering all the common scenarios:
1. The "Database is Still Starting" Race Condition
This is the most frequent culprit. Docker Compose spins up services in parallel by default. If your app container starts before your database container has finished initializing and accepting connections, your app will fail its health check, or worse, crash if it doesn’t have retry logic.
-
Diagnosis: Look for logs in your application container showing connection refused, timeout errors, or "database is not available" messages. Check your database container logs for any startup errors or messages indicating it’s ready to accept connections.
-
Fix: Use the
depends_ondirective with acondition: service_healthyin yourdocker-compose.yml. This tells Docker Compose to wait for the dependent service to report as healthy before starting the current service.version: '3.8' services: db: image: postgres:14 environment: POSTGRES_DB: mydatabase POSTGRES_USER: myuser POSTGRES_PASSWORD: mypassword healthcheck: test: ["CMD-SHELL", "pg_isready -U myuser"] interval: 5s timeout: 5s retries: 5 start_period: 10s # Give Postgres a bit longer to start up initially app: build: . ports: - "8000:8000" depends_on: db: condition: service_healthy environment: DATABASE_URL: postgresql://myuser:mypassword@db:5432/mydatabase- Why it works: The
healthcheckon thedbservice periodically probes the database.depends_onwithservice_healthyon theappservice will only letappstart once thedb’s health check has passedstart_periodand then successfully passed itstestcommand. Thepg_isreadycommand is a standard PostgreSQL utility that checks if the server is ready to accept connections.start_periodis crucial here; it gives the database container a grace period to begin its startup process before the healthcheck starts failing repeatedly, preventing it from ever becoming healthy.
- Why it works: The
2. Incorrect Database Health Check Configuration
Your healthcheck command for the database might be too aggressive or not specific enough, causing it to fail even when the database is mostly ready.
-
Diagnosis: Manually run the
healthcheckcommand defined in yourdocker-compose.ymlinside the database container. For example, if your healthcheck istest: ["CMD-SHELL", "pg_isready -U myuser"], you’d rundocker-compose exec db sh -c "pg_isready -U myuser"(or the equivalent command for your specific database). Check the exit code (0 for success, non-zero for failure). -
Fix: Adjust the
healthcheckparameters. Increaseinterval,timeout, orretries. For PostgreSQL, ensuring thepg_isreadycommand is correct and uses the right user is key. For other databases, use their specific readiness checks (e.g.,mysqladmin pingfor MySQL). Consider adding astart_periodto give the database more time to initialize before the health checks begin.# Example for MySQL services: db: image: mysql:8.0 environment: MYSQL_DATABASE: mydatabase MYSQL_USER: myuser MYSQL_PASSWORD: mypassword MYSQL_ROOT_PASSWORD: rootpassword healthcheck: test: ["CMD", "mysqladmin" ,"ping", "-h", "localhost", "-u", "myuser", "-pmyuser"] # Note: password on command line is insecure, use env vars or secrets for production interval: 10s timeout: 5s retries: 5 start_period: 30s # Give MySQL more time to initialize- Why it works: A more lenient health check allows the database more time to fully initialize its data files, caches, and background processes.
start_periodis vital for databases that have a lengthy initial setup process.
- Why it works: A more lenient health check allows the database more time to fully initialize its data files, caches, and background processes.
3. Application’s Database Connection String or Credentials are Wrong
Even if the database is up, your application can’t connect if it’s looking in the wrong place or using the wrong keys.
-
Diagnosis: Verify the
DATABASE_URLor equivalent environment variables in yourappservice configuration. Double-check the hostname (should be the service name, e.g.,db), port, username, password, and database name against your database service’s configuration. -
Fix: Correct the environment variables in your
docker-compose.ymlfor theappservice.services: app: build: . ports: - "8000:8000" depends_on: db: condition: service_healthy environment: # Ensure these match your db service's environment variables DATABASE_URL: postgresql://myuser:mypassword@db:5432/mydatabase- Why it works: A correctly formed connection string ensures your application is attempting to connect to the correct network address and with the right authentication details, allowing it to establish a connection once the database is available.
4. Network Issues Within Docker
Sometimes, Docker’s internal networking can be the bottleneck. If your containers are on different networks or there’s a DNS resolution problem, connections will fail.
-
Diagnosis: Ensure both
dbandappservices are on the same Docker network (this is the default behavior fordocker-compose up). Trypingingthe database service name from within the app container:docker-compose exec app ping db. If this fails, there’s a network or DNS issue. -
Fix: While usually handled automatically, explicitly defining networks can sometimes help isolate issues.
version: '3.8' services: db: image: postgres:14 # ... other db config ... networks: - app-network app: build: . # ... other app config ... depends_on: db: condition: service_healthy networks: - app-network networks: app-network: driver: bridge- Why it works: Explicitly defining and assigning services to the same network ensures they can resolve each other’s hostnames and communicate directly over Docker’s virtual network.
5. Database Resource Constraints or Overload
If your database container is starved of CPU or memory, it might be slow to start or respond to health checks.
-
Diagnosis: Monitor CPU and memory usage for your
dbcontainer usingdocker stats. Check the database’s own logs for resource-related errors or slow query warnings during startup. -
Fix: Allocate more resources to the database container.
services: db: image: postgres:14 deploy: resources: limits: cpus: '1' memory: 512M reservations: cpus: '0.5' memory: 256M # ... other db config ...- Why it works: Providing adequate resources ensures the database process can run efficiently, start up promptly, and respond to connection requests and health checks without being throttled by the host system.
6. Application Retry Logic is Missing or Insufficient
While depends_on with service_healthy is the preferred Docker Compose way, your application should also have its own resilience. If your app crashes immediately without retrying, the depends_on condition might not be sufficient if the database takes a very long time to become healthy initially.
-
Diagnosis: Examine your application’s code. Does it implement exponential backoff or retry mechanisms when attempting to connect to the database?
-
Fix: Implement retry logic in your application. Libraries like
tenacity(Python),resilience4j(Java), or built-in mechanisms in ORMs can handle this.# Example using tenacity in Python from tenacity import retry, stop_after_attempt, wait_fixed import psycopg2 @retry(stop=stop_after_attempt(10), wait=wait_fixed(5)) def get_db_connection(): conn = psycopg2.connect( dbname="mydatabase", user="myuser", password="mypassword", host="db", port="5432" ) return conn try: conn = get_db_connection() print("Database connected successfully!") except Exception as e: print(f"Failed to connect to database: {e}")- Why it works: Even if the database isn’t instantly ready when the app process starts, robust retry logic allows the application to periodically attempt connections until the database is fully available, preventing a single startup race condition from causing a permanent failure.
After implementing depends_on with service_healthy and ensuring your database healthcheck is robust, you’ll likely encounter issues with your application’s own internal state management as it starts up.