The Docker daemon itself is failing to properly forward signals from the host to the primary process (PID 1) inside your container, often due to an incorrect signal mask or signal handling implementation within the container’s entrypoint.
Here are the common causes and their fixes:
-
Incorrect Signal Handling in
ENTRYPOINTScript:- Diagnosis: Inspect your
ENTRYPOINTscript (often ashellscript). Look for how it traps signals likeSIGTERMorSIGINT. If it doesn’t explicitly trap and handle these signals, or if it exits without forwarding them to the child process, this is the likely culprit. - Fix: Modify your
ENTRYPOINTscript to properly trap and forward signals. For a shell script, this typically involves usingtrapcommands.
This script traps#!/bin/bash # Trap SIGTERM and SIGINT trap 'echo "Caught signal, forwarding to process..."; kill -TERM $MAIN_PID; wait $MAIN_PID' SIGTERM SIGINT # Start your main application in the background your_application & MAIN_PID=$! # Wait for the main process to exit wait $MAIN_PIDSIGTERMandSIGINT, prints a message, then sends the same signal to the backgroundedyour_applicationusing its PID ($MAIN_PID), and finally waits for it to terminate. - Why it works: The
trapcommand in Bash allows you to define actions to be taken when specific signals are received. By forwarding the signal to the main process, you ensure it receives the termination request and can shut down gracefully.
- Diagnosis: Inspect your
-
Using
CMDfor the Primary Process Instead ofENTRYPOINT:- Diagnosis: If your
DockerfileusesCMD ["your_application", "arg1"]andyour_applicationis directly the process that needs to receive signals, Docker’s signal forwarding might not work as expected, especially ifyour_applicationisn’t PID 1. - Fix: Convert your
CMDinto anENTRYPOINTthat executes your application directly or via a shell script.
If you use# Instead of: # CMD ["your_application", "arg1"] # Use ENTRYPOINT with a shell script (as above) or directly: ENTRYPOINT ["your_application", "arg1"]ENTRYPOINT ["your_application"], your application becomes PID 1 and receives signals directly. - Why it works: When an
ENTRYPOINTis used inexecform (the["executable", "param1"]syntax), the executable is run as PID 1. When aCMDis used inexecform, it’s passed as arguments to theENTRYPOINT. If there’s noENTRYPOINT,CMDcan be PID 1, butENTRYPOINTis the more explicit and reliable way to ensure your primary process is PID 1.
- Diagnosis: If your
-
ENTRYPOINTis a Shell and Not Executing inexecForm:- Diagnosis: If your
ENTRYPOINTis a shell script specified using theshellform, likeENTRYPOINT ./my_script.sh, the shell itself is PID 1. If the shell doesn’t correctlyexecthe child process or forward signals, the child process might not receive them. - Fix: Ensure your
ENTRYPOINTscript usesexecto replace the shell process with your application.#!/bin/bash # ... signal trapping logic ... # Replace the shell process with your application exec your_application arg1 - Why it works: The
execcommand replaces the current shell process with the new program (your_application). This makesyour_applicationbecome PID 1 inside the container, and it will receive signals directly from the Docker daemon.
- Diagnosis: If your
-
Process is Not PID 1 and Signal Forwarding is Broken:
- Diagnosis: If your container’s primary process (the one you expect to shut down gracefully) is not PID 1, it’s likely a child process of an
initsystem, a shell, or a wrapper script. If this parent process doesn’t correctly forward signals, the child won’t receive them. - Fix: Use a minimal
initsystem liketiniordumb-initas yourENTRYPOINT.
Or, ifFROM ubuntu:latest RUN apt-get update && apt-get install -y tini ENTRYPOINT ["/usr/bin/tini", "--"] CMD ["your_application", "arg1"]your_applicationis theENTRYPOINT:
EnsureFROM ubuntu:latest RUN apt-get update && apt-get install -y tini ENTRYPOINT ["/usr/bin/tini", "--", "your_application", "arg1"]tiniis installed and configured as theENTRYPOINTbefore your actual application. - Why it works:
tini(anddumb-init) is a lightweight init system that correctly handles signals and zombie processes. It becomes PID 1, starts your application as a child, and ensures signals are forwarded correctly to your application.
- Diagnosis: If your container’s primary process (the one you expect to shut down gracefully) is not PID 1, it’s likely a child process of an
-
Application’s Signal Mask Prevents Receiving Signals:
- Diagnosis: Some applications, especially those written in languages like Go, might have their signal masks set in a way that ignores certain signals (e.g.,
SIGTERM) by default, unless explicitly handled. - Fix: This is application-specific. For Go, you’d typically add code to your
mainfunction to handle signals:import ( "os" "os/signal" "syscall" ) func main() { // ... your application logic ... // Set up channel to listen for signals sigChan := make(chan os.Signal, 1) signal.Notify(sigChan, syscall.SIGTERM, syscall.SIGINT) // Block until a signal is received sig := <-sigChan fmt.Printf("Received signal: %v, shutting down.\n", sig) // ... perform graceful shutdown ... } - Why it works: By explicitly registering a channel to receive
SIGTERMandSIGINT, you override the default signal mask and ensure your application is aware of and can respond to these termination signals.
- Diagnosis: Some applications, especially those written in languages like Go, might have their signal masks set in a way that ignores certain signals (e.g.,
-
Docker Daemon Configuration (Less Common):
- Diagnosis: While rare, it’s possible that the Docker daemon’s own signal handling is misconfigured or there’s an issue with the host’s signal propagation. Check Docker daemon logs for any related errors.
- Fix: Restart the Docker daemon. On most Linux systems:
sudo systemctl restart docker. Ensure your Docker version is up-to-date. - Why it works: Restarting the daemon can clear transient issues with its internal state or signal handling mechanisms.
The next error you’ll likely hit after fixing PID 1 signal handling is related to graceful shutdown of sub-processes or application-specific cleanup routines that were previously bypassed.