Your Docker container died with exit code 137 because the Linux kernel killed it to reclaim memory.
This is the kernel’s Out-Of-Memory (OOM) killer doing its job. When a process, in this case your container, consumes more RAM than the system can afford, the OOM killer steps in to prevent a complete system crash. It picks a victim (your container) and terminates it abruptly to free up resources. The exit code 137 is a specific signal (SIGKILL, code 9) combined with a standard exit code (128), indicating it was killed by a signal.
Here are the most common reasons your container might be hitting this limit:
-
Insufficient Memory Allocation for the Container: You haven’t told Docker how much RAM your container is allowed to use, or the default is too low for its workload.
- Diagnosis: Check your
docker runcommand ordocker-compose.ymlfile for any--memoryormem_limitdirectives. If none exist, the container is trying to use as much memory as the host allows, which can quickly become an issue for memory-intensive applications. - Fix: When running with
docker run, add the--memoryflag:docker run --memory=2g my_image. Fordocker-compose, usedeploy.resources.limits.memory: "2g"within your service definition. - Why it works: This explicitly caps the container’s memory usage, preventing it from exceeding the allocated limit and triggering the OOM killer.
- Diagnosis: Check your
-
Application Memory Leak: The application running inside your container has a bug where it continuously allocates memory without releasing it. Over time, this leak consumes all available memory.
- Diagnosis: Monitor the container’s memory usage over time using
docker stats <container_id>. If you see a steady, upward trend that never plateaus, it’s likely a leak. Use application-specific profiling tools (e.g.,pproffor Go,memory_profilerfor Python) within the container to pinpoint the exact source. - Fix: Fix the application code to properly manage memory. This might involve closing file handles, releasing database connections, optimizing data structures, or identifying and correcting the faulty allocation/deallocation logic.
- Why it works: Eliminating the leak stops the continuous memory growth, allowing the application to operate within its allocated (or system’s) memory boundaries.
- Diagnosis: Monitor the container’s memory usage over time using
-
Under-provisioned Host System: The Docker host itself doesn’t have enough RAM to support the sum of all running containers and the host OS processes.
- Diagnosis: Use
htoporfree -mon the Docker host to check overall system memory usage. If the host is consistently near 100% memory utilization, it’s a strong indicator. - Fix: Add more RAM to the host machine or reduce the number of containers running on it.
- Why it works: Increasing the host’s physical memory or reducing the load ensures there’s enough RAM for all processes, including Docker containers, to operate without triggering the OOM killer.
- Diagnosis: Use
-
Large Data Processing or Unexpected Load: The container is performing a legitimate, but extremely memory-intensive task (e.g., sorting a massive dataset, loading a huge file into memory, or handling a sudden traffic spike) that temporarily exceeds its memory limit.
- Diagnosis: Review application logs for any specific operations that precede the OOM kill. Correlate this with spikes in
docker statsmemory usage. - Fix: Increase the memory limit for the container (
--memoryflag ormem_limitin compose) to accommodate these peak loads. Alternatively, refactor the application to process data in smaller chunks or use disk-based operations instead of in-memory ones. - Why it works: A higher memory limit provides the necessary headroom for these intensive operations. Refactoring reduces the peak memory demand.
- Diagnosis: Review application logs for any specific operations that precede the OOM kill. Correlate this with spikes in
-
Misconfigured Swappiness: Linux uses swap space as an extension of RAM. If the system is too aggressive in moving memory pages to swap (high
vm.swappiness), it can lead to performance issues and, in extreme cases, OOM situations if swap is exhausted or if processes are unwilling to be swapped out.- Diagnosis: Check the current swappiness value with
sysctl vm.swappiness. A value of 100 means the kernel will swap aggressively. - Fix: Temporarily reduce swappiness with
sudo sysctl vm.swappiness=10and make it permanent by editing/etc/sysctl.conf(addvm.swappiness=10and runsudo sysctl -p). - Why it works: Lowering swappiness tells the kernel to prefer keeping processes in RAM rather than swapping them out, which can sometimes prevent OOM kills if the bottleneck is truly RAM and not just a symptom of aggressive swapping. Note: this is less common for container OOM kills unless the host is severely memory constrained.
- Diagnosis: Check the current swappiness value with
-
Docker Daemon Memory Limits: While less common, the Docker daemon itself might be consuming excessive memory, indirectly impacting container performance or the OOM killer’s decision-making process.
- Diagnosis: Monitor the memory usage of the
dockerdprocess on the host. - Fix: Restart the Docker daemon (
sudo systemctl restart docker). If the problem persists, investigate its configuration or potential bugs within the daemon itself. - Why it works: A simple restart can clear up temporary memory bloat in the daemon. Persistent issues might require deeper investigation into Docker’s internal workings or configuration.
- Diagnosis: Monitor the memory usage of the
After fixing the OOM issue, the next problem you’re likely to encounter is a docker exec command failing with a "containerd error: you cannot execute a non-existent container" if the container was stopped abruptly and you tried to interact with it.