Docker systems can be used for a wide range of applications, from setting up development environments to hosting web instances.

Live Docker containers that crash often can end up ruining their purpose. As a result, a major concern faced by Docker providers is ensuring the container uptime.

Containers crash due to many reasons, the main issue being lack of enough memory. During a crash, container will show an exit code that explains the reason for its crash.

Today we’ll see what causes Docker ‘Exited (137)’ crash message and how to fix it.

What causes error 137 in Docker

Docker crashes are often denoted by the message ‘Exited’ in the container ‘STATUS’ when listing the container using ‘docker ps -a’ command.

Error 137 in Docker denotes that the container was ‘KILL’ed by ‘oom-killer’ (Out of Memory). This happens when there isn’t enough memory in the container for running the process.

‘OOM killer’ is a proactive process that jumps in to save the system when its memory level goes too low, by killing the resource-abusive processes to free up memory for the system.

Here is a snippet that shows that the Mysql container Exited with error 137:

Docker container exited with error 137

When the MySQL process running in the container exceeded its memory usage, OOM-killer killed the container and it exited with code 137.

[ Running a Docker infrastructure doesn’t have to be hard, or costly. Get world class Docker management services at affordable pricing. ]

How to debug error 137 in Docker

Each Docker container has a log file associated with it. These log files store all the relevant information and updates related to that container.

Examining the container log file is vital in troubleshooting its crash. Details of the crash can be identified by checking the container logs using ‘docker logs’ command.

For the MySQL container that crashed, the log files showed the following information:

Docker error 137 – Out of memory

The logs in this case clearly shows that the MySQL container was killed due to the mysqld process taking up too much memory.

Reasons for ‘Out of memory’ error 137 in Docker

In Docker architecture, the containers are hosted in a single physical machine. Error 137 in Docker usually happens due to 2 main out-of-memory reasons:

1. Docker container has run out of memory

By default, Docker containers use the available memory in the host machine. To prevent a single container from abusing the host resources, we set memory limits per container.

But if the memory usage by the processes in the container exceeds this memory limit set for the container, the OOM-killer would kill the application and the container crashes.

An application can use up too much memory due to improper configuration, service not optimized, high traffic or usage or resource abuse by users.

Docker system architecture

2. Docker host has no free memory

The memory limit that can be allotted to the Docker containers is limited by the total available memory in the host machine which hosts them.

Many often, when usage and traffic increases, the available free memory may be insufficient for all the containers. As a result, containers may crash.

[ Never let your business be affected by crashing containers! Our Docker experts take care of your infrastructure and promptly resolves all container issues. ]

How to resolve error 137 in Docker

When a Docker container exits with OOM error, it shows that there is a lack of memory. But, the first resort should not be to increase the RAM in the host machine.

Improperly configured services, abusive processes or peak traffic can lead to memory shortage. So the first option is to identify the cause of this memory usage.

After identifying the cause, the following corrective actions can be done in the Docker system to avoid further such OOM crashes.

1. Optimize the services

Unoptimized applications could take up more than optimal memory. For instance, an improperly configured MySQL service can quickly consume the entire host memory.

So, the first step is to monitor the application running in the container and to optimize the service. This can be done by editing the configuration file or recompiling the service.

2. Mount config files from outside

It is always advisable to mount the config files of services from outside the Docker container. This will allow to edit them easily without recompiling the Docker image.

For instance, in MySQL docker image, “/etc/mysql/conf.d” can be mounted as a volume. Any configuration changes for MySQL service can be done without affecting that image.

3. Monitor the container usage

Monitoring the container’s memory usage to detect abusive users, resource-depleted processes, traffic spikes, etc. is very vital in Docker system management.

Depending on the traffic and resource usage of processes, memory limits for the containers can be changed to suit their business purpose better.

4. Add more RAM to the host machine

After optimizing the services and setting memory limits, if the containers are running at their maximum memory limits, then we should add more RAM.

Adding more RAM and ensuring enough swap memory in the host machine would help the containers to utilize that memory whenever there is a memory crunch.

In short..

Today we saw how to fix error 137 in Docker using a systematic debugging method. But there may be scenarios where the error code may not be displayed, especially when the container is launched from some shell script.

At Bobcares, examining Docker logs, optimizing applications, limiting resource usage for containers, monitoring the traffic, etc. are routinely done to avoid crashes.

Before deploying Docker containers for live hosting or development environment setup, we perform stress test by simulating the estimated peak traffic.

This helps us to minimize crashes in the live server. If you’d like to know how to manage your Docker resources efficiently for your business purpose, we’d be happy to talk to you.