Select Page

Fixing Nginx Docker error “16440 upstream prematurely closed connection”

Here at Bobcares, we help Website owners, SaaS providers and Hosting companies run highly reliable web services.

One way we achieve that is by having server experts monitor production servers 24/7 and quickly resolving any issue that can affect uptime.

Recently, engineers at our 24×7 monitoring service noticed that a site hosted by our customer in a Docker server was offline.

We quickly logged in to the Docker host system and looked at the log files. This error showed up from the Nginx Gateway Docker container:

16440 upstream prematurely closed connection while reading response header from upstream, client:

[ Not using Docker, and still getting this error? Our Server Experts can fix it for you in a few minutes. Click here for help. ]

 

What does this error mean?

Every site hosted in this Docker server used an Nginx gateway to act as a fast reverse proxy.

So, all visits to any site will first come to the Nginx container, and then to the website’s Docker container as shown here:

Nginx Docker container log : 16440 upstream prematurely closed connection while reading response header from upstream, client:

To understand the reason for a website downtime, it is important to look at both the gateway and website Docker containers.

The part of the error that says “upstream prematurely closed connection” means that the Nginx container was unable to get website content from the back-end website.

So, it is clear that the Website’s Docker container has a faulty web server.

[ Not using Docker, and still getting this error? Our Server Experts can fix it for you in a few minutes. Click here for help. ]

 

How we fixed “16440 upstream prematurely closed connection”

Getting at the root cause

Once we determined that the web server in the website’s Docker container isn’t working, we opened a terminal and looked at its log file.

We saw this error:

[10498834.773328] Memory cgroup out of memory: Kill process 19597 (apache2) score 171 or sacrifice child
[10498834.779985] Killed process 19597 (apache2) total-vm:326776kB, anon-rss:76648kB, file-rss:10400kB, shmem-rss:64kB

It means that the Docker container is killing the web server processes due to excess memory usage.

 

Determining how it happened

How did the website that worked fine till now suddenly start using up more memory?

We looked at the website’s file changes, and found that it was recently upgraded with additional plugins.

Additionally, we saw an increasing trend in the website traffic as well.

So, that means each visit to the site requires executing more scripts (there by taking more memory per connection), and there are a higher number of connections than 1 month back.

 

Solution alternatives

At this point, we had 2 solution alternatives:

  1. Remove the new plugins.
  2. Increase the memory allowance to the Docker container.

Considering that the website’s traffic is on an increasing trend, removing the plugins will only be a stopgap measure.

So, we contacted the website owner (on behalf of our customer, the Docker web host) and recommended to buy additional memory.

 

Resolving the issue

The website owner approved the upgrade and we increased the Docker memory by adjusting the “mem_limit” parameter in the Docker Compose configuration file.

mem_limit: 2500m

Once the container was recreated, the webserver started working fine again, and the site came back online.

 

Remember: There are other reasons for this error

While the issue presented here occurred in a Docker platform, it can happen in any system that runs Nginx web server.

Also, memory limit is one among the many reasons why a Docker container might fail.

So, if you are facing this error, and is finding it difficult to fix this error, we can fix it for you in a few minutes.

Click here to talk to our Server Experts. We are online 24/7.

 

 


SPEED UP YOUR SERVER TODAY!

Never again lose customers to poor page speed! Let us help you.

Contact Us once. Enjoy Peace Of Mind For Ever!

GET EXPERT ASSISTANCE FOR JUST $45/hr

Submit a Comment

Your email address will not be published. Required fields are marked *

Bobcares
Bobcares is a server management company that helps businesses deliver uninterrupted and secure online services. Our engineers manage close to 51,500 servers that include virtualized servers, cloud infrastructure, physical server clusters, and more.
MORE ABOUT BOBCARES

Privacy Preference Center

Necessary

Necessary cookies help make a website usable by enabling basic functions like page navigation and access to secure areas of the website. The website cannot function properly without these cookies.

PHPSESSID - Preserves user session state across page requests.

gdpr[consent_types] - Used to store user consents.

gdpr[allowed_cookies] - Used to store user allowed cookies.

PHPSESSID, gdpr[consent_types], gdpr[allowed_cookies]
PHPSESSID
WHMCSpKDlPzh2chML

Statistics

Statistic cookies help website owners to understand how visitors interact with websites by collecting and reporting information anonymously.

_ga - Preserves user session state across page requests.

_gat - Used by Google Analytics to throttle request rate

_gid - Registers a unique ID that is used to generate statistical data on how you use the website.

smartlookCookie - Used to collect user device and location information of the site visitors to improve the websites User Experience.

_ga, _gat, _gid
_ga, _gat, _gid
smartlookCookie

Marketing

Marketing cookies are used to track visitors across websites. The intention is to display ads that are relevant and engaging for the individual user and thereby more valuable for publishers and third party advertisers.

IDE - Used by Google DoubleClick to register and report the website user's actions after viewing or clicking one of the advertiser's ads with the purpose of measuring the efficacy of an ad and to present targeted ads to the user.

test_cookie - Used to check if the user's browser supports cookies.

1P_JAR - Google cookie. These cookies are used to collect website statistics and track conversion rates.

NID - Registers a unique ID that identifies a returning user's device. The ID is used for serving ads that are most relevant to the user.

DV - Google ad personalisation

IDE, test_cookie, 1P_JAR, NID, DV, NID
IDE, test_cookie
1P_JAR, NID, DV
NID

Close your account?

Your account will be closed and all data will be permanently deleted and cannot be recovered. Are you sure?