Bobcares

Nginx upstream errors – What is it and how to fix it?

by | Nov 8, 2018

Nginx is the all-time favorite webserver and reverse proxy server for high traffic websites.

It stays strong under high traffic, and help achieve high uptime.

But like any other server, Nginx too is susceptible to configuration and connection issues that cause nginx upstream errors.

Result is website downtime and in turn, negative impact on SEO.

502 bad gateway” is one such common nginx upstream error.

At Bobcares, engineers at our 24/7 monitoring service monitor the servers and quickly resolve such nginx upstream errors.

 

What are Nginx Upstream Errors?

With reverse proxy, it accepts a request from the client, forwards it to Upstream server(server that can complete the request), and returns the server’s response to the client.

Nginx upstream errors arise while the proxy server receive an invalid response or no response from the origin server.

 

nginx upstream error - proxy work flow

 

Visitors will see different variations of these errors like:

502 bad gateway
502 bad gateway nginx
502 Proxy error
504 Gateway Timeout
HTTP 504 Gateway Timeout

 

These messages are cryptic.

So, many webmasters look at their error log:

2018/11/01 18:15:02 [error] 1713#0: *1 upstream timed out (110: Connection timed out) while connecting to upstream, client: xx.xx.xxx.xxx, server: , request: "GET / HTTP/1.1", upstream: "http://xx.xx.xx.xx:8082/", host: "xx.xx.xx.xx"

 

You can see that something is messed up but not sure where to start.

That’s where you need an expert to figure it out quickly and fix it permanently.

Today, let’s discuss the various reasons for nginx upstream errors and how our Dedicated Server Administrators fix them.

 

Nginx Upstream Errors – Top 6 reasons and solutions

In our experience handling nginx upstream errors, we’ll see the major causes that we’ve come across.

1. High load on origin server

Heavy load spike cause services to not respond.

Hence, Nginx can’t communicate with the origin server and result in this error.

The most common reasons for load spikes are:

  • Compromised server that sends malware or spam.
  • Heavy website traffic(can be marketing, promotional etc.).
  • Brute force attacks to exploit web applications.
  • Application bugs that cause resource hogging and memory leakage.
How we fix?

Our Hosting Engineers first identify the resource that is being abused. Then we find out which service is abusing that resource.

At this point, we identify the user who owns the script or software for abusing the service.

 

2. Service downtime in origin server

Nginx depends on various services like apache, PHP-FPM, database services, etc. to run applications.

If any of these services crash, Nginx won’t get any data and result in errors.

How we fix?

The reasons for service failure can be traffic spikes, resource outages, DDOS attacks, disk errors and so on.

We’ll identify this reason and fix it.

If a backend service fails or doesn’t respond, we’ll kill all the dead processes and restart the service.

 

3. Firewall blocks a request

Firewall is the keystone of server security. If not configured properly, it can block legitimate requests or services.

By default, firewall block uncommon ports in the server.

If you have a new service(eg: Ruby) enabled in the server and it runs on a custom port, there can be chances that this port is blocked in the firewall.

Hence, Nginx can’t communicate with this service and lead to this error.

How we fix?

To fix it, we look at which port each service runs on using the command netstat.

If we find any service running on custom ports, we edit the firewall configuration to allow these custom ports.

 

4. Network problems

DNS issues, routing problems and ISP problems can lead to Nginx upstream errors.

If recent DNS changes were made, like changing nameservers, hosting servers, etc. it will take some time to propagate globally. The domain may be unroutable during this period.

Also, sometimes ISPs may block access to a particular site.

All these can lead to such errors.

How we fix?

We check the DNS connectivity of the domain using the command:

dig domain.com

Also, we access the domain from third-party proxy servers to identify if this error is specific to the customer.

If any DNS conflicts found, we will quickly correct it. If the issue is at ISP end, then this needs to be fixed at their end.

5. Server software timeouts

Nginx upstream errors can also occur when a webserver takes more time to complete the request.

By that time, the caching server will reaches its timeout values(timeout for the connection between proxy and upstream server).

Slow queries can lead to such problems.

How we fix?

We will fine tune the following Nginx timeout values in the Nginx configuration file.

proxy_connect_timeout 1200s;
proxy_send_timeout 1200s;
proxy_read_timeout 1200s;
fastcgi_send_timeout 1200s;
fastcgi_read_timeout 1200s;

 

Once the timeout values are added, need to reload nginx to save these parameters.

 

6. Application code bugs

If all other checks failed, it can be an error in your web application code.

Sometimes, your application code may be incompatible with the server version prompting this error to show up.

How we fix?

We will diagnose the issue by analyzing the application and web server logs.

Also, we review the software requirements of your application, re-configure the services to match the required version.

 

Conclusion

Nginx upstream errors occur when Nginx runs as a reverse proxy and is unable to connect to the upstream servers. Today, we’ve seen the top 6 reasons for this error and how we fix it.

PREVENT YOUR SERVER FROM CRASHING!

Never again lose customers to poor server speed! Let us help you.

Our server experts will monitor & maintain your server 24/7 so that it remains lightning fast and secure.

SEE SERVER ADMIN PLANS

var google_conversion_label = "owonCMyG5nEQ0aD71QM";

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

Never again lose customers to poor
server speed! Let us help you.

Privacy Preference Center

Necessary

Necessary cookies help make a website usable by enabling basic functions like page navigation and access to secure areas of the website. The website cannot function properly without these cookies.

PHPSESSID - Preserves user session state across page requests.

gdpr[consent_types] - Used to store user consents.

gdpr[allowed_cookies] - Used to store user allowed cookies.

PHPSESSID, gdpr[consent_types], gdpr[allowed_cookies]
PHPSESSID
WHMCSpKDlPzh2chML

Statistics

Statistic cookies help website owners to understand how visitors interact with websites by collecting and reporting information anonymously.

_ga - Preserves user session state across page requests.

_gat - Used by Google Analytics to throttle request rate

_gid - Registers a unique ID that is used to generate statistical data on how you use the website.

smartlookCookie - Used to collect user device and location information of the site visitors to improve the websites User Experience.

_ga, _gat, _gid
_ga, _gat, _gid
smartlookCookie
_clck, _clsk, CLID, ANONCHK, MR, MUID, SM

Marketing

Marketing cookies are used to track visitors across websites. The intention is to display ads that are relevant and engaging for the individual user and thereby more valuable for publishers and third party advertisers.

IDE - Used by Google DoubleClick to register and report the website user's actions after viewing or clicking one of the advertiser's ads with the purpose of measuring the efficacy of an ad and to present targeted ads to the user.

test_cookie - Used to check if the user's browser supports cookies.

1P_JAR - Google cookie. These cookies are used to collect website statistics and track conversion rates.

NID - Registers a unique ID that identifies a returning user's device. The ID is used for serving ads that are most relevant to the user.

DV - Google ad personalisation

_reb2bgeo - The visitor's geographical location

_reb2bloaded - Whether or not the script loaded for the visitor

_reb2bref - The referring URL for the visit

_reb2bsessionID - The visitor's RB2B session ID

_reb2buid - The visitor's RB2B user ID

IDE, test_cookie, 1P_JAR, NID, DV, NID
IDE, test_cookie
1P_JAR, NID, DV
NID
hblid
_reb2bgeo, _reb2bloaded, _reb2bref, _reb2bsessionID, _reb2buid

Security

These are essential site cookies, used by the google reCAPTCHA. These cookies use an unique identifier to verify if a visitor is human or a bot.

SID, APISID, HSID, NID, PREF
SID, APISID, HSID, NID, PREF