Nginx is the all-time favorite webserver and reverse proxy server for high traffic websites.
It stays strong under high traffic, and help achieve high uptime.
But like any other server, Nginx too is susceptible to configuration and connection issues that cause nginx upstream errors.
Result is website downtime and in turn, negative impact on SEO.
“502 bad gateway” is one such common nginx upstream error.
At Bobcares, engineers at our 24/7 monitoring service monitor the servers and quickly resolve such nginx upstream errors.
What are Nginx Upstream Errors?
With reverse proxy, it accepts a request from the client, forwards it to Upstream server(server that can complete the request), and returns the server’s response to the client.
Nginx upstream errors arise while the proxy server receive an invalid response or no response from the origin server.
Visitors will see different variations of these errors like:
502 bad gateway 502 bad gateway nginx 502 Proxy error 504 Gateway Timeout HTTP 504 Gateway Timeout
These messages are cryptic.
So, many webmasters look at their error log:
2018/11/01 18:15:02 [error] 1713#0: *1 upstream timed out (110: Connection timed out) while connecting to upstream, client: xx.xx.xxx.xxx, server: , request: "GET / HTTP/1.1", upstream: "http://xx.xx.xx.xx:8082/", host: "xx.xx.xx.xx"
You can see that something is messed up but not sure where to start.
That’s where you need an expert to figure it out quickly and fix it permanently.
Today, let’s discuss the various reasons for nginx upstream errors and how our Dedicated Server Administrators fix them.
Nginx Upstream Errors – Top 6 reasons and solutions
In our experience handling nginx upstream errors, we’ll see the major causes that we’ve come across.
1. High load on origin server
Heavy load spike cause services to not respond.
Hence, Nginx can’t communicate with the origin server and result in this error.
The most common reasons for load spikes are:
- Compromised server that sends malware or spam.
- Heavy website traffic(can be marketing, promotional etc.).
- Brute force attacks to exploit web applications.
- Application bugs that cause resource hogging and memory leakage.
How we fix?
Our Hosting Engineers first identify the resource that is being abused. Then we find out which service is abusing that resource.
At this point, we identify the user who owns the script or software for abusing the service.
2. Service downtime in origin server
Nginx depends on various services like apache, PHP-FPM, database services, etc. to run applications.
If any of these services crash, Nginx won’t get any data and result in errors.
How we fix?
The reasons for service failure can be traffic spikes, resource outages, DDOS attacks, disk errors and so on.
We’ll identify this reason and fix it.
If a backend service fails or doesn’t respond, we’ll kill all the dead processes and restart the service.
3. Firewall blocks a request
Firewall is the keystone of server security. If not configured properly, it can block legitimate requests or services.
By default, firewall block uncommon ports in the server.
If you have a new service(eg: Ruby) enabled in the server and it runs on a custom port, there can be chances that this port is blocked in the firewall.
Hence, Nginx can’t communicate with this service and lead to this error.
How we fix?
To fix it, we look at which port each service runs on using the command netstat.
If we find any service running on custom ports, we edit the firewall configuration to allow these custom ports.
4. Network problems
DNS issues, routing problems and ISP problems can lead to Nginx upstream errors.
If recent DNS changes were made, like changing nameservers, hosting servers, etc. it will take some time to propagate globally. The domain may be unroutable during this period.
Also, sometimes ISPs may block access to a particular site.
All these can lead to such errors.
How we fix?
We check the DNS connectivity of the domain using the command:
dig domain.com
Also, we access the domain from third-party proxy servers to identify if this error is specific to the customer.
If any DNS conflicts found, we will quickly correct it. If the issue is at ISP end, then this needs to be fixed at their end.
5. Server software timeouts
Nginx upstream errors can also occur when a webserver takes more time to complete the request.
By that time, the caching server will reaches its timeout values(timeout for the connection between proxy and upstream server).
Slow queries can lead to such problems.
How we fix?
We will fine tune the following Nginx timeout values in the Nginx configuration file.
proxy_connect_timeout 1200s;
proxy_send_timeout 1200s;
proxy_read_timeout 1200s;
fastcgi_send_timeout 1200s;
fastcgi_read_timeout 1200s;
Once the timeout values are added, need to reload nginx to save these parameters.
6. Application code bugs
If all other checks failed, it can be an error in your web application code.
Sometimes, your application code may be incompatible with the server version prompting this error to show up.
How we fix?
We will diagnose the issue by analyzing the application and web server logs.
Also, we review the software requirements of your application, re-configure the services to match the required version.
Conclusion
Nginx upstream errors occur when Nginx runs as a reverse proxy and is unable to connect to the upstream servers. Today, we’ve seen the top 6 reasons for this error and how we fix it.
0 Comments