Learn how to resolve the 502 bad gateway AWS Load Balancer error. Our AWS Support team is here to help you with your questions and concerns.

502 bad gateway AWS Load Balancer Error | Resolved

According to our experts. the 502 bad gateway error may be due to either the user target or the load balancer. Today, we are going to take a look at a few troubleshooting tips to resolve this issue. As the error may appear on Application Load Balancer and Classic Load Balancer, we have to consider both scenarios.

502 Bad Gateway On AWS Application Load Balancer

If the HTTP 502 bad gateway errors are occurring on the AWS Application Load Balancer, we can find the source and cause of the error via access logs and metrics from Amazon CloudWatch. However, we have to enable access logging before troubleshooting the error from the Application Load Balancer.

While using access logs, If the elb_status_code is “502” and the target_status_code is “-“, then the load balancer is the source of the HTTP 502 errors. If the elb_status_code and the target_status_code are both “502”, then the target is causing the error.

While using CloudWatch metrics, If data points appear under the HTTPCode_ELB_502_Count metric, then the load balancer is the source of the HTTP 502 errors. If they appear under the HTTP_Target_5XX_Count metric, then the target is the source.

Troubleshooting Tips

After we find the reason behind the error, it is time to choose a troubleshooting method for the error.

• Cause: The load balancer receives a TCP RST from the target when trying to establish a connection.

  Solution:

  • Verify the TargetConnectionErrorCount metric’s data points. This metric shows how many connections between the load balancer and the target are unsuccessful.
  • Ensure that the access log’s request_processing_time, response_processing_time, and target_processing_time fields are all set to -1. This value indicates that as the load balancer requires a successful connection, it is unable to dispatch the request to the target.
• Cause: When trying to connect, the load balancer runs into an unexpected response from the target.

  Solution:

  • Verify if the access log’s request_processing_time, target_processing_time, and response_processing_time fields are all set to value -1.
  • Verify whether traffic is allowed on the target port from the load balancer subnets to the targets.
• Cause: While the load balancer had an open request to the target, the target closed the connection with a TCP RST or TCP FIN.

  Solution:

  This happens when the target’s keep-alive timeout is shorter than the load balancer’s idle timeout value. Therefore, make sure the keep-alive timeout duration is longer than the idle timeout value.
• Cause: Target response contains incorrect or invalid HTTP headers.

  Solution:

  To understand the target response, perform a packet capture on the target for the relevant period of time.
• Cause: When connecting to a target, the load balancer experienced an SSL handshake error or SSL handshake timeout (10 seconds).

  Solution: When the target group is using the HTTPS protocol, use any of the following:

  • Verify that all three fields in the access logs, request processing time, target processing time, and response processing time, are set to the value -1.
  • For the TargetTLSNegotiationErrorCount metric, see if there are any data points.
  • To confirm that an SSL handshake is the cause of the problem, perform a packet capture on the target during the issue timeframe. If so, follow the instructions in the section titled “Perform a packet capture.”
  • Verify whether the ciphers or protocols are incompatible.
• Cause: When a deregistered target handles the request, the deregistration delay period has passed.

  Solution:

  Check the CloudTrail events for an API call made with the DeregisterTargets action during the issue’s timeframe. Deregistering a target to each Application Load Balancer and Classic Load Balancer will cause this error if an API call with DeregisterTargets occurs during this timeframe. Increase the deregistration delay period to fix this problem and ensure that the time-consuming operations succeed without error.

502 Bad Gateway On AWS Classic Load Balancer

When the web server or associated backend application servers running on EC2 instances return a message that the Classic Load Balancer is unable to parse or when the web server or associated backend application servers return their own 502 error message, HTTP 502 (bad gateway) errors can happen.

We can find the cause of the error by enabling Elastic Load Balancing (ELB) access logs on the Classic Load Balancer or by viewing the load balancer CloudWatch metrics.

For ELB access log, elb_status_code and a backend_status_code finds the source of the 502 error.

For CloudWatch metrics, the backend-generated 502 errors, are seen under the HTTPCode_Backend_5XX metric.

Troubleshooting Tips

We can easily fix this error with these steps:

1. First, we have to check if the response body returned by the backend application complies with HTTP specifications as mentioned in the following RFCs:

   
RFC 7230 - HTTP/1.1: Message Syntax and Routing
RFC 7231 - HTTP/1.1: Semantics and Content
RFC 7232 - HTTP/1.1: Conditional Requests
RFC 7233 - HTTP/1.1: Range Requests
RFC 7234 - HTTP/1.1: Caching
RFC 7235 - HTTP/1.1: Authentication

2. Then, we must verify that the response header contains a key and a value in the proper syntax. Also, make sure the HTTP response header includes Content-Length or transfer encoding. Run a command similar to the one below to examine the HTTP responses produced:

   curl -vko /dev/null server_instance_IP
3. Finally, search for duplicate HTTP 502 errors in the ELB access log. The backend status code and elb status code 502 errors signify that one or more of the web server instances are malfunctioning. Additionally, we should check the web server logs of the backend web server instances after determining which web server instances are displaying the issue.

[Need assistance with a different issue? Our team is available 24/7.]

Conclusion

To conclude, our Support Techs gave us a quick look at different ways to fix the 502 bad gateway AWS Load Balancer Error.