Looking for tips to resolve HTTP 504 errors in Amazon EKS? Do they occur when you connect your Kubernetes service through Application Load Balancer or Classic Load Balancer in Amazon EKS?

Look no further, Bobcares is here to help! Let’s take a look at what the experts at Bobcares have to say.

How do you resolve HTTP 504 Errors in Amazon EKS?

HTTP 504 errors may occur due to different reasons like:

• Although the load balancer is connected to the target, the latter does not respond within the idle timeout period. The timeout period is 60 seconds by default for the Application Load Balancer and the Classic Load Balancer.
• Failure to establish a connection by the load balancer to backend target within the connection timeout period of 10 seconds.
• The network access control list for the subnet prevents traffic to the load balancer nodes on the ephemeral ports (1024-65535) from the targets.

Solving The Problem On Hand

Make sure the load balancer’s idle timeout has been set correctly to resolve HTTP 504 error

1. Check the Amazon CloudWatch metrics for the Application Load Balancer or Classic Load Balance

In case the latency data points are the same as the current load balancer timeout value, and data points are present in the HTTPCode_ELB_5XX metric, it is apparent that no less than one request has timed out.

2. Next, modify the load balancer’s idle timeout. This allows the HTTP request to be completed within the idle timeout period. Moreover, it pushes the application to respond faster.

In order to modify the idle timeout value for the Classic Load Balancer, you need to change the service definition. It needs to include service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout annotation.

Make sure the backend instances do not have connection errors to resolve HTTP 504 error

In case the backend instance closes a TCP connection before the load balancer reaches its idle timeout value, the load balancer will not be able to fulfill the request.

1. Check the metrics of CloudWatch BackendConnectionErrors for the Classic Load Balancer. In addition, keep an eye on the target group’s TargetConnectionErrorCount for the Application Load Balancer.

2. Next, enable the keep-alive settings on the backend worker pods or node. Then, set the value of the keep-alive timeout higher than the

1.    Take a look at the CloudWatch BackendConnectionErrors metrics for your Classic Load Balancer and the target group’s TargetConnectionErrorCount for your Application Load Balancer.

2.    Check the keep-alive settings on your backend worker node or pods. Then, set the keep-alive timeout to a value greater than the load balancer’s idle timeout.

To make sure the keep-alive timeout is not higher than the idle timeout, take a look at the keep-alive value in the worker node or pods.

For instance, for pods:

For instance, for nodes:

Check the backend target are able to receive traffic from the load balancer along the ephemeral port range

Initially, configure network ACLs and security groups to allow data movement between backend targets and the load balancer. For example, the targets may be instances or IP addresses depending on the type of the load balancer.

Moreover, connect the security group egress rule of the pods and nodes to the security group of the load balancer’s security group. In other words, configure security groups for seamless ephemeral port access.

[If you need assistance with EKS,  AWS Support Services is just a click away!]

Conclusion to resolving HTTP 504 error in Amazon EKS

In conclusion, this easy resolution comes via our Support Techs. Our team is well-versed in solving any challenges offered by AWS.