Resolving EC2 to Application Load Balancer Connectivity Issues
A client has put his web application behind Application Load Balancer (ALB) on Amazon EC2, but the users could not access the web application through the ALB. The target group was always reporting the instances in the EC2 state to be unhealthy, although they were running and ALB was running.
Bobcares performed an investigation into the application, its intended uses, the health check used and the network configuration to determine the cause of the failure. The investigation found that there is an EC2 security group rule denying access to the ALB, which will not allow the health check to reach the application. The update of the rule restored connectivity and access to applications.
Business Challenge
Despite the EC2 instances being running and the application functioning directly, the web application was not accessible via the ALB. ALB failed health checks caused the ALB to not route client requests and it was necessary to determine and rectify the underlying network configuration.
Technology Used
Blueprint
The application ran on EC2 instances behind an Application Load Balancer. The ALB used HTTP health checks to determine target availability, while EC2 security groups controlled inbound traffic. Correcting the security group relationship allowed ALB health checks to reach the application and restored traffic forwarding.
Key Outcomes
- Restored communication between the ALB and EC2 instances.
- Changed target status from Unhealthy to Healthy.
- Restored application accessibility through the ALB DNS endpoint.
- Enabled the ALB to distribute traffic correctly between the instances.
- Replaced unrestricted public access with traffic permitted from the ALB security group.
Customer Profile
The client operated a web application hosted on Amazon EC2 instances within an Amazon VPC. Application traffic was exposed through an Application Load Balancer using a target group and HTTP health checks. Security groups controlled traffic between the load balancer and backend instances. The client needed the application to be accessible through the ALB while maintaining appropriate network access controls.
The Challenges
- The ALB target group continuously reported the EC2 instances as unhealthy.
- Users could not access the application through the ALB DNS endpoint.
- The application worked when accessed directly through the EC2 instances.
- The EC2 security group did not allow inbound HTTP traffic from the ALB security group.
- Failed ALB health checks prevented client traffic from reaching the backend instances.
Why Bobcares?
- Investigated the application and infrastructure systematically rather than assuming the ALB or application was faulty.
- Verified the web server and application directly on the EC2 instances.
- Reviewed target registration, listener rules, health check configuration, protocol, and application port.
- Traced the failure to the security group relationship between the ALB and EC2 instances.
- Applied a targeted network configuration change without altering the application deployment.
What We Delivered
- Verified application availability directly on the EC2 instances.
- Reviewed the ALB target group and health check configuration.
- Inspected the EC2 security group rules.
- Identified the missing inbound access from the ALB security group.
- Updated the EC2 security group to permit inbound HTTP traffic from the ALB security group.
- Revalidated target health checks after the configuration change.
- Confirmed that traffic was distributed correctly between the EC2 instances.
Results
Target Health
- Before: The registered EC2 instances were reported as Unhealthy because ALB health checks could not reach the application.
- After: The registered targets changed to Healthy after the security group was corrected.
Application Accessibility
- Before: Users could not access the application through the ALB DNS endpoint.
- After: The application became accessible through the ALB.
ALB to EC2 Connectivity
- Before: The EC2 security group blocked inbound HTTP requests originating from the ALB.
- After: The EC2 security group allowed HTTP traffic from the ALB security group.
Traffic Distribution
- Before: The ALB could not forward client requests because the targets were unhealthy.
- After: The ALB successfully forwarded and distributed requests between the EC2 instances.
The Business Impact
- Restored application availability through the intended ALB endpoint.
- Removed the network configuration preventing ALB health checks from succeeding.
- Reduced reliance on unrestricted public access to the EC2 instances.
- Allowed the ALB to handle traffic again between the back-end instances.
- Wrote a more explicit security group policy for the load balancer to application servers.
Ongoing Collaboration
- Check security group rules when deploying to AWS.
- Validate ALB health check configuration and target health.
- Verify application ports against target group configuration.
- Test application access directly and through the load balancer during troubleshooting.
- Review ALB to EC2 connectivity when target health changes unexpectedly.
Conclusion
A security group configuration prevented the ALB from reaching the EC2 instances, causing failed health checks and blocking application traffic. Bobcares identified the missing ALB-to-EC2 access rule and corrected the configuration. The targets became healthy, traffic resumed, and the application became accessible through the load balancer.
