Stuck with EC2 Auto Scaling terminate an unhealthy instance? We can help you.

Amazon EC2 Auto Scaling is able to automatically determine the health status of an instance using Amazon EC2 status checks and Elastic Load Balancing (ELB) health checks.

At Bobcares we assist our customers with several AWS queries as part of our AWS Support Services for AWS users, and online service providers.

Today, let us discuss how our Support Techs resolve this Auto Scaling issue.

EC2 Auto Scaling terminate an unhealthy instance

All scaling actions of an Amazon EC2 Auto Scaling group are logged in Activity History on the Amazon EC2 console.

Sometimes you can’t determine why Amazon EC2 Auto Scaling didn’t terminate an unhealthy instance from Activity History alone.

Today, let us see how our Support Techs terminate that instance, within the Amazon EC2 console.

Firstly, note the state of the instance in Amazon EC2 Auto Scaling:

• Sign in to the Amazon EC2 console. In the navigation pane under Auto Scaling, choose Auto Scaling Groups, and then select the instance’s group.
• Then, choose the Instances view and note the health state of the instance.

Health Check Grace Period

Amazon EC2 Auto Scaling doesn’t terminate an instance that came into service based on EC2 status checks and ELB health checks until the health check grace period expires.

To find the grace period length:

• On the Amazon EC2 console navigation pane, under Auto Scaling, choose Auto Scaling Groups, and then select the instance’s group.
• Then, choose the Details view and note the Health Check Grace Period length.

Suspended Processes

The suspension of processes such as HealthCheck, ReplaceUnhealthy, or Terminate affects Amazon EC2 Auto Scaling’s capability to detect, replace, or terminate unhealthy instances:

• Under Auto Scaling in the navigation pane of the Amazon EC2 console navigation pane, choose Auto Scaling Groups, and then select the instance’s group.
• Then, choose the Details view.
• Next, choose Edit and remove any of the following processes from Suspended Processes if they are present: HealthCheck, ReplaceUnhealthy, or Terminate.
• Finally, choose Save to resume the processes.

Instance State in Amazon EC2 Console

Amazon EC2 Auto Scaling does not immediately terminate instances with an Impaired status.

Instead, Amazon EC2 Auto Scaling waits a few minutes for the instance to recover.

To check if an instance is impaired:

• On the Amazon EC2 console navigation pane, under Instances, choose Instances, and then select the instance.
• Choose the Status Checks view and note if the instance’s status is Impaired.

Amazon EC2 Auto Scaling might also delay or not terminate instances that fail to report data for status checks.

This usually happens when there is insufficient data for the status check metrics in Amazon CloudWatch.

To terminate these instances manually:

• On the Amazon EC2 console navigation pane, under Instances, choose Instances, and then select the instance.
• Choose the Monitoring view and note the status of the instance.
• If the status is Insufficient Data, select the instance again, choose the Actions menu, choose Instance State, and then choose Terminate.

Instance State in Auto Scaling Group

Amazon EC2 Auto Scaling does not perform health checks on instances in the Standby state.

To set Standby instances back to the InService state:

• On the Amazon EC2 console navigation pane, under Auto Scaling Groups, select the instance’s group, and then choose the Instances view.
• Choose the filter menu Any Lifecycle State, and then select Standby.
• To resume health checks, open the context (right-click) menu for an instance, and then choose Set to InService, which exits the Standby state.

Amazon EC2 Auto Scaling waits to terminate an instance if it is waiting for a lifecycle hook to complete.

To find the lifecycle status and complete the lifecycle hook:

• On the Amazon EC2 console navigation pane, under Auto Scaling, choose Auto Scaling Groups, and then select the instance’s group.
• Choose the Instances view and note the Lifecycle status for the instance.
• If the status is terminating:wait, you can check the heartbeat timeout and then run completing-lifecycle-action to complete the lifecycle hook.

If Amazon EC2 Auto Scaling is waiting for an ELB connection draining period to complete, it waits to terminate the instance:

• On the Amazon EC2 console navigation pane, under Auto Scaling, choose Auto Scaling Groups, and then select the instance’s group.
• Choose the Instances view and confirm that the instance’s Lifecycle is terminating.
• Choose the Activity History view.
• For Filter, select Waiting for ELB connection draining to confirm if the group is waiting to terminate the instance.

ELB Health Checks

ELB settings can affect health checks and instance replacements.

Please note the instance’s status in on the ELB console:

• On the Amazon EC2 console navigation pane, under Load Balancing, choose Load Balancers, and then select the load balancer to which the instance is registered.
• Choose the Instances view and note the instance’s status and description.

Amazon EC2 Auto Scaling doesn’t use the results of ELB health checks to determinate an instance’s health status when the group’s health check configuration is set to EC2.

As a result, Amazon EC2 Auto Scaling doesn’t terminate instances that fail ELB health checks.

If an instance’s status is OutofService on the ELB console, but the instance’s status is Healthy on the Amazon EC2 Auto Scaling console, confirm that the health check type is set to ELB:

• On the Amazon EC2 console navigation pane, under Auto Scaling, choose Auto Scaling Groups, and then select the instance’s group.
• Choose the Details view and note the Health Check Type.
• Then, choose Edit and select ELB for Health Check Type, and then choose Save.

If the group’s health check type is already ELB and the instance’s status on the ELB console is OutofService.

Use the status description that you noted earlier to determine further steps:

• Firstly, instance registration is still in progress: Wait for load balancer to complete instance registration and for the instance to enter the InService state.
• Secondly, instance is in the Amazon EC2 Availability Zone for which LoadBalancer is not configured to route traffic to: Edit the subnets of the Auto Scaling group or load balancer to be sure they are same as the instance’s subnets.
• Finally, instance hasn’t passed the configured HealthyThreshold number of health checks consecutively: Wait for ELB to complete health checks and the instance to enter the InService state.

[Need help with EC2 issues? We’d be happy to assist]

Conclusion

In short, today we saw how our Support techs resolved EC2 Auto Scaling terminate an unhealthy instance issue.