Bobcares

EC2 Auto Scaling terminate an unhealthy instance – How to resolve

by | Jul 10, 2021

Stuck with EC2 Auto Scaling terminate an unhealthy instance? We can help you.

Amazon EC2 Auto Scaling is able to automatically determine the health status of an instance using Amazon EC2 status checks and Elastic Load Balancing (ELB) health checks.

At Bobcares we assist our customers with several AWS queries as part of our AWS Support Services for AWS users, and online service providers.

Today, let us discuss how our Support Techs resolve this Auto Scaling issue.

 

EC2 Auto Scaling terminate an unhealthy instance

All scaling actions of an Amazon EC2 Auto Scaling group are logged in Activity History on the Amazon EC2 console.

Sometimes you can’t determine why Amazon EC2 Auto Scaling didn’t terminate an unhealthy instance from Activity History alone.

Today, let us see how our Support Techs terminate that instance, within the Amazon EC2 console.

Firstly, note the state of the instance in Amazon EC2 Auto Scaling:

  • Sign in to the Amazon EC2 console. In the navigation pane under Auto Scaling, choose Auto Scaling Groups, and then select the instance’s group.
  • Then, choose the Instances view and note the health state of the instance.

Health Check Grace Period

Amazon EC2 Auto Scaling doesn’t terminate an instance that came into service based on EC2 status checks and ELB health checks until the health check grace period expires.

To find the grace period length:

  • On the Amazon EC2 console navigation pane, under Auto Scaling, choose Auto Scaling Groups, and then select the instance’s group.
  • Then, choose the Details view and note the Health Check Grace Period length.

Suspended Processes

The suspension of processes such as HealthCheck, ReplaceUnhealthy, or Terminate affects Amazon EC2 Auto Scaling’s capability to detect, replace, or terminate unhealthy instances:

  • Under Auto Scaling in the navigation pane of the Amazon EC2 console navigation pane, choose Auto Scaling Groups, and then select the instance’s group.
  • Then, choose the Details view.
  • Next, choose Edit and remove any of the following processes from Suspended Processes if they are present: HealthCheck, ReplaceUnhealthy, or Terminate.
  • Finally, choose Save to resume the processes.

 

Instance State in Amazon EC2 Console

Amazon EC2 Auto Scaling does not immediately terminate instances with an Impaired status.

Instead, Amazon EC2 Auto Scaling waits a few minutes for the instance to recover.

To check if an instance is impaired:

  • On the Amazon EC2 console navigation pane, under Instances, choose Instances, and then select the instance.
  • Choose the Status Checks view and note if the instance’s status is Impaired.

Amazon EC2 Auto Scaling might also delay or not terminate instances that fail to report data for status checks.

This usually happens when there is insufficient data for the status check metrics in Amazon CloudWatch.

To terminate these instances manually:

  • On the Amazon EC2 console navigation pane, under Instances, choose Instances, and then select the instance.
  • Choose the Monitoring view and note the status of the instance.
  • If the status is Insufficient Data, select the instance again, choose the Actions menu, choose Instance State, and then choose Terminate.

 

Instance State in Auto Scaling Group

Amazon EC2 Auto Scaling does not perform health checks on instances in the Standby state.

To set Standby instances back to the InService state:

  • On the Amazon EC2 console navigation pane, under Auto Scaling Groups, select the instance’s group, and then choose the Instances view.
  • Choose the filter menu Any Lifecycle State, and then select Standby.
  • To resume health checks, open the context (right-click) menu for an instance, and then choose Set to InService, which exits the Standby state.

Amazon EC2 Auto Scaling waits to terminate an instance if it is waiting for a lifecycle hook to complete.

To find the lifecycle status and complete the lifecycle hook:

  • On the Amazon EC2 console navigation pane, under Auto Scaling, choose Auto Scaling Groups, and then select the instance’s group.
  • Choose the Instances view and note the Lifecycle status for the instance.
  • If the status is terminating:wait, you can check the heartbeat timeout and then run completing-lifecycle-action to complete the lifecycle hook.

If Amazon EC2 Auto Scaling is waiting for an ELB connection draining period to complete, it waits to terminate the instance:

  • On the Amazon EC2 console navigation pane, under Auto Scaling, choose Auto Scaling Groups, and then select the instance’s group.
  • Choose the Instances view and confirm that the instance’s Lifecycle is terminating.
  • Choose the Activity History view.
  • For Filter, select Waiting for ELB connection draining to confirm if the group is waiting to terminate the instance.

 

ELB Health Checks

ELB settings can affect health checks and instance replacements.

Please note the instance’s status in on the ELB console:

  • On the Amazon EC2 console navigation pane, under Load Balancing, choose Load Balancers, and then select the load balancer to which the instance is registered.
  • Choose the Instances view and note the instance’s status and description.

Amazon EC2 Auto Scaling doesn’t use the results of ELB health checks to determinate an instance’s health status when the group’s health check configuration is set to EC2.

As a result, Amazon EC2 Auto Scaling doesn’t terminate instances that fail ELB health checks.

If an instance’s status is OutofService on the ELB console, but the instance’s status is Healthy on the Amazon EC2 Auto Scaling console, confirm that the health check type is set to ELB:

  • On the Amazon EC2 console navigation pane, under Auto Scaling, choose Auto Scaling Groups, and then select the instance’s group.
  • Choose the Details view and note the Health Check Type.
  • Then, choose Edit and select ELB for Health Check Type, and then choose Save.

If the group’s health check type is already ELB and the instance’s status on the ELB console is OutofService.

Use the status description that you noted earlier to determine further steps:

  • Firstly, instance registration is still in progress: Wait for load balancer to complete instance registration and for the instance to enter the InService state.
  • Secondly, instance is in the Amazon EC2 Availability Zone for which LoadBalancer is not configured to route traffic to: Edit the subnets of the Auto Scaling group or load balancer to be sure they are same as the instance’s subnets.
  • Finally, instance hasn’t passed the configured HealthyThreshold number of health checks consecutively: Wait for ELB to complete health checks and the instance to enter the InService state.

 

[Need help with EC2 issues? We’d be happy to assist]

Conclusion

In short, today we saw how our Support techs resolved EC2 Auto Scaling terminate an unhealthy instance issue.

Are you using Docker based apps?

There are proven ways to get even more out of your Docker containers! Let us help you.

Spend your time in growing business and we will take care of Docker Infrastructure for you.

GET STARTED

var google_conversion_label = "owonCMyG5nEQ0aD71QM";

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

Never again lose customers to poor
server speed! Let us help you.

Privacy Preference Center

Necessary

Necessary cookies help make a website usable by enabling basic functions like page navigation and access to secure areas of the website. The website cannot function properly without these cookies.

PHPSESSID - Preserves user session state across page requests.

gdpr[consent_types] - Used to store user consents.

gdpr[allowed_cookies] - Used to store user allowed cookies.

PHPSESSID, gdpr[consent_types], gdpr[allowed_cookies]
PHPSESSID
WHMCSpKDlPzh2chML

Statistics

Statistic cookies help website owners to understand how visitors interact with websites by collecting and reporting information anonymously.

_ga - Preserves user session state across page requests.

_gat - Used by Google Analytics to throttle request rate

_gid - Registers a unique ID that is used to generate statistical data on how you use the website.

smartlookCookie - Used to collect user device and location information of the site visitors to improve the websites User Experience.

_ga, _gat, _gid
_ga, _gat, _gid
smartlookCookie
_clck, _clsk, CLID, ANONCHK, MR, MUID, SM

Marketing

Marketing cookies are used to track visitors across websites. The intention is to display ads that are relevant and engaging for the individual user and thereby more valuable for publishers and third party advertisers.

IDE - Used by Google DoubleClick to register and report the website user's actions after viewing or clicking one of the advertiser's ads with the purpose of measuring the efficacy of an ad and to present targeted ads to the user.

test_cookie - Used to check if the user's browser supports cookies.

1P_JAR - Google cookie. These cookies are used to collect website statistics and track conversion rates.

NID - Registers a unique ID that identifies a returning user's device. The ID is used for serving ads that are most relevant to the user.

DV - Google ad personalisation

_reb2bgeo - The visitor's geographical location

_reb2bloaded - Whether or not the script loaded for the visitor

_reb2bref - The referring URL for the visit

_reb2bsessionID - The visitor's RB2B session ID

_reb2buid - The visitor's RB2B user ID

IDE, test_cookie, 1P_JAR, NID, DV, NID
IDE, test_cookie
1P_JAR, NID, DV
NID
hblid
_reb2bgeo, _reb2bloaded, _reb2bref, _reb2bsessionID, _reb2buid

Security

These are essential site cookies, used by the google reCAPTCHA. These cookies use an unique identifier to verify if a visitor is human or a bot.

SID, APISID, HSID, NID, PREF
SID, APISID, HSID, NID, PREF