Need help?

Our experts have had an average response time of 11.7 minutes in August 2021 to fix urgent issues.

We will keep your servers stable, secure, and fast at all times for one fixed price.

AWS EC2 instance going to emergency mode on booting

by | Jun 29, 2021

Stuck with AWS EC2 instance going to emergency mode on booting? We can help you.

At Bobcares we assist our customers with several AWS queries as part of our AWS Support Services for AWS users, and online service providers.

Today, let us discuss how our Support Techs resolved above error.

 

AWS EC2 instance going to emergency mode on booting

Basically, while booting Amazon Elastic Compute Cloud (Amazon EC2) Linux instance, the instance goes into emergency mode and the boot process fails. Then, the instance is inaccessible.

Causes for instance going to emergency mode on booting

  • A corrupted kernel.
  • Auto-mount failures because of incorrect entries in the /etc/fstab.

In order to verify what type of error is occurring, view the instance’s console output.

You might see either of the below possible errors:

  • Kernel panic error message in the console output if the kernel is corrupted.
  • Dependency failed messages appear in the console output if auto-mount failures occur.

 

How to resolve it?

Today, let us see the steps followed by our Support Techs to resolve those errors

Kernel panic errors

Kernel panic error messages occur when the grub configuration or initramfs file is corrupted.

Typical error looks as shown below in the console output:

Kernel panic – not syncing: VFS: Unable to mount root fs on unknown-block(8,1)

In order to resolve kernel panic errors:

  • Firstly, revert the kernel to a previous, stable kernel.
  • After you revert to a previous kernel, reboot the instance. Then, correct the issues on the corrupted kernel.

 

Dependency failed errors

Auto-mount failures caused by syntax errors in the /etc/fstab file can cause the instance to enter emergency mode.

Also, if the Amazon Elastic Block Store (Amazon EBS) volume listed in the file is detached from the instance, then the instance boot process might enter emergency mode.

If either of these problems occur, then the console output looks as shown below:

[[1;33mDEPEND[0m] Dependency failed for /mnt.
[[1;33mDEPEND[0m] Dependency failed for Local File Systems.
[[1;33mDEPEND[0m]
Dependency failed for Migrate local... structure to the new structure.
[[1;33mDEPEND[0m] Dependency failed for Relabel all filesystems, if necessary.
[[1;33mDEPEND[0m] Dependency failed for Mark the need to relabel after reboot.
[[1;33mDEPEND[0m]
Dependency failed for File System Check on /dev/xvdf.

To prevent the boot sequence from entering emergency mode due to mount failures:

  • Add a nofail option in the /etc/fstab file for the secondary partitions.

When the nofail option is present, the boot sequence isn’t interrupted, even if mounting of any volume or partition fails.

  • Add 0 as the last column of the /etc/fstab file for the respective mount point.

Adding the 0 column disables the file system check, allowing the instance to successfully boot.

There are three methods to correct the /etc/fstab file.

Method 1: Use the EC2 Serial Console
  • If you’ve enabled EC2 Serial Console for Linux, you can use it to troubleshoot supported Nitro-based instance types.
  • The serial console helps you troubleshoot boot issues, network configuration, and SSH configuration issues.
  • The serial console connects to your instance without the need for a working network connection.
  • You can access the serial console using the Amazon EC2 console or the AWS Command Line Interface (AWS CLI).
  • Before using the serial console, grant access to it at the account level.
  • Then, create AWS Identity and Access Management (IAM) policies granting access to your IAM users.
  • Every instance using the serial console must include at least one password-based user.
  • If your instance is unreachable and you haven’t configured access to the serial console, then follow the instructions in Method 2.

 

Method 2: Run the AWSSupport-ExecuteEC2Rescue automation document

 

  • If your instance is configured for AWS Systems Manager, you can run the AWSSupport-ExecuteEC2Rescue automation document to correct boot issues.
  • Manual intervention isn’t needed when using this method.

 

Method 3: Manually edit the file using a rescue instance

 

  • Firstly, open the Amazon EC2 console.
  • Then, choose Instances from the navigation pane, and then select the instance that’s in emergency mode.
  • Stop the instance.
  • Detach the Amazon EBS root volume ( /dev/xvda or /dev/sda1) from the stopped instance.
  • Launch a new EC2 instance in same Availability Zone as the impaired instance. The new instance becomes your rescue instance.
  • Attach the root volume you detached in step 4 to the rescue instance as a secondary device.
  • Next, connect to your rescue instance using SSH.
  • Create a mount point directory for the new volume attached to the rescue instance in step 6. In the following example, the mount point directory is /mnt/rescue.
$ sudo mkdir /mnt/rescue
  • Mount the volume at the directory you created in step 8.
$ sudo mount /dev/xvdf /mnt/rescue
  • After the volume is mounted, run the following command to open the /etc/fstab file.
$ sudo vi /mnt/rescue/etc/fstab
  • Edit the entries in /etc/fstab as needed.

The following example output shows three EBS volumes defined with UUIDs, the nofail option added for both secondary volumes, and a 0 as the last column for each entry.

------------------------------------------------------------------------------------------
$ cat /etc/fstab
UUID=e75a1891-3463-448b-8f59-5e3353af90ba / xfs defaults,noatime 1 0
UUID=87b29e4c-a03c-49f3-9503-54f5d6364b58 /mnt/rescue ext4 defaults,noatime,nofail 1 0
UUID=ce917c0c-9e37-4ae9-bb21-f6e5022d5381 /mnt ext4 defaults,noatime,nofail 1 0
------------------------------------------------------------------------------------------
  • Save the file, and then run the umount command to unmount the volume.
$ sudo umount /mnt/rescue
  • Detach the volume from the temporary instance.
  • Attach the volume to original instance, and then start the instance to confirm that it boots successfully.

 

[Need help with AWS error? We’d be happy to assist]

Conclusion

In short, today we saw how our Support techs resolved instance going to emergency mode on booting issue.

Are you using Docker based apps?

There are proven ways to get even more out of your Docker containers! Let us help you.

Spend your time in growing business and we will take care of Docker Infrastructure for you.

GET STARTED

var google_conversion_label = "owonCMyG5nEQ0aD71QM";

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

Privacy Preference Center

Necessary

Necessary cookies help make a website usable by enabling basic functions like page navigation and access to secure areas of the website. The website cannot function properly without these cookies.

PHPSESSID - Preserves user session state across page requests.

gdpr[consent_types] - Used to store user consents.

gdpr[allowed_cookies] - Used to store user allowed cookies.

PHPSESSID, gdpr[consent_types], gdpr[allowed_cookies]
PHPSESSID
WHMCSpKDlPzh2chML

Statistics

Statistic cookies help website owners to understand how visitors interact with websites by collecting and reporting information anonymously.

_ga - Preserves user session state across page requests.

_gat - Used by Google Analytics to throttle request rate

_gid - Registers a unique ID that is used to generate statistical data on how you use the website.

smartlookCookie - Used to collect user device and location information of the site visitors to improve the websites User Experience.

_ga, _gat, _gid
_ga, _gat, _gid
smartlookCookie

Marketing

Marketing cookies are used to track visitors across websites. The intention is to display ads that are relevant and engaging for the individual user and thereby more valuable for publishers and third party advertisers.

IDE - Used by Google DoubleClick to register and report the website user's actions after viewing or clicking one of the advertiser's ads with the purpose of measuring the efficacy of an ad and to present targeted ads to the user.

test_cookie - Used to check if the user's browser supports cookies.

1P_JAR - Google cookie. These cookies are used to collect website statistics and track conversion rates.

NID - Registers a unique ID that identifies a returning user's device. The ID is used for serving ads that are most relevant to the user.

DV - Google ad personalisation

IDE, test_cookie, 1P_JAR, NID, DV, NID
IDE, test_cookie
1P_JAR, NID, DV
NID
hblid

Security

These are essential site cookies, used by the google reCAPTCHA. These cookies use an unique identifier to verify if a visitor is human or a bot.

SID, APISID, HSID, NID, PREF
SID, APISID, HSID, NID, PREF