Stuck with AWS EC2 instance going to emergency mode on booting? We can help you.

At Bobcares we assist our customers with several AWS queries as part of our AWS Support Services for AWS users, and online service providers.

Today, let us discuss how our Support Techs resolved above error.

AWS EC2 instance going to emergency mode on booting

Basically, while booting Amazon Elastic Compute Cloud (Amazon EC2) Linux instance, the instance goes into emergency mode and the boot process fails. Then, the instance is inaccessible.

Causes for instance going to emergency mode on booting

• A corrupted kernel.
• Auto-mount failures because of incorrect entries in the /etc/fstab.

In order to verify what type of error is occurring, view the instance’s console output.

You might see either of the below possible errors:

• Kernel panic error message in the console output if the kernel is corrupted.
• Dependency failed messages appear in the console output if auto-mount failures occur.

How to resolve it?

Today, let us see the steps followed by our Support Techs to resolve those errors

Kernel panic errors

Kernel panic error messages occur when the grub configuration or initramfs file is corrupted.

Typical error looks as shown below in the console output:

Kernel panic – not syncing: VFS: Unable to mount root fs on unknown-block(8,1)

In order to resolve kernel panic errors:

• Firstly, revert the kernel to a previous, stable kernel.
• After you revert to a previous kernel, reboot the instance. Then, correct the issues on the corrupted kernel.

Dependency failed errors

Auto-mount failures caused by syntax errors in the /etc/fstab file can cause the instance to enter emergency mode.

Also, if the Amazon Elastic Block Store (Amazon EBS) volume listed in the file is detached from the instance, then the instance boot process might enter emergency mode.

If either of these problems occur, then the console output looks as shown below:

[[1;33mDEPEND[0m] Dependency failed for /mnt.
[[1;33mDEPEND[0m] Dependency failed for Local File Systems.
[[1;33mDEPEND[0m]
Dependency failed for Migrate local... structure to the new structure.
[[1;33mDEPEND[0m] Dependency failed for Relabel all filesystems, if necessary.
[[1;33mDEPEND[0m] Dependency failed for Mark the need to relabel after reboot.
[[1;33mDEPEND[0m]
Dependency failed for File System Check on /dev/xvdf.

To prevent the boot sequence from entering emergency mode due to mount failures:

• Add a nofail option in the /etc/fstab file for the secondary partitions.

When the nofail option is present, the boot sequence isn’t interrupted, even if mounting of any volume or partition fails.

• Add 0 as the last column of the /etc/fstab file for the respective mount point.

Adding the 0 column disables the file system check, allowing the instance to successfully boot.

There are three methods to correct the /etc/fstab file.

Method 1: Use the EC2 Serial Console

• If you’ve enabled EC2 Serial Console for Linux, you can use it to troubleshoot supported Nitro-based instance types.
• The serial console helps you troubleshoot boot issues, network configuration, and SSH configuration issues.
• The serial console connects to your instance without the need for a working network connection.
• You can access the serial console using the Amazon EC2 console or the AWS Command Line Interface (AWS CLI).
• Before using the serial console, grant access to it at the account level.
• Then, create AWS Identity and Access Management (IAM) policies granting access to your IAM users.
• Every instance using the serial console must include at least one password-based user.
• If your instance is unreachable and you haven’t configured access to the serial console, then follow the instructions in Method 2.

Method 2: Run the AWSSupport-ExecuteEC2Rescue automation document

• If your instance is configured for AWS Systems Manager, you can run the AWSSupport-ExecuteEC2Rescue automation document to correct boot issues.
• Manual intervention isn’t needed when using this method.

Method 3: Manually edit the file using a rescue instance

• Firstly, open the Amazon EC2 console.
• Then, choose Instances from the navigation pane, and then select the instance that’s in emergency mode.
• Stop the instance.
• Detach the Amazon EBS root volume ( /dev/xvda or /dev/sda1) from the stopped instance.
• Launch a new EC2 instance in same Availability Zone as the impaired instance. The new instance becomes your rescue instance.
• Attach the root volume you detached in step 4 to the rescue instance as a secondary device.
• Next, connect to your rescue instance using SSH.
• Create a mount point directory for the new volume attached to the rescue instance in step 6. In the following example, the mount point directory is /mnt/rescue.

$ sudo mkdir /mnt/rescue

• Mount the volume at the directory you created in step 8.

$ sudo mount /dev/xvdf /mnt/rescue

• After the volume is mounted, run the following command to open the /etc/fstab file.

$ sudo vi /mnt/rescue/etc/fstab

• Edit the entries in /etc/fstab as needed.

The following example output shows three EBS volumes defined with UUIDs, the nofail option added for both secondary volumes, and a 0 as the last column for each entry.

------------------------------------------------------------------------------------------
$ cat /etc/fstab
UUID=e75a1891-3463-448b-8f59-5e3353af90ba / xfs defaults,noatime 1 0
UUID=87b29e4c-a03c-49f3-9503-54f5d6364b58 /mnt/rescue ext4 defaults,noatime,nofail 1 0
UUID=ce917c0c-9e37-4ae9-bb21-f6e5022d5381 /mnt ext4 defaults,noatime,nofail 1 0
------------------------------------------------------------------------------------------

• Save the file, and then run the umount command to unmount the volume.

$ sudo umount /mnt/rescue

• Detach the volume from the temporary instance.
• Attach the volume to original instance, and then start the instance to confirm that it boots successfully.

[Need help with AWS error? We’d be happy to assist]

Conclusion

In short, today we saw how our Support techs resolved instance going to emergency mode on booting issue.