Bobcares

WeSupport

Call Us! 1-800-383-5193
Call Us! 1-800-383-5193
Call Us! 1-800-383-5193

Need Help?

Emergency Response Time custom

Our experts have had an average response time of 11.06 minutes in March 2021 to fix urgent issues.

We will keep your servers stable, secure and fast at all times for one fixed price.

AWS EC2 kernel panic after system upgrade – How to resolve

by | Jun 29, 2021

Stuck with AWS EC2 kernel panic after system upgrade? We can help you.

At Bobcares we assist our customers with several AWS queries as part of our AWS Support Services for AWS users, and online service providers.

Today, let us discuss how our Support Techs resolved above error.

AWS EC2 kernel panic after system upgrade

The instance fails to boot after a kernel or system upgrade or after a system reboot on Amazon Elastic Compute Cloud (Amazon EC2) instance.

Typical error looks as shown below:

VFS: Cannot open root device XXX or unknown-block(0,0)
Please append a correct “root=” boot option; here are the available partitions:
Kernel panic – not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

 

What causes AWS EC2 kernel panic after system upgrade?

Following are some of the causes:

  • The initramfs or initrd image is missing from the newly updated kernel configuration in /boot/grub/grub.conf.
  • The kernel or system packages weren’t fully installed during the upgrade process due to insufficient space.
  • Third-party modules are missing from the initrd or initramfs image. For example, NVMe, LVM, or RAID modules.

 

How to resolve it?

Let us see the methods used by our Support Techs to resolve the errors.
 

The initramfs or initrd image is missing from the /boot/grub/grub.conf or /boot directory

 

Method 1: Use the EC2 Serial Console

 

  • If you’ve enabled EC2 Serial Console for Linux, you can use it to troubleshoot supported Nitro-based instance types.
  • The serial console helps you troubleshoot boot issues, network configuration, and SSH configuration issues.
  • The serial console connects to your instance without the need for a working network connection.
  • You can access the serial console using the Amazon EC2 console or the AWS Command Line Interface (AWS CLI).
  • Before using the serial console, grant access to it at the account level.
  • Then, create AWS Identity and Access Management (IAM) policies granting access to your IAM users.
  • Every instance using the serial console must include at least one password-based user.
  • If your instance is unreachable and you haven’t configured access to the serial console, then follow the instructions in Method 2.

 

Method 2: Use a rescue instance

 

  • Firstly, open the Amazon EC2 console.
  • Then, choose Instances from the navigation pane, and then select the impaired instance.
  • Choose Actions, Instance State, Stop instance.
  • In the Storage tab, select the Root device, and then select the Volume ID.
  • Choose Actions, Detach Volume (/dev/sda1 or /dev/xvda), and then choose Yes, Detach.
  • Then, verify that the State is Available.
  • Launch a new EC2 instance in the same Availability Zone and with the same operating system as the original instance. This new instance is your rescue instance.
  • After the rescue instance launches, choose Volumes from the navigation pane, and then select the detached root volume of the original instance.
  • Choose Actions, Attach Volume.
  • Select the rescue instance ID (1-xxxx) and then enter /dev/xvdf.
  • Run the following command to verify that the root volume of the impaired instance attached to the rescue instance successfully:
$ lsblk
  • Then, create a mount directory and then mount under /mnt.
$ mount /dev/xvdf1 /mnt
  • Invoke a chroot environment by running the following command:
$ for i in dev proc sys run; do mount -o bind /$i /mnt/$i; done
  • Run the chroot command on the mounted /mnt file system:
$ chroot /mnt
  • Run the following commands based on your operating system.

RPM-based operating systems:

$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg
$ sudo dracut -f -vvvvv

Debian-based operating systems:

$ sudo update-grub && sudo update-grub2
$ sudo update-initramfs -u -vvvvv
  • Verify that the initrd or initramfs image is present in the /boot directory and that the image has a corresponding kernel image.
  • After verifying that the latest kernel has a corresponding initrd or initramfs image, run the following commands to exit and cleanup the chroot environment:
$ exit
$ for i in dev proc sys run; do sudo umount /mnt/$i; done
  • Detach the root volume from the rescue instance and attach the volume to the original instance.
  • Finally, start the original instance.

 

Third-party modules are missing from the initrd or initramfs image

 

To determine if the kernel panic error is caused by a missing third-party module or modules, do the following:

1.Use Method 1: Use the EC2 Serial Console in the preceding section to create a chroot environment in root volume of the non-booting instance.

Or follow steps 1-14 in Method 2: Use a rescue instance in the preceding section to create a chroot environment in the root volume of the non-booting instance.

2.Use one of the following three options to determine which module or modules are missing from the initramfs or initrd image:

Option 1: Run the dracut -f -v command in the /boot directory to determine if rebuilding the initrd or initramfs image fails.

Also use the dracut -f -v command to list which module or modules is missing.

Option 2: Run the lsinitrd initramfs-4.14.138-114.102.amzn2.x86_64.img | less command to view the contents of the initrd or initramfs file.

Replace initramfs-4.14.138-114.102.amzn2.x86_64.img with the name of your image.

Option 3: Inspect the /usr/lib/modules directory.

3.If you find a missing module, you can try to add it back to the kernel.

[Need help with AWS error? We’d be happy to assist]

Conclusion

In short, today we saw how our Support techs resolved instance going to emergency mode on booting issue.

Are you using Docker based apps?

There are proven ways to get even more out of your Docker containers! Let us help you.

Spend your time in growing business and we will take care of Docker Infrastructure for you.

GET STARTED

var google_conversion_label = "owonCMyG5nEQ0aD71QM";

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

AWS Support

Use your time to build great apps. Leave your servers to us.

Managing a server is time consuming. Whether you are an expert or a newbie, that is time you could use to focus on your product or service. Leave your server management to us, and use that time to focus on the growth and success of your business.

TALK TO USOr click here to learn more.
AWS Support

Use your time to build great apps. Leave your servers to us.

Managing a server is time consuming. Whether you are an expert or a newbie, that is time you could use to focus on your product or service. Leave your server management to us, and use that time to focus on the growth and success of your business.

TALK TO USOr click here to learn more.

Categories:

Tags:

Privacy Preference Center

Necessary

Necessary cookies help make a website usable by enabling basic functions like page navigation and access to secure areas of the website. The website cannot function properly without these cookies.

PHPSESSID - Preserves user session state across page requests.

gdpr[consent_types] - Used to store user consents.

gdpr[allowed_cookies] - Used to store user allowed cookies.

PHPSESSID, gdpr[consent_types], gdpr[allowed_cookies]
PHPSESSID
WHMCSpKDlPzh2chML

Statistics

Statistic cookies help website owners to understand how visitors interact with websites by collecting and reporting information anonymously.

_ga - Preserves user session state across page requests.

_gat - Used by Google Analytics to throttle request rate

_gid - Registers a unique ID that is used to generate statistical data on how you use the website.

smartlookCookie - Used to collect user device and location information of the site visitors to improve the websites User Experience.

_ga, _gat, _gid
_ga, _gat, _gid
smartlookCookie

Marketing

Marketing cookies are used to track visitors across websites. The intention is to display ads that are relevant and engaging for the individual user and thereby more valuable for publishers and third party advertisers.

IDE - Used by Google DoubleClick to register and report the website user's actions after viewing or clicking one of the advertiser's ads with the purpose of measuring the efficacy of an ad and to present targeted ads to the user.

test_cookie - Used to check if the user's browser supports cookies.

1P_JAR - Google cookie. These cookies are used to collect website statistics and track conversion rates.

NID - Registers a unique ID that identifies a returning user's device. The ID is used for serving ads that are most relevant to the user.

DV - Google ad personalisation

IDE, test_cookie, 1P_JAR, NID, DV, NID
IDE, test_cookie
1P_JAR, NID, DV
NID
hblid

Security

These are essential site cookies, used by the google reCAPTCHA. These cookies use an unique identifier to verify if a visitor is human or a bot.

SID, APISID, HSID, NID, PREF
SID, APISID, HSID, NID, PREF