Need help?

Our experts have had an average response time of 11.7 minutes in August 2021 to fix urgent issues.

We will keep your servers stable, secure, and fast at all times for one fixed price.

AWS EC2 revert to a known stable kernel – How to do it

by | Jun 29, 2021

Stuck with AWS EC2 revert to a known stable kernel? We can help you.

At Bobcares we assist our customers with several AWS queries as part of our AWS Support Services for AWS users, and online service providers.

Today, let us discuss how our Support Techs revert kernel.

 

AWS EC2 revert to a known stable kernel

If you performed a kernel update to your EC2 Linux instance but the kernel is now corrupt, then the instance can’t reboot.

You can’t use SSH to connect to the impaired instance.

There are two steps followed by our Support Techs and let us discuss them in detail.

  1. Access the instance’s root volume.
  2. Update the default kernel in the GRUB bootloader.

Access the instance’s root volume

There are two methods to access the root volume:

Method 1: Use the EC2 Serial Console
  • If you’ve enabled EC2 Serial Console for Linux, you can use it to troubleshoot supported Nitro-based instance types.
  • The serial console helps you troubleshoot boot issues, network configuration, and SSH configuration issues.
  • The serial console connects to your instance without the need for a working network connection.
  • You can access the serial console using the Amazon EC2 console or the AWS Command Line Interface (AWS CLI).
  • Before using the serial console, grant access to it at the account level.
  • Then, create AWS Identity and Access Management (IAM) policies granting access to your IAM users.
  • Every instance using the serial console must include at least one password-based user.
  • If your instance is unreachable and you haven’t configured access to the serial console, then follow the instructions in Method 2.

 

Method 2: Use a rescue instance
  • Firstly, create an EBS snapshot of the root volume.
  • Then, open the Amazon EC2 console.
  • Select Instances from the navigation pane, and then choose the impaired instance.
  • Choose Instance State, Stop instance, and then select Stop.
  • In the Storage tab, under Block devices, select the Volume ID for /dev/sda1.
  • Choose Actions, Detach Volume, and then select Yes, Detach. Note the Availability Zone.
  • Launch a rescue EC2 instance in the same Availability Zone.
  • After the rescue instance launches, choose Volumes from the navigation pane, and then choose the detached root volume of the impaired instance.
  • Choose Actions, Attach Volume.
  • Choose the rescue instance ID ( id-xxxxx), and then set an unused device. In this example, /dev/xvdb.
  • Use SSH to connect to the rescue instance.
  • Run the lsblk command to view your available disk devices.
  • Create a mount directory, and then mount the root partition of the mounted volume to this new directory. In the preceding example, /dev/xvdb1 is the root partition of the mounted volume.
    sudo mkdir /mount
    sudo mount /dev/xvdb1 /mount

    You can now access the data of the impaired instance through the mount directory.

 

Update the default kernel in the GRUB bootloader

First and foremost, call the chroot function to change into the mount directory:

sudo chroot /mount

The current corrupt kernel is in position 0 (zero) in the list. The last stable kernel is in position 1.

To replace the corrupt kernel with the stable kernel, you can use one of the following procedures:

GRUB1 (Legacy GRUB) for Red Hat 6 and Amazon Linux

GRUB2 for Ubuntu 14 LTS and 16.04

GRUB2 for RHEL 7.5 and Amazon Linux 2

Finally, GRUB2 for RHEL 8 and CentOS 8

GRUB1 (Legacy GRUB) for Red Hat 6 and Amazon Linux 1

Use the sed command to replace the corrupt kernel with the stable kernel in the /boot/grub/grub.conf file:

sudo sed -i '/^default/ s/0/1/' /boot/grub/grub.conf

 

GRUB2 for Ubuntu 14 LTS and 16.04
  • Firstly, replace the corrupt GRUB_DEFAULT=0 default menu entry with the stable GRUB_DEFAULT=saved value in the /etc/default/grub file:
sed -i 's/GRUB_DEFAULT=0/GRUB_DEFAULT=saved/g' /etc/default/grub
  • Run the update-grub command so that GRUB recognizes the change:
sudo update-grub
  • Run the grub-set-default command so that the stable kernel loads at the next reboot.

 

GRUB2 for RHEL 7.5 and Amazon Linux 2
  • Replace the corrupt GRUB_DEFAULT=0 default menu entry with the stable GRUB_DEFAULT-saved value in the /etc/default/grub file:
sed -i 's/GRUB_DEFAULT=0/GRUB_DEFAULT=saved/g' /etc/default/grub
  • Update GRUB to regenerate the /boot/grub2/grub.cfg file:
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
  • Run the grub2-set-default command so that the stable kernel loads at the next reboot.

 

GRUB2 for RHEL 8 and CentOS 8

GRUB2 in RHEL 8 and CentOS 8 uses blscfg files and entries in /boot/loader for the boot configuration, instead of the previous grub.cfg format.

If the blscfg files are missing from this location or corrupted, grubby doesn’t show any results.

You must regenerate the files to recover functionality.

Therefore, the indexing of the kernels depends on the .conf files located under /boot/loader/entries and on the kernel versions.

Indexing is configured to keep the latest kernel with the lowest index.

  • Run the grubby –default-kernel command to see the current default kernel:
grubby --default-kernel
  • Run the grubby –info=ALL command to see all available kernels and their indexes:
grubby --info=ALL

Note the path of the kernel that you want to set as the default for your instance.

  • Run the grubby –set-default command to change the default kernel of the instance:
grubby --set-default=/boot/vmlinuz-0-rescue-4.18.0-80.4.2.el8_1.x86_64
  • Run the grubby –default-kernel command to verify that the preceding command worked:
grubby --default-kernel

If you’re accessing the instance using the EC2 Serial Console, then the stable kernel now loads and you can reboot the instance.

If you’re using a rescue instance, then follow the below steps:

Unmount volumes, detach the root volume from the rescue instance, and then attach the volume to the impaired instance

  • Exit from chroot, and unmount /dev, /run, /proc, and /sys:
exit
sudo umount /mount/dev
sudo umount /mount/run
sudo umount /mount/proc
sudo umount /mount/sys
sudo umount /mount
  • From the Amazon EC2 console, choose Instances, and then choose the rescue instance.
  • Choose Instance State, Stop instance, and then select Yes, Stop.
  • Detach the root volume id-xxxxx (the volume from the impaired instance) from the rescue instance.
  • Attach the root volume you detached in step 4 to the impaired instance as the root volume (/dev/sda1), and then start the instance.

 

[Need help with AWS issues? We’d be happy to assist]

Conclusion

In short, today we saw how our Support techs reverted AWS EC2 to a known stable kernel.

Are you using Docker based apps?

There are proven ways to get even more out of your Docker containers! Let us help you.

Spend your time in growing business and we will take care of Docker Infrastructure for you.

GET STARTED

var google_conversion_label = "owonCMyG5nEQ0aD71QM";

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

Privacy Preference Center

Necessary

Necessary cookies help make a website usable by enabling basic functions like page navigation and access to secure areas of the website. The website cannot function properly without these cookies.

PHPSESSID - Preserves user session state across page requests.

gdpr[consent_types] - Used to store user consents.

gdpr[allowed_cookies] - Used to store user allowed cookies.

PHPSESSID, gdpr[consent_types], gdpr[allowed_cookies]
PHPSESSID
WHMCSpKDlPzh2chML

Statistics

Statistic cookies help website owners to understand how visitors interact with websites by collecting and reporting information anonymously.

_ga - Preserves user session state across page requests.

_gat - Used by Google Analytics to throttle request rate

_gid - Registers a unique ID that is used to generate statistical data on how you use the website.

smartlookCookie - Used to collect user device and location information of the site visitors to improve the websites User Experience.

_ga, _gat, _gid
_ga, _gat, _gid
smartlookCookie

Marketing

Marketing cookies are used to track visitors across websites. The intention is to display ads that are relevant and engaging for the individual user and thereby more valuable for publishers and third party advertisers.

IDE - Used by Google DoubleClick to register and report the website user's actions after viewing or clicking one of the advertiser's ads with the purpose of measuring the efficacy of an ad and to present targeted ads to the user.

test_cookie - Used to check if the user's browser supports cookies.

1P_JAR - Google cookie. These cookies are used to collect website statistics and track conversion rates.

NID - Registers a unique ID that identifies a returning user's device. The ID is used for serving ads that are most relevant to the user.

DV - Google ad personalisation

IDE, test_cookie, 1P_JAR, NID, DV, NID
IDE, test_cookie
1P_JAR, NID, DV
NID
hblid

Security

These are essential site cookies, used by the google reCAPTCHA. These cookies use an unique identifier to verify if a visitor is human or a bot.

SID, APISID, HSID, NID, PREF
SID, APISID, HSID, NID, PREF