Stuck with AWS EC2 revert to a known stable kernel? We can help you.

At Bobcares we assist our customers with several AWS queries as part of our AWS Support Services for AWS users, and online service providers.

Today, let us discuss how our Support Techs revert kernel.

AWS EC2 revert to a known stable kernel

If you performed a kernel update to your EC2 Linux instance but the kernel is now corrupt, then the instance can’t reboot.

You can’t use SSH to connect to the impaired instance.

There are two steps followed by our Support Techs and let us discuss them in detail.

1. Access the instance’s root volume.
2. Update the default kernel in the GRUB bootloader.

Access the instance’s root volume

There are two methods to access the root volume:

Method 1: Use the EC2 Serial Console

• If you’ve enabled EC2 Serial Console for Linux, you can use it to troubleshoot supported Nitro-based instance types.
• The serial console helps you troubleshoot boot issues, network configuration, and SSH configuration issues.
• The serial console connects to your instance without the need for a working network connection.
• You can access the serial console using the Amazon EC2 console or the AWS Command Line Interface (AWS CLI).
• Before using the serial console, grant access to it at the account level.
• Then, create AWS Identity and Access Management (IAM) policies granting access to your IAM users.
• Every instance using the serial console must include at least one password-based user.
• If your instance is unreachable and you haven’t configured access to the serial console, then follow the instructions in Method 2.

Method 2: Use a rescue instance

• Firstly, create an EBS snapshot of the root volume.
• Then, open the Amazon EC2 console.
• Select Instances from the navigation pane, and then choose the impaired instance.
• Choose Instance State, Stop instance, and then select Stop.
• In the Storage tab, under Block devices, select the Volume ID for /dev/sda1.
• Choose Actions, Detach Volume, and then select Yes, Detach. Note the Availability Zone.
• Launch a rescue EC2 instance in the same Availability Zone.
• After the rescue instance launches, choose Volumes from the navigation pane, and then choose the detached root volume of the impaired instance.
• Choose Actions, Attach Volume.
• Choose the rescue instance ID ( id-xxxxx), and then set an unused device. In this example, /dev/xvdb.
• Use SSH to connect to the rescue instance.
• Run the lsblk command to view your available disk devices.
• Create a mount directory, and then mount the root partition of the mounted volume to this new directory. In the preceding example, /dev/xvdb1 is the root partition of the mounted volume.

  sudo mkdir /mount
  sudo mount /dev/xvdb1 /mount

  You can now access the data of the impaired instance through the mount directory.

Update the default kernel in the GRUB bootloader

First and foremost, call the chroot function to change into the mount directory:

sudo chroot /mount

The current corrupt kernel is in position 0 (zero) in the list. The last stable kernel is in position 1.

To replace the corrupt kernel with the stable kernel, you can use one of the following procedures:

GRUB1 (Legacy GRUB) for Red Hat 6 and Amazon Linux

GRUB2 for Ubuntu 14 LTS and 16.04

GRUB2 for RHEL 7.5 and Amazon Linux 2

Finally, GRUB2 for RHEL 8 and CentOS 8

GRUB1 (Legacy GRUB) for Red Hat 6 and Amazon Linux 1

sudo sed -i '/^default/ s/0/1/' /boot/grub/grub.conf

sed -i 's/GRUB_DEFAULT=0/GRUB_DEFAULT=saved/g' /etc/default/grub

sudo update-grub

sed -i 's/GRUB_DEFAULT=0/GRUB_DEFAULT=saved/g' /etc/default/grub

sudo grub2-mkconfig -o /boot/grub2/grub.cfg

GRUB2 for RHEL 8 and CentOS 8

GRUB2 in RHEL 8 and CentOS 8 uses blscfg files and entries in /boot/loader for the boot configuration, instead of the previous grub.cfg format.

If the blscfg files are missing from this location or corrupted, grubby doesn’t show any results.

You must regenerate the files to recover functionality.

Therefore, the indexing of the kernels depends on the .conf files located under /boot/loader/entries and on the kernel versions.

Indexing is configured to keep the latest kernel with the lowest index.

• Run the grubby –default-kernel command to see the current default kernel:

grubby --default-kernel

• Run the grubby –info=ALL command to see all available kernels and their indexes:

grubby --info=ALL

Note the path of the kernel that you want to set as the default for your instance.

• Run the grubby –set-default command to change the default kernel of the instance:

grubby --set-default=/boot/vmlinuz-0-rescue-4.18.0-80.4.2.el8_1.x86_64

• Run the grubby –default-kernel command to verify that the preceding command worked:

grubby --default-kernel

If you’re accessing the instance using the EC2 Serial Console, then the stable kernel now loads and you can reboot the instance.

If you’re using a rescue instance, then follow the below steps:

Unmount volumes, detach the root volume from the rescue instance, and then attach the volume to the impaired instance

• Exit from chroot, and unmount /dev, /run, /proc, and /sys:

exit
sudo umount /mount/dev
sudo umount /mount/run
sudo umount /mount/proc
sudo umount /mount/sys
sudo umount /mount

• From the Amazon EC2 console, choose Instances, and then choose the rescue instance.
• Choose Instance State, Stop instance, and then select Yes, Stop.
• Detach the root volume id-xxxxx (the volume from the impaired instance) from the rescue instance.
• Attach the root volume you detached in step 4 to the impaired instance as the root volume (/dev/sda1), and then start the instance.

[Need help with AWS issues? We’d be happy to assist]

Conclusion

In short, today we saw how our Support techs reverted AWS EC2 to a known stable kernel.