Many times we receive requests from our customers to recover the data from a terminated, corrupted, or lost EC2 instance.

Here, at Bobcares, we assist our customers with similar AWS queries as part of our AWS Support Services.

Today, let us see how our Support Techs recover the data from the Instance.

How to retrieve data from the EC2 instance?

In AWS, every instance creates a volume, when an instance is lost, corrupted, etc. Their volume exists there. So it is possible to retrieve data stored in an unreachable instance.

For that first we need to stop the instance, then detach the root volume, and attach that volume to a rescue instance.

If the EC2 instance is instance store backed or has an instance store containing data, do not perform this procedure. Because this retrieval procedure needs to stop and start of the instance. And, this will leads to loss of data from instance store volumes.

If the instance is part of an Amazon EC2 Auto Scaling group then stopping the instance could terminate the instance.

Also, if the instance is launched by services that use AWS Auto Scaling, such as Amazon EMR, AWS CloudFormation, etc., then stopping the instance could lead to its termination.

In these cases, the instance termination depends on the instance scale-in protection settings for the Auto Scaling group.

So If the instance is a part of an Auto Scaling group, remove the instance temporarily from the Auto Scaling group first. Then we can proceed with the recovery.

Steps to retrieve EC2 Instance Data

Today, let us see the steps followed by our Support Engineers to recover the data.

To get the latest data from the unreachable instance, we need to create a new rescue instance & Attach the new Volume with this instance.

Procedure

1.Initially, open the Amazon EC2 console.

2. From the AWS console, Choose Instances and then select the impaired instance.

3. ChangeInstance State to stop, Stop instance.

4. Then, In the Storage tab, select the Root device.

5.Choose Actions, Detach Volume.

6. When prompted for confirmation, choose Yes, Detach.

7. Then we need to verify that the Volume Status is available.

8. Create a new EC2 instance in the same Availability Zone as the impaired instance.

9. The new instance becomes the “rescue” instance. If there is any other accessible instance in the same Availability Zone and if it uses the same Amazon Machine Image (AMI), we can use it instead of creating a new instance.

10. After that, from the navigation pane, choose Volumes.

11. Select the detached root volume of the impaired instance.

12. Choose Actions, Attach Volume.

13. Select the rescue instance ID (i-xxxxx) and then enter a device name. The device name must be specified as /dev/sd[f-p].

14. Choose Attach.

15. Then connect the rescue instance via SSH.

16. Run the lsblk command to verify the device name of the attached volume.

$ lsblk

NAME    MAJ:MIN   RM  SIZE RO TYPE MOUNTPOINT
xvda    202:0     0   15G  0 disk
└─xvda1 202:1     0   15G  0 part /
xvdf    202:80    0  15G  0 disk
    └─xvdf1 202:0 0  15G  0 part

17. Create a mount directory and then mount it with root privileges.

$ sudo mount -o nouuid /dev/xvdf1 /mnt

18. Check the logs in the mount directory. If necessary, upload logs to Amazon Simple Storage Service (Amazon S3) using the AWS Command Line Interface (AWS CLI) or copy them to your local PC with the scp command.

$ sudo cp /mnt/var/log/messages .
$ sudo chmod 644 messages
$ exit

19. After checking the logs, run unmount /mnt to detach the attached volume.

$ sudo umount /mnt

20. Finally, attach the volume to the original instance and the device name is /dev/xvda.

Conclusion

In short, today we saw how our Support Techs retrieve data from an unreachable EC2 instance.