What if, your server suffered an unexpected hard disk failure and the entire data on your server suddenly vanished? You frantically look for ways to recover the lost data, only to find that there are no backups available.

A single disk failure instance has turned out to be detrimental to your business. Data loss due to disk failures, human errors or virus attack can lead to business downtime and financial loss.

In spite of that, many business owners tend to overlook this risk and skip the backup process, leading to catastrophic events. For every business that depends on data availability, it is therefore very critical to safeguard and backup data.

For instance, in the case of a server virtualization system, one disk failure can affect multiple customers who have their server instances associated with that disk.

For effective data recovery and retrieval in such scenarios, Bobcares engineers design superior backup solutions to ensure minimal downtime.

 

To know more about the backup management in cloud servers, I consulted Bobcares engineer Jason, who works on OnApp cloud management software. His team manages the cloud setup for our clients who are cloud providers. They resolve issues that range from memory errors in a VM, to load balancing the entire cloud infrastructure. They configure and manage both manual and auto-backups to securely store and recover the data.

 

[ You don’t have to lose your sleep to keep your customers happy. Our Hosting Support Specialists cover your servers and support your customers 24/7 at just $9.99/hour. ]

“Around 3% of backup tasks end up throwing errors, even in a stable cloud setup”, says Jason, “and that’s when we jump in to fix those”. He then gave me an insight into some errors that they tackled.

He said, one day he was contacted by a VM owner to upgrade his OS to CentOS 6. Before the upgrade, his team decided to take a backup of the VM as a proactive measure.

Jason logged into the OnApp admin panel, clicked on “Backups -> Files -> Manual backup” and initiated the backup. However, the backup task failed, throwing up an error “Cannot snapshot a degraded VDisk 1cyys5g9v32r7i”.

Before we discuss the solution here, let me tell you how this all works. The OnApp cloud system architecture comprises of hypervisors that contain server instances known as Virtual machines (VM).

There are data stores alloted to each VM and the data is usually stored in RAID arrays.

 

RAID disks help to overcome the shortcomings of a single storage disk with regard to fault tolerance, overall performance and storage capacity. There are, but times, when they experience a failure and go into degraded mode.

When a RAID array gets into a degraded mode, its performance gets affected. A degraded disk needs to be repaired, before it can be expected to perform as needed.

The error message that Jason got indicated that the VM’s disk is degraded and it is not possible to backup a degraded disk. To resolve this particular error, Jason decided to run a repair process on the degraded VM disk first.

Once the repair was done successfully, he confirmed that the disk was healthy, before he reattempted the backup.

[ Use your time to build your business. We’ll take care of your customers. Hire Our Hosting Support Specialists at $9.99/hr. ]

Resolving “Cannot snapshot a degraded VDisk 1cyys5g9v32r7i”

This is how he did it. Jason logged in to the hypervisor of that VM as root user. He ran the OnApp commandline tool ‘getdegradedvdisks’, that lists the degraded disks in each data store.

[root@HV ~]# ./getdegradedvdisks 
Datastore: xxxxxxxxxx 
vdisks_with_no_redundancy: 5
degraded_snapshots: 1 (1cyys5g9v32r7i) 
vdisks: 1 (1cyys5g9v32r7i)

As you can see, the disk with label ‘1cyys5g9v32r7i’ was degraded.

So, he executed the tool to repair all the degraded disks, one after another.

 [root@HV ~]# ./repairvdisks
 Success!

The disk was repaired successfully and he was able to take the backup of the VM without any further errors.

Now, all this may seem very simple when you read it. But there are chances that if the repair is done without attention to detail, the disk may end up totally unrepairable. Also, once repaired, we ensure that no disks are out of sync, before attempting the next backup.

[ Running a hosting business doesn’t have to be hard, or costly. Get world class Hosting Support Specialists at $9.99/hour (bulk discounts available) ]

Bobcares has successfully completed such tasks, mitigating all the risks that are usually associated with the process. Our 24/7 team of engineers help data centers and cloud service providers deliver reliable and secure services to their customers with our troubleshooting skills and technical expertise.