“It makes no sense!”, so started a tech support chat one Tuesday morning. An e-commerce site had seen a sudden drop in online orders, and the reason was traced to Google branding it as a malware source. The web site looked perfectly normal, and the webmaster was frustrated that Google thought his site was infected. A quick check of the files showed malicious drive-by-download code inserted into all pages in the website. So, Google was indeed right.

For web site recovery, either we could clean all pages, re-upload the site files, or restore from backup. Cleaning the site would take the better part of a day, and re-uploading was not an option as the webmaster had only the files before he upgraded the CMS. We had a cPanel backup taken yesterday, and one from a week back. Yesterday’s backup was also infected, and the week old backup was too old as new products were added since then. With no other option, the week old backup was used, and site updates were applied again for a quick restore.

If only we had an arbitrary-day backup recovery system! This was not the first such issue, and it was time we did something about it.

Thus began our hunt for a better backup solution. A look at discussion forums revealed that we’re not alone, and reasons for restore requests varied from botched website upgrades to accidentally canceled hosting accounts.

A quick review of the support requests showed that customers asked for website backup up to 14 days old. cPanel backup tool could create one copy each of daily, weekly and monthly backup, but new backup data always replaced the old ones, which couldn’t give a data restore from, say 5 days back. To achieve that, each day’s backup need to be stored in a separate directory. The 500 GB backup disks connected to each server was not sufficient to store 14 separate copies of daily backups. So, something was needed to complement cPanel backups.

After an exhaustive study of various backup tools, an elegant backup system was implemented which could store 14 days worth of daily backups, 4 weeks worth of weekly backups and 3 months worth of monthly backups, all with a combination of the native cPanel backup tool and the open source RSnapShot backup tool, which used the RSync command available in all Linux distros.

The cPanel servers were configured to run daily incremental backup into the backup disk without using compression. RSnapShot tool was then configured to copy only the changed files to a central backup server, which copied each changed file into a different folder, while just creating hard links for un-changed files in that folder. RSnapShot was configured to store 14 differential copies of daily backups, 4 differential copies of weekly backups, and 3 differential copies of monthly backups, all from one seed copy of the server backp. This essentially resulted in storing 21 copies of a 300 GB server backup in just 450 GB of disk space, by using the Linux capabilities of hard linking, and change synching.

In one server pool with 6 shared servers, 14-day arbitrary-day restore was implemented with just two 2 TB central backup disks, and 500 GB secondary disks in the servers. In addition to meeting all customer requests, the new backup method resulted in higher server stability (load never going above 4.0 in a quad core server) by avoiding CPU usage in compression and redundant data copy. Backups now were completed in less than 30 minutes in contrast to full backups that took more than 3 hours every day. In summary, the backups now were Fast, Not resource hungry, and Many times more capable.

Good backup management practices go beyond setting up a well-designed system. From our experience, here are a few tips to maintain a high capacity, low overhead, reliable backup system:

1. Use incremental and differential backups where possible

Incremental backup in the source server keeps the data fresh with low server load overhead, and differential backup taken to a central backup server increases recovery points without using CPU intensive compression.

2. Avoid compression in the source server to reduce load

Keep only one updated incremental backup in the source server. This obviates the need for using CPU intensive compression. The objective of storing multiple backup snapshots can be achieved using differential backups.

3. Use a central (off site) backup system to store weekly and monthly backups

All backups should ideally be synched to a central backup server through a private bandwidth connection. This will enable you to restore the data to a new server even if the source server becomes in-accessible.

4. Use random restore checks to ensure integrity of backups

Once a week, randomly restore 3 accounts to test environment to make sure the backups are reliable. A Bask or Perl script can come in handy for this. The scripts can also automatically scan backup logs to locate any errors.

5. Define a cPanel backup policy that covers all your data restore contingencies

Plan for all kinds of server contingencies ranging from account level hack to data center downtime. Define a cPanel backup policy that will ensure data availability in all such situations.

Business challenges vary from one host to another. Bobcares cPanel administrators design and implement custom solutions for managing servers and improving support processes. Are you looking for ways to improve your service?

See how we can help