How we setup reliable backups in a server virtualization solution

In the online world, data loss can happen due to many reasons. Human errors, hardware issues or even natural calamities can end up causing a business downtime. Maintaining server backups is a preventive measure to minimize the business downtime due to such mishaps.

When we setup a server virtualization solution for a VPS hosting provider, maintaining reliable backups was an important consideration in our design. In our server virtualization system, individual VPS instances were hosted in LXC containers. Here is a walk-through of how we configured backups for these VPS servers.

Backup solution for LXC server virtualization

In our solution, the backup of each container was taken using a feature called ‘snapshot’. By default, snapshots are stored in the host server itself. But storing the backups locally, made them vulnerable to data loss, in case of a hard disk failure. For redundancy, we copied these backups to an external backup server daily.

Our backup solution included a system for creating container backups, syncing the backups to the backup server and restoring containers from these backups. This helped us to ensure that the services could be restored in no time. We automated the entire process to save time.

1. Creating backups

Backups of containers were created first, using ‘lxc-snapshot’ command. The container backup (aka ‘snapshot’) contained all the information of the container at any moment. Using this snapshot, we could restore the container at a later stage.

root@host:~# lxc-snapshot --name Ubuntu-01
lxc-snapshot: lxccontainer.c: do_lxcapi_snapshot: 3209 Snapshot of directory-backed container requested.
lxc-snapshot: lxccontainer.c: do_lxcapi_snapshot: 3210 Making a copy-clone.  If you do want snapshots,then
lxc-snapshot: lxccontainer.c: do_lxcapi_snapshot: 3211 please create an aufs or overlayfs clone first,snapshot that
lxc-snapshot: lxccontainer.c: do_lxcapi_snapshot: 3212 and keep the original container pristine.

2. Storing the backups

By default, snapshots are stored inside the container’s private folder with generic names such as snap0, snap1 etc. For instance, the first snapshot for the container ‘Ubuntu-01’ was named ‘snap0’ and was stored in ‘/var/lib/lxc/Ubuntu-01/snaps’ folder.

root@host:~# lxc-snapshot --name Ubuntu-01  --list
 snap0 (/var/lib/lxc/Ubuntu-01/snaps) 2016:01:19 10:00:40

root@host:/var/lib/lxc/Ubuntu-01/snaps# ls
 snap0

For redundant storage purposes, we setup an external backup server. To automate the backup process, we synced the backups from containers to the backup server. This was done using daily cron jobs.

Since our backup solution stored backups of the entire VPS containers into an external backup server, we wrote a custom script that renamed the backups to include date and time of backup, along with the container name. This enabled us to easily identify the snapshots, with their container name and date, for restoring them later on.

3. Automating the backups

Manually doing backups of the LXC containers was a time-consuming process, which was not feasible in our solution. For efficient and fast backups, we automated the backup process with a custom script. The script would stop each container at off-peak hours, take its snapshot, start the container and verify its running fine.

The script then labelled the backups with appropriate identifiers and copied the snapshot files to our external backup server over the network. The backup script was scheduled considering the factors like importance of data, storage space requirements, frequency of data changes etc. Backups were audited regularly to ensure its adequacy and rotated on a monthly basis to save storage space.

4. Restoring the backups

The main aim of a backup solution is to ensure that the backed up data is identical to the original data. We validated the backups by performing test restores of the container from these snapshots.

root@host:~# lxc-stop -n Ubuntu-01
root@host:~# lxc-snapshot -n Ubuntu-01 -r snap0

The LXC backup design we implemented, helped to provide an affordable and reliable solution for continuous data protection. Backup management involves both backup configuration and restoration solutions. Bobcares helps VPS providers minimize business downtime with our backup management services, which range from formulating the backup and recovery plan to restoring the data within no time.