In IT systems, disaster can strike any moment and in any form. A hard disk failure, third party attack or even human errors can lead to data loss. To recover valuable data in such scenarios, backup management plays an inevitable role. But many times people just configure the backup once and leave it there. They assume that everything would just work fine when things go wrong.

Installing a backup software alone will not guarantee automatic recovery in case of data loss. Integrity check of the backups is also a very crucial aspect in backup management. This means verifying the backup for adequacy, completeness, accuracy and consistency. If the backup is not retrievable at the time of a disaster, that backup solution is of no use.

Are your backups serving their purpose?

Recently we were contacted by a hosting provider who had a proprietary backup software for his VPS servers running on RedHat Enterprise Linux. But many times the backups he obtained were incomplete or the backup process failed in between. Even after investing a huge amount, he had a backup solution that was of not much use. That’s when he came to Bobcares for a solution.

Upon analysis, we saw that backups failed in the existing solution due to backup process timing out often. This was because the backups of the entire VPS servers were configured to be taken as a single file, that caused the Disk I/O to shoot up during peak hours. It also led to disk space issues, causing data to be incomplete. Further, there was no integrity check done to ensure that the backups were adequate.

To provide the customer with a reliable and affordable backup solution to keep VPS server data for 7-10 days, we installed Bacula. Bacula is an open source network software that worked in a client-server model. Its capability to provide continuous data protection made it an ideal choice for our requirement. Bacula consists of Director daemon, Console, File daemon and Storage daemon. One server was configured as the Backup server and others as the clients. A server with 4 TB disk space was configured as the storage server.

Bacula backup for RHEL servers

Setting up the Bacula Server

The Bacula server managed the backup, storage and restore process of the data from the various client servers. We first installed and configured MySQL server in it for Bacula database purposes. Then the bacula-server package was installed to obtain the server deamons. These daemons were then configured using corresponding configuration files.

1. Director deamon

This daemon controls the backups and the restore services. The file “/etc/bacula/bacula-dir.conf” was used to manage the backup jobs, schedules, clients to be backed up, catalog, etc and to configure the Console program. Bacula Console is a command-line interface to control the Bacula Director.

root@bacula:/var/lib/bacula# bconsole
Connecting to Director 172.17.254.46:9101
1000 OK: bacula-dir Version: 5.2.6 (21 February 2012)
Enter a period to cancel a command.
*

2. Catalog

Catalog contained the list of files that had to be backed up. This list was stored in a MySQL database. The DB name, username and password were configured in the file “/etc/bacula/conf.d/catalogs.conf”. The database contained tables specific to various configuration settings of the bacula system. For example, the Client table contained the list of servers to be backed up.

mysql> show tables;
+------------------+
| Tables_in_bacula |
+------------------+
| BaseFiles |
| CDImages |
| Client |
| Counters |
| Device |
| File |
| FileSet |
| Filename |
| Job |
| JobHisto |
| JobMedia |
| Location |
| LocationLog |
| Log |
| Media |
| MediaType |
| Path |
| PathHierarchy |
| PathVisibility |
| Pool |
| RestoreObject |
| Status |
| Storage |
| UnsavedFiles |
| Version |
+------------------+
25 rows in set (0.00 sec)

 3. Backup schedule and files

We scheduled a weekly full backup and a daily incremental backup of the client servers using the config file /etc/bacula/conf.d/schedules.conf.

Schedule {
Name = PrimaryBackupCycle
Run = Level=Full sun at 17:00
Run = Level=Incremental mon,tue,wed,thu,fri,sat at 19:00
}

The files and folders to be backed up were mentioned in the file /etc/bacula/conf.d/filesets.conf. In this config file, it was possible to exclude any irrelevant files or folders too. We scheduled different backup timings for different VPS servers, to avoid resource overhead.

4. Storage deamon

The Storage program performs the storage and recovery of data to the backup location. The storage server was configured in the same network as the dedicated servers, for fast file transfer between them. The IP address, port and Director details were configured in the file “/etc/bacula/bacula-sd.conf”.

After making changes to the config files, we verified the settings and then restarted the services:

# bacula-dir -t
# bacula-sd -t
# service bacula-sd restart
# service bacula-director restart

Configuring the Bacula Clients

For every VPS server to be backed up, the File daemon was installed in it using the bacula-client package. The configuration file “/etc/bacula/bacula-fd.conf” contained the details of the Director.

root@bacula:/var/lib/bacula# /etc/init.d/bacula-fd status
 * bacula-fd is running

We wrote a custom script addserver.sh in the Bacula server to add the new VPS client to the server config files. It presented with an easy command line interface to create the config file for the client.

Enter backup policy name of your choice. If you have only one policy for a server, provide the server hostname here:
BackupClient1
Enter server IP address:
***.**.***.**
Enter disk number[1-3]
1

The script generated a conf file for the new client, which was then moved to the location “/etc/bacula/conf.d/clients”. The directory to restore the backup was created in this newly added VPS server. After configuring the file bacula-fd.conf in the server with the required details, the settings were verified and the file daemon was restarted.

bacula-fd -tc /etc/bacula/bacula-fd.conf
service bacula-fd restart

Testing and verifying the backups

The main aim of a backup solution is to ensure that the backed up data is identical to the original data. Validating a backup can be done using various methods such as comparing the disk space size of backup with the original and periodically performing test restores from the backups. After configuring the clients and the server, we tested and verified the backup process.

The following snippet confirmed that the backup server status was fine and listed the backup jobs in it.

*status director
 bacula-dir Version: 5.2.13 (19 February 2013) x86_64-redhat-linux-gnu redhat Enterprise release
 Daemon started 22-Nov-15 16:23. Jobs: run=0, running=2 mode=0,0
 Heap: heap=278,528 smbytes=110,703 max_bytes=118,641 bufs=330 max_bufs=334
 Scheduled Jobs:
 Level Type Pri Scheduled Name Volume
 ==========================================================
 Differential Backup 10 22-Nov-15 23:05 BackupClient1 Local-0001
 Differential Backup 10 22-Nov-15 23:05 BackupClient2 Local-0001
 Full Backup 11 22-Nov-15 23:10 BackupCatalog Local-0001
 
 Running Jobs:
 Console connected at 22-Nov-15 16:26
 JobId Level Name Status
 ==========================================================
 32 Full BackupClient1.2015-11-22_16.23.56_03 is running
 33 Full BackupClient2.2015-11-22_16.24.36_04 is waiting on max Storage jobs

To confirm whether the scheduled backup took place correctly for a particular client, the following check was done.

 *status client=BackupClient1
Connecting to Client BackupClient1 at ***.**.***.**:9102
BackupClient1.com-fd Version: 5.2.6 (21 February 2012) x86_64-pc-linux-gnu ubuntu 14.04
Daemon started 05-Dec-15 07:25. Jobs: run=32 running=0.
 Heap: heap=155,648 smbytes=16,795 max_bytes=841,467 bufs=54 max_bufs=123
 Sizeof: boffset_t=8 size_t=8 debug=0 trace=0 
Running Jobs:
Director connected at: 06-Jan-16 14:43
No Jobs running.
====
Terminated Jobs:
 JobId Level Files Bytes Status Finished Name 
============================================
3590 Incr 14 2.584 G OK 29-Dec-15 19:37 zabbix
 3609 Incr 14 2.601 G OK 30-Dec-15 19:25 zabbix
 3628 Incr 16 2.601 G OK 31-Dec-15 19:44 zabbix
 3647 Incr 15 2.599 G OK 01-Jan-16 19:16 zabbix
 3666 Incr 18 2.587 G OK 02-Jan-16 19:35 zabbix
 3703 Incr 23 5.137 G OK 04-Jan-16 19:33 zabbix
 3722 Incr 15 2.566 G OK 05-Jan-16 19:23 zabbix

The backup folders were then verified and compared with the client server data to ensure that the required files were copied as per our configuration. We also confirmed that we were able to restore the backups to the client VPS directories. The Bacula setup we configured helped to provide an affordable and reliable solution for continuous data protection.

Backup management involves both backup configuration and restoration solutions. Here we’ve covered how Bacula backups were designed and implemented for a VPS provider. Bobcares helps cloud providers minimize business downtime with our backup management services, which range from formulating the backup and recovery plan to restoring the data within no time.