Are your scheduled backups no longer working in Nagios? We can help you with this problem.

Here at Bobcares, we have seen several such Nagios-related errors as part of our Server Management Services for web hosts and online service providers.

Today we’ll take a look at the causes for this error and see the fix.

Why are the scheduled backups no longer working in Nagios?

Sometimes, when you look at the location where the scheduled backups must be stored, you don’t find any xxxxxxx.tar.gz files being created.

Moreover, you can find many folders not being removed as per the backup limit setting in /store/backups/nagiosxi/ location.

Our Engineers found that the root cause of this problem is database corruption.

How we fix the ‘scheduled backups no longer working Nagios’ problem

In order to identify why the backups are failing, we suggest our customers do a manual backup in an SSH on the Nagios XI server. This will generate a verbose output that will help in determining why the backup is failing.

Here is a step-by-step procedure our Support Engineers provide to do a manual backup.

Creating A Backup From The Command Line

First, open a terminal or SSH session and log into the Nagios XI server as the root user.

Next, run the below script to create a backup of your Nagios XI.

# /usr/local/nagiosxi/scripts/backup_xi.sh

As a result, a successful backup will complete with the following message:

===============
BACKUP COMPLETE
===============

The backup will be stored in /store/backups/nagiosxi/1xxxxx.tar.gz location.

Creating A Manual Backup In The Web Interface

To create manual backups in the web UI via Admin > System Backups > Local Backup Archives.

Next, click on the Create Backup button and the backup process will begin. The backup process status doesn’t appear on the page. However, you will know the backup completion once the .tar.gz file appears in the list of backups.

Database corruption is one of the main reasons for the failure of backups. Here is backup failed due to database corruption which we came across recently.

Running configuration check…
Stopping nagios: done.
Starting nagios: done.
Backing up Core Config Manager (NagiosQL)…
tar: Removing leading `/’ from member names
tar: Removing leading `/’ from member names
Backing up Nagios Core…
tar: Removing leading `/’ from member names
tar: /usr/local/nagios/var/ndo.sock: socket ignored
tar: /usr/local/nagios/var/rw/nagios.qh: socket ignored
Backing up Nagios XI…
tar: Removing leading `/’ from member names
Backing up MRTG…
tar: Removing leading `/’ from member names
Backing up NRDP…
tar: Removing leading `/’ from member names
Backing up Nagvis…
tar: Removing leading `/’ from member names
Backing up MySQL databases…
mysqldump: Got error: 130: Incorrect file format ‘nagios_flappinghistory’ when using LOCK TABLES
Error backing up MySQL database ‘nagios’ – check the password in this script!

Here the error “check the password in this script” is just a generic message and is not the cause. The line before it explains the issue here.

In order to fix this problem, we repaired the Nagios and nagiosql databases. For that, we ran the following commands in the command line as the root user:

# /usr/local/nagiosxi/scripts/repairmysql.sh nagios
# /usr/local/nagiosxi/scripts/repairmysql.sh nagiosql
# /usr/local/nagiosxi/scripts/repairmysql.sh nagiosxi

In case, if you are running Nagios XI 2014 onwards, you can use:

# cd /usr/local/nagiosxi/scripts/
# ./repair_databases.sh

Then we run a force repair on the tables. To do that, here are the commands that we run.

RHEL 6|CentOS 6|Oracle Linux 6

# service mysqld stop
# cd /var/lib/mysql/nagios
# myisamchk -r -f nagios_<corrupted_table>
# service mysqld start
# rm -f /usr/local/nagiosxi/var/dbmaint.lock
# php /usr/local/nagiosxi/cron/dbmaint.php

RHEL 7|CentOS 7|Oracle Linux 7|Debian 9

# systemctl stop mariadb.service
# cd /var/lib/mysql/nagios
# myisamchk -r -f nagios_<corrupted_table>
# systemctl start mariadb.service
# rm -f /usr/local/nagiosxi/var/dbmaint.lock
# php /usr/local/nagiosxi/cron/dbmaint.php

Ubuntu 14

# service mysql stop
# cd /var/lib/mysql/nagios
# myisamchk -r -f nagios_<corrupted_table>
# service mysql start
# rm -f /usr/local/nagiosxi/var/dbmaint.lock
# php /usr/local/nagiosxi/cron/dbmaint.php

Debian 8|Ubuntu 16/18

# systemctl stop mysql.service
# cd /var/lib/mysql/nagios
# myisamchk -r -f nagios_<corrupted_table>
# systemctl start mysql.service
# rm -f /usr/local/nagiosxi/var/dbmaint.lock
# php /usr/local/nagiosxi/cron/dbmaint.php

After repairing the database, we run a manual backup to confirm that the issue is been resolved.

Then to remove the directories of failed backup files, execute the following command.

# find /store/backups/nagiosxi/ -mindepth 1 -maxdepth 1 -type d -exec rm -rf ‘{}’ \;

Watching Scheduled Backups

If you wish to see what exactly is happening when the scheduled backups are running, then you can watch the output live by executing the following command:

# tail /usr/local/nagiosxi/var/cmdsubsys.log -f

Press CTRL + C when you have finished.

[Need any further assistance in fixing Nagios errors? – We are here to help you]

Conclusion

In short, the main reason for the scheduled backups to fail is due to database corruption. Today, we saw the solution our Support Engineers provide to this problem.