Wondering how to fix Event Data getting Stale in Nagios? We can help you.

Often our customers using Nagios reported a bug relating to event data in some Nagios versions. However, this bug has been patched and will be available in releases later. But if we are experiencing this error and/or the Nagios service is taking an excessively long time to start, we may have a corrupted MySQL table.

Here at Bobcares, we handle similar issues as a part of our Server Management Services.

Today let’s see how our Support Engineers fix this issue for our customers.

How to fix Event Data getting Stale in Nagios

Following are the two methods with which we can fix this issue:

1. Repair MySQL database

We can repair the Nagios and nagiosql databases by running the following commands in the command line as the root user:

# /usr/local/nagiosxi/scripts/repairmysql.sh nagios
# /usr/local/nagiosxi/scripts/repairmysql.sh nagiosql
# /usr/local/nagiosxi/scripts/repairmysql.sh nagiosxi

Alternatively, we can use the following command for Nagios XI 2014 and further versions:

# cd /usr/local/nagiosxi/scripts/
# ./repair_databases.sh

This will actually repair both databases.

If we encounter an error, similar to the following:

SQL: DELETE FROM nagios_logentries WHERE logentry_time < FROM_UNIXTIME(1293570334)
SQL: SQL Error [ndoutils] :</b> Table './nagios/nagios_logentries' is marked as crashed and last (automatic?) repair failed CLEANING ndoutils TABLE 'notifications'...

We may need to run a force repair on the tables using the following commands:

RHEL 7|CentOS 7|Oracle Linux 7|Debian 9

# systemctl stop mariadb.service
# cd /var/lib/mysql/nagios
# myisamchk -r -f nagios_<corrupted_table>
# systemctl start mariadb.service
# rm -f /usr/local/nagiosxi/var/dbmaint.lock
# php /usr/local/nagiosxi/cron/dbmaint.php

Debian 8|Ubuntu 16/18

# systemctl stop mysql.service
# cd /var/lib/mysql/nagios
# myisamchk -r -f nagios_<corrupted_table>
# systemctl start mysql.service
# rm -f /usr/local/nagiosxi/var/dbmaint.lock
# php /usr/local/nagiosxi/cron/dbmaint.php

2. Repair MySQL tables

Firstly, we need to stop the following services:

# service nagios stop
# service ndo2db stop
# service mysqld stop

After that, we can run the following repair script for MySQL tables:

# /usr/local/nagiosxi/scripts/repairmysql.sh nagios

Then unzip and copy the following dbmaint file to /usr/local/nagiosxi/cron/. This will overwrite the previous version.

# cd /tmp
# wget http://assets.nagios.com/downloads/nagiosxi/patches/dbmaint.zip
# unzip dbmaint.zip
# chmod +x dbmaint.php
# cp dbmaint.php /usr/local/nagiosxi/cron

After that, we need to run the following commands:

# service mysqld start
# rm -f /usr/local/nagiosxi/var/dbmaint.lock
# /usr/local/nagiosxi/cron/dbmaint.php

After running this script, start these services:

# service ndo2db start
# service Nagios start

However, if we see any error output from this script, similar to this one:

SQL: DELETE FROM nagios_logentries WHERE logentry_time < FROM_UNIXTIME(1293570334)
SQL: SQL Error [ndoutils] :</b> Table './nagios/nagios_logentries' is marked as crashed and last (automatic?) repair failedCLEANING ndoutils TABLE 'notifications'...

We may need to run a force repair on the tables:

# service mysqld stop
# cd /var/lib/mysql/nagios
# myisamchk -r -f nagios_<corrupted_table>
# service mysqld start
# rm -f /usr/local/nagiosxi/var/dbmaint.lock
/usr/local/nagiosxi/cron/dbmaint.php

[Need Assistance? We are happy to help you!]

Conclusion

In short, we saw the causes for event data getting stale along with the steps that our Support Engineers follow to fix Event Data getting Stale in Nagios.