Bobcares

WeSupport

Call Us! 1-800-383-5193
Call Us! 1-800-383-5193
Call Us! 1-800-383-5193

Need Help?

Emergency Response Time custom

Our experts have had an average response time of 11.06 minutes in March 2021 to fix urgent issues.

We will keep your servers stable, secure and fast at all times for one fixed price.

Crashed Database Tables in Nagios XI – How to resolve

by | Dec 24, 2020

Crashed Database Tables in Nagios might trigger if the MySQL process appears to be consuming large amounts of CPU.

Here at Bobcares, we assist our customers with several Nagios-related issues as part of our Server Management Services.

Today, let us discuss managing Crashed Database Tables in Nagios XI.

Crashed Database Tables in Nagios

Crashed Database Tables

If we experience a high load on the Nagios XI server and the MySQL process (mysqld) consume large amounts of CPU, it may be a symptom of one or more crashed tables.

In order to find out if we have crashed database tables or any other errors, we can check the mysqld.log:

tail /var/log/mysqld.log

We may see entries similar to this one:

150110 9:05:01 [ERROR] /usr/libexec/mysqld: Table ‘./nagios/nagios_logentries’ is marked as crashed and last (automatic?) repair failed

 

Causes of Crashed Database Tables in Nagios

Database corruption usually causes by power outages, low disk space, or improper shutting down of the Nagios XI server.

Nagios is constantly reading from and writing to the MySQL database, so killing the mysqld process will lead to crashed tables.

Many times users would right-click on the Nagios XI VM in VMWare Workstation/ESXi/Vsphere and select “Power Off”. This is wrong – it is like pulling off the power cord.

However, to shut down the Nagios XI server, the proper way would be to issue the following command:

shutdown -h now

If the machine has insufficient disk space, we come across an error when we run the repair database script:

/usr/local/nagiosxi/scripts/repairmysql.sh: line 59: 11735 Segmentation fault (core dumped) $cmd $t –sort_buffer_size=256M
Timeout error occurred trying to start MySQL Daemon.
Starting mysqld: [FAILED]

[Looking for an easy fix? Contact us now!]

 

Solutions for Crashed Database Tables in Nagios

We can repair the Nagios and nagiosql databases using the following commands as the root user:

/usr/local/nagiosxi/scripts/repairmysql.sh nagios
/usr/local/nagiosxi/scripts/repairmysql.sh nagiosql
/usr/local/nagiosxi/scripts/repairmysql.sh nagiosxi

Alternatively, if we are running Nagios XI 2014 onwards, we can use:

cd /usr/local/nagiosxi/scripts/repair_databases.sh

This will repair both databases (Nagios and nagiosql) and  (nagiosxi in XI 5.x).

If we receive an error similar to this one:

SQL: DELETE FROM nagios_logentries WHERE logentry_time < FROM_UNIXTIME(1293570334)
SQL: SQL Error [ndoutils] :</b> Table ‘./nagios/nagios_logentries’ is marked as crashed and last (automatic?) repair failed CLEANING ndoutils TABLE ‘notifications’…

We may need to run a force repair on the tables using the commands below:

RHEL 6|CentOS 6|Oracle Linux 6

service mysqld stop
cd /var/lib/mysql/nagios
myisamchk -r -f nagios_<corrupted_table>
service mysqld start
rm -f /usr/local/nagiosxi/var/dbmaint.lock
php /usr/local/nagiosxi/cron/dbmaint.php

RHEL 7|CentOS 7|Oracle Linux 7|Debian 9

systemctl stop mariadb.service
cd /var/lib/mysql/nagios
myisamchk -r -f nagios_<corrupted_table>
systemctl start mariadb.service
rm -f /usr/local/nagiosxi/var/dbmaint.lock
php /usr/local/nagiosxi/cron/dbmaint.php

Ubuntu 14

service mysql stop
cd /var/lib/mysql/nagios
myisamchk -r -f nagios_<corrupted_table>
service mysql start
rm -f /usr/local/nagiosxi/var/dbmaint.lock
php /usr/local/nagiosxi/cron/dbmaint.php

Debian 8|Ubuntu 16/18

systemctl stop mysql.service
cd /var/lib/mysql/nagios
myisamchk -r -f nagios_<corrupted_table>
systemctl start mysql.service
rm -f /usr/local/nagiosxi/var/dbmaint.lock
php /usr/local/nagiosxi/cron/dbmaint.php

In certain instances, it may be necessary to truncate (empty) one or more tables.

The following commands provide examples of how to truncate both the nagios_logentries and nagios_notifications tables in the Nagios MySQL database:

mysql -u ndoutils -pn@gweb nagios -e ‘TRUNCATE TABLE nagios_logentries’
mysql -u ndoutils -pn@gweb nagios -e ‘TRUNCATE TABLE nagios_notifications’

These commands will clear all entries from the affected tables. After we truncate tables, we should repeat the repair process outlined above.

Finally, if we have run out of disk space on the Nagios XI server, clear unnecessary data. Once we have resolved the disk space issue, we should run the repair database script again.

[Stuck with a crashed database? We are here for you!]

 

Conclusion

To conclude, if the MySQL process appears to be consuming large amounts of CPU, this error occurs. Today, we saw an effective way our Support Engineers employ in order to fix it.

PREVENT YOUR SERVER FROM CRASHING!

Never again lose customers to poor server speed! Let us help you.

Our server experts will monitor & maintain your server 24/7 so that it remains lightning fast and secure.

GET STARTED

var google_conversion_label = "owonCMyG5nEQ0aD71QM";

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

Privacy Preference Center

Necessary

Necessary cookies help make a website usable by enabling basic functions like page navigation and access to secure areas of the website. The website cannot function properly without these cookies.

PHPSESSID - Preserves user session state across page requests.

gdpr[consent_types] - Used to store user consents.

gdpr[allowed_cookies] - Used to store user allowed cookies.

PHPSESSID, gdpr[consent_types], gdpr[allowed_cookies]
PHPSESSID
WHMCSpKDlPzh2chML

Statistics

Statistic cookies help website owners to understand how visitors interact with websites by collecting and reporting information anonymously.

_ga - Preserves user session state across page requests.

_gat - Used by Google Analytics to throttle request rate

_gid - Registers a unique ID that is used to generate statistical data on how you use the website.

smartlookCookie - Used to collect user device and location information of the site visitors to improve the websites User Experience.

_ga, _gat, _gid
_ga, _gat, _gid
smartlookCookie

Marketing

Marketing cookies are used to track visitors across websites. The intention is to display ads that are relevant and engaging for the individual user and thereby more valuable for publishers and third party advertisers.

IDE - Used by Google DoubleClick to register and report the website user's actions after viewing or clicking one of the advertiser's ads with the purpose of measuring the efficacy of an ad and to present targeted ads to the user.

test_cookie - Used to check if the user's browser supports cookies.

1P_JAR - Google cookie. These cookies are used to collect website statistics and track conversion rates.

NID - Registers a unique ID that identifies a returning user's device. The ID is used for serving ads that are most relevant to the user.

DV - Google ad personalisation

IDE, test_cookie, 1P_JAR, NID, DV, NID
IDE, test_cookie
1P_JAR, NID, DV
NID
hblid

Security

These are essential site cookies, used by the google reCAPTCHA. These cookies use an unique identifier to verify if a visitor is human or a bot.

SID, APISID, HSID, NID, PREF
SID, APISID, HSID, NID, PREF