Nobody’s been killed in a server crash

Imagine a server that keeps crashing every other day. For most webhosts’ this isn’t something too hard to imagine. Every host would have gone through this phase, where they are clueless as to why their server keeps going down.

Most of the time, the blame falls on faulty hardware. Usually this is true, circumstantial evidence proving that no recent changes were made to the software, and hence the source of issue would most likely be hardware. This doesn’t always have to be correct, as there are other things that could go wrong.

How to proceed

In the case of Linux servers, a quick look at the output of the command

last

dmesg

smartctl

Tools like

mpstat

iostat

sysstat

sar

Mostly servers get overloaded, causing them to crash. In such cases, the definitive edge in performing an in depth analysis can be obtained by logging the state of the server, it’s processes, and resource usage, so that it can be reviewed at a later stage. Here is a script that can help you with that.

Create a folder /var/log/cpu_mem/ and add a line in

motd

mkdir /var/log/cpu_mem/; echo “Check logs at /var/log/cpu_mem/ for detailed log for analysis” >>/etc/motd; touch /root/loadmon.sh;chmod 755 /root/loadmon.sh

Add the following content to the file /root/loadmon.sh using any popular text editor.

#!/bin/bash

#This simple script is to record the status of processes, memory usage, disk usage, CPU state, mysql process list, maillog etc. More stuff can be added to the list easily, by adding the command corresponding to the desirable output.

#Script written by Sankar.H

#Sets the variable LOAD to the value picked from proc

CPU=$(grep -c processor /proc/cpuinfo)

LOAD=$(awk '{print int($1)}' /proc/loadavg)

#Replace '$CPU' in the below if statement with the load average in integer, above which you need the logging enabled. -Not recommended

if [ $LOAD -ge $CPU ]

then

{

printf "n";date

printf "nn================nn Memory usage stats nn================nn"

printf " output of free -m Look for memory usage and swap usagen n"

free -m

printf "n Look for swap in and swap outn n"

vmstat

printf "nn================nTOP Snapshotn================nn"

top -n1 -b

printf "nn================nDisk Usagen================nn"

df -h

printf "nn================nMySQL Process Listn================nn"

mysqladmin proc stat

#Comment the above line, and uncomment the line below, if the server is having Plesk installed

#mysqladmin proc stat -u admin -p`cat /etc/psa/.psa.shadow`

printf "nn====n Disk I/O performance- check await and util n===nn"

iostat -xdk

printf "nn===n CPU usage - check for usr sys iowait idle percentages n===nn"

mpstat

printf "nn===nNetwork Stats - approximate no of connections. Check script for enabling more details.n===nn"

netstat -plan |wc -l

#If you need more network related information, uncomment the following line

#printf "nDetailed network logs: n";netstat -plan;netstat -s

printf "nn===nLook for errors or firewall messages in the dmesg o/p below n===nn"

dmesg|tail -30

}

>/root/ldmon;touch /var/log/cpu_mem/log$(date +%F-%H:%M); cat /root/ldmon >/var/log/cpu_mem/log$(date +%F-%H:%M)

fi 

Create a cron job for the periodic execution of the script. Setting the interval to every 2 or 5 minutes should be enough. Note that the script will record the details only if the server is overloaded. If you would want to test the script, you will have to replace the $CPU with 0, so that the script logs the details, even when server load is 0. Read the script for exact details.

The script can create log files which could take up a lot of space in your server, and it is important to clear old logs periodically. The following script can be set as a daily cron, to clear logs that are older than 2 days. Create a file /root/clear_old_logs.sh

touch /root/clear_old_logs.sh ;chmod 755 /root/clear_old_logs.sh

Add the following lines to the file /root/clear_old_logs.sh with any of your favorite text editors.

#!/bin/bash

Run the following from shell, to set the crons. In some servers like those running Ubuntu, the cron file would be at /var/spool/cron/crontabs/root , in those cases you might have to edit the following script with that path.

echo "*/2 * * * * /bin/sh /root/loadmon.sh >/dev/null 2>&1">> /var/spool/cron/root

A crashing server may not kill people, but it definitely kills business.

About the Author :

Sankar works as a Senior Software Engineer in Bobcares. He joined Bobcares back in April 2006. He loves grooming/mentoring people. During his free time, he listens to music, and enjoys singing..