We all know that a server’s stability is dependent on its load, directly. Load, in computing, is a measure of the amount of processing a computer is currently performing, usually as some variation on a percentage. Load spike is something from which you would always want your servers to stay away from.

This article mainly focus on finding the probable causes for a high load. I hope this article will help you in fighting those load spike nightmares.

 

 

Load Checking Commands

1.w

2.uptime

3.pstree

4.top

5.mysqladmin proc stat

Causes for Load

• High resource usage by some processes
• Perl script attacks
• Spam Attack
• Vulnerable PHP scripts
• MySQL Abuse
• DOS attack

High Resource Usage by Processes

You can find the resource usage, by the command ‘top -c’ and ‘pstree -apu’. Install CSF and configure it to get alerts on “Excessive resource usage.”

To get the email alerts, you can set your email address in this file: /etc/csf/resalert.txt

Perl Script Attacks

These attacks are usually caused by poor coding or vulnerabilities in the software version. If Perl scripts are running in the server, execute ‘pstree -apu’. You can see something similar to the snippet given below.

├─perl,30478,user
│   └─sh,30479 -c echo 40"`uname 40-a`";echo 40"`id`";/bin/sh
│       └─sh,30485

To find the source of the Perl script, execute ls -alh /proc/30478 |grep cwd

lrwxrwxrwx    1 user  user        0 Sep 28 06:16
cwd -> /home/user/public_html/media/

This means that, the Perl script is running from the location, /home/user/public_html/media/.

Also, you can use the command “ lsof -p 30478 |less ” to get more information.

Preventive measures for blocking such Perl attacks include:

• • Disable allow_url_fopen and allow_url_include in your php.ini (/usr/local/lib/php.ini) file
  • Disable the following functions in PHP:

disable_functions = exec, shellexec, passthru, system, escapeshellcmd, escapeshellarg

• Install mod_security (this also helps in reducing Iframe attacks )

Spamming

Spamming is basically flooding a server with multiple copies of the same message. This can be either incoming or outgoing.

Incoming spamming might be concentrated to some domain or some email accounts. Most of the incoming spamming are normally caused due to the “catch-all” being set as main account. So, it is always better to set the “catch-all” parameter as fail.

WHM >>Main >> Server Configuration >> Tweak Settings:(Under “Mail” section)

This is the default catch-all/default address behavior for new accounts. “fail” is usually the best choice if you are getting mail attacks.

Also, enable RBL and spamassasin options in your cPanel server.(WHM >> Main >> Exim Configuration)

For outgoing spamming, users might be use PHP scripts to send spam mails. You can find the source of those PHP scripts using the command 'ps -C exim -fH ewww |grep home‘.

Preventive measures to block spam include:

• Limit the emails that can be sent.
• Tweak the Mail server settings as Follows:  WHM >>Main >> Server Configuration >> Tweak Settings:”The maximum each domain can send out per hour (0 is unlimited) : 300″

Some helpful exim commands include:

exim -bp(For showing the mail queue)
exim -bpc(Count of mail queue)
ps -C exim -fH ewww (For finding script sources )
exim -bp |exiqsumm (To get full exim queue summary )
exiqgrep -z -i | xargs exim -Mrm (Remove all frozen messages)

PHP Scripts Causing High Load

This can easily be found out through the results of the 'pstree apu‘ and the ‘top -c‘ commands, during load spikes.

MySQL Abuse

You can track down MySQL usage using the command ‘mysqladmin proc stat‘.

DOS Attack (Denial Of Service)

By the name itself, we can understand what this would mean: denial of services like httpd, exim, ftp, etc. Mostly we deal with DOS attacks on Apache services, which may cause the sites to slow down or not load at all.

A DOS attack is a situation where there is a high number of simultaneous connections from a certain IP address or IP addresses. We can easily find the culprit IP by using the ‘netstat’ command. Once caught, it is better to block those IP’s using firewalls, without much delay.

The command for listing IP addresses and the number of connections is:

 netstat -plan |grep :80|awk '{print $5}' |cut -d: -f1 |sort |uniq -c |sort -n

The command for finding the total connections is:

 netstat -plan |grep :80 |wc -l

If you do find any IP address having too many connections (more than 50), then you have a good reason to block it either by using iptables, APF, or CSF.

Using iptables, you can block an IP by issuing the command:

iptables -A INPUT -s --dport 80 -p tcp -j DROP
/etc/rc.d/init.d/iptables save

If you are using APF, this command may be used:

apf -d IP
Using CSF:
csf -d IP

DDOS is distributed denial of service, which is an advanced form of DOS attack. The number of connections from an individual IP might be low, but the connections will be from many IPs or a range of IPs.

Most of times attack will be concentrated to one domain. In such cases, its always prudent to kill the dns of domain to reduce the attack.

Preventive measures for DDOS attacks include:

• Install firewalls like APF or CSF and configure them to block the high connection IPs.
• Install mod_dosevasive module (only during the DOS attack)

Conclusion

As we all know, “prevention is better than cure”. So, it is always wiser to take necessary measures before our servers go out of control. Happy monitoring!!

• Install System Integrity Monitor (SIM) or CSF and configure it to get alerts during load spikes.
• Update your old and vulnerable software
• Take all necessary preventive measures to avoid load spiking.
• Secure your servers and tweak apache, exim, etc for better performance.

---

About the Author :

Sudheesh has been with Bobcares for 3 years as a Linux Server Administrator. He is highly passionate about Linux. His other technical interests include Server security and Shell scripting. During his free time, Sudheesh loves listening to music and browsing the internet.

---