“This definitely is a problem with your monitoring system! I never used this bandwidth. I was on holiday!”
The accounts department of the data center we managed referred this customer concern to us. His un-managed dedicated server showed a bandwidth spike of 20 times the normal usage, and had resulted in bandwidth overages charges.
The monitoring system was showing perfect stats for all other servers, and it looked like something that happened in the customer’s server.
We went back to the start date of the bandwidth spike. The nature and volume of the usage hinted at a malicious activity – probably a botnet based outbound DDoS. A further investigation of log files confirmed our suspicions. Hackers were able to download malware into the server using the infamous Shell Shock vulnerability, and was able to turn the server into a high bandwidth weapon to launch DDoS attacks on remote servers.
A quick update fixed the Bash vulnerability, and a thorough scan cleaned out all the malware present in the server. The hackers got only account level access, which ensured that the issue wont happen again.
As a follow-up action, monitoring systems were updated to alert us when any server showed more than 100% times the normal bandwidth usage, and procedures were updated to notify the subscriber of a possible abuse. Botnets are a serious threat to server and network reputation. One sure fire way to locate a botnet infection is to monitor bandwidth usage anomalies.
Bobcares systems administrators take care of tech support and infrastructure management for data centers and web hosts. Are you looking for ways to improve your service quality?
0 Comments