In our role as Outsourced Tech Support for web hosts, VPS customers hosted in OpenVZ or Virtuozzo servers often report their sites crashing with the error OOM killed process httpd.
It often happens with high traffic websites or with sites hosted in starter plans. Today we’ll discuss what this error means and how we fix it for our customers. (more…)
“I did nothing. It just crashes all the time!”
So began a professional administration request at the help desk of a data center we managed. The customer’s unmanaged Windows 2008 R2 VPS started crashing one fine day without any apparent reason.
The event logs didn’t show anything out of the ordinary. So, the next step was to analyze the crash dump.
It was a peaceful night shift at a data center we managed. Just a few routine server provisioning and customer queries were keeping us occupied. Suddenly all alarm bells started ringing.
25+ managed server instances had gone offline, and the alert priority was among the highest. Each passing minute was eating into our SLA guarantee. An OpenVZ node had gone down with almost no warning at all. The monitors had shown a slight increase in load, but well within normal range.
OK, first order of business, bring the server back online. The OpenVZ kernel booted up, and all instances were back online in less than 15 minutes, but that cut our uptime to 99.96%. We just cannot afford to let it happen again, and so, we started digging.
“This definitely is a problem with your monitoring system! I never used this bandwidth. I was on holiday!”
The accounts department of the data center we managed referred this customer concern to us. His un-managed dedicated server showed a bandwidth spike of 20 times the normal usage, and had resulted in bandwidth overages charges.
The monitoring system was showing perfect stats for all other servers, and it looked like something that happened in the customer’s server.
Sometimes our network service won’t behave as per our requirement and in most of the cases the log do not help us to trace out the issue. Here comes the role of tcpdump and wireshark utility.
tcpdump is a packet sniffer. It captures the packets that fly in and out of the machine. tcpdump understands protocols and host names. First thing you have to remember is that you have to login to the server as root or be a sudoer. It is better to redirect the output to a file for debugging purpose. You can play on the recorded file to filter the packets that you are not interested to see. (more…)