Spamming has long been a pet peeve of server owners. It’s a problem that just refuses to die. Time and again it keeps re-appearing by taking advantage of new vulnerabilities in web or mail servers. As a cPanel server management company, spamming is one of the common issues we see on a daily basis.

For instance, one of our customers faced a similar issue in their network recently. This customer was a data center, and used our Dedicated Support services for technical support and server management. As part of this service, we monitor the server infrastructure round the clock, provide 24/7 emergency server support, and help server owners resolve technical issues. This data center had a zero-tolerance policy on spamming. We enforce this policy by detecting spam sources within the network and proactively contacting server owners to block spamming.

One day, we noticed one of the network IPs listed in a Real-time Blackhole List (RBL). We found the source was an un-managed server that was hosted in the network. It was an cPanel Exim mail server and was owned and operated by a web host who used the data center for co-location. We immediately informed the web host, and he, in-turn requested our assistance in resolving the issue. The web host explained that all possible anti-spam measures had been implemented in the server, and this spam campaign seemed to use a new un-known Exim server vulnerability.

Our first priority in responding to security issues (such as spamming or hack) is to immediately stop to an ongoing attack. This prevents further damage to server reputation and user data. When we logged into the server, we found that more than 60,000 spam mails had gone out and the mail queue was filled with approx. 10,000 spam and bounced mails. We quickly cleared the mail queue and blocked the IPs from where the spam originated. That put an effective end to the spam campaign.

Mitigating the spam campaign

Stopping an ongoing spam campaign is always the first step in spam mitigation. Once we are sure that the server is safe, our next priority is to find out how the attackers got access to the server. Finding the exploit path is critical in preventing similar attacks in the future. So, we started by analyzing the server logs to find the exact origin of the spam mails. We found that all spam mails had the FROM address of only one particular domain name (let’s say domain.com), but with random user names. It looked something like this:

WTSEGRFTDM@domain.com
TBNQQLDXGS@domain.com
PJNXCFVGOI@domain.com
UILLEDNHBQ@domain.com
WHRLHSKDFI@domain.com
TRHVZNJPAR@domain.com
RUPSPKDXPL@domain.com
MFWDCPDIIX@domain.com
DOFRRIYZMV@domain.com
GXEKHRIKOG@domain.com

For a short while it appeared as though spammers were using stolen login details to a valid email account in domain.com, and were spoofing the username. But we soon learned from log files that the spam mails were not using any authentication method – which meant only one thing – the server was acting as an open relay!

But, a quick check of the Exim server settings confirmed that the server was indeed locked down. Only authenticated relaying was allowed. So, we suspected what the server owner said might be true – Maybe the spammers were using a new un-disclosed Exim vulnerability. It was time to dive deeper.

We looked at the various security system logs to see if there has been a systematic hack attempt. None were in evidence. Then we turned our attention to a spam mail sample. While analyzing the spam mail header, we saw a curious header : acl_m_is_whitelisted.

This meant that all spam mails were considered as whitelisted by Exim. This was good news because of three reasons:

• It indicated Exim itself was not vulnerable.
• We could easily get the domain under attack off the whitelist, and prevent future spam attempts through this domain.
• The issue looked like an Exim mis-configuration, and could be fixed fast.

We took a look at the Exim configuration file and saw these entries:

domainlist whitelist_domains = nwildlsearch;/etc/virtual/whitelist_domains
   warn set acl_m_is_whitelisted = 0
   accept  condition = ${if eq{$acl_m_is_whitelisted}{1}{1}{0}}
   accept  sender_domains = +whitelist_domains
           logwrite = $sender_host_address whitelisted in local domains 
whitelist
           set acl_m_is_whitelisted = 1

The configuration settings meant that any domain name listed in /etc/virtual/whitelist_domains would be allowed to send mails through the Exim server without any authentication. That means, for all practical purposes, Exim acted as an open relay for mails sent from the domains listed in that file.

So, we quickly removed the affected domain from /etc/virtual/whitelist_domains and disabled the whitelisting ACLs from Exim. With that, we prevented any further possibility of a future spam episode using the same method.

So, was that game over? No, not yet!

Something had caused the Exim mis-configuration, and unless we identified what led to this issue, there was a chance that all of this could happen again. We’ve seen such issues in the past where an innocuous looking setting in the server update preferences or network settings in fact led to complicated issues later on. So, when we receive a support request, we make it a point to run a lead down until we know the whole sequence of exactly what happened.

Tracing the cPanel spam origin

As I mentioned earlier, the server was found to be properly locked down. It had only authenticated relaying enabled. But, there were whiltelisting configuration entries (aka ACLs) which were not standard in Exim servers. So, it could have been included by the server administrator for a specific purpose.

While investigating the whitelisting ACLs, we found that these Exim rules were part of the SpamBlockerTechnology security suite which is usually used in DirectAdmin Exim servers. We discussed this finding with the customer and found that this rule set was applied earlier that week to improve server security. The only problem was that this was a cPanel/WHM server and the DirectAdmin rule set was not entirely compatible with the Exim service running in it.

To make sure such issues don’t recur, we removed the SpamBlocker Technology rules from this server. We then helped the server owner lock down Exim service with custom anti-spam configuration settings that were compatible with cPanel/WHM servers.

Spamming is not new, and this won’t be the last we hear about it. Most spamming campaigns use stolen login details, vulnerable websites, or un-secured servers. However, as this case shows, even the best informed server owners can inadvertently create loop holes which spammers can exploit. That is why we always recommend our customers to do periodic top-down security audits to keep servers locked down air-tight.