According to surveys, 68% of the internet runs on Linux. So, if you have a website, chances are that it runs on a Linux server.

From big cloud providers to small VPS hosters, everyone sells Linux servers by default.

The trouble is, a lot of website owners are not familiar with Linux as they use Windows or iOS in their laptop.

This unfamiliarity leads to a poorly maintained server that becomes vulnerable to malware, hackers and resource bottlenecks.

Here at Bobcares, we help website owners and online service providers (eg. web hosts) maintain their Linux servers.

A core activity in our maintenance services is auditing the linux servers for security, performance or stability issues.

Today we’ll take a look at why you should regularly audit your server(s), and how to do it.

 

What is auditing, and why you should audit your Linux server(s)

Every software has bugs. Software vendors patch these bugs from time to time to prevent security, performance or usability issues.

But what if you are not aware that a bug exists?

Which is why it is important to look under the hood of your server once in a while, and see if any software updates are pending, or if any services need tuning.

This process is called “server auditing“.

Here at Bobcares, we audit our customer servers at least once a month.

During these audits we look at hardware health, pending updates, server performance, malware infection, and more. This helps us be proactive and prevent server issues than just react to them after the server has gone down.

 

How to audit your servers

The whole purpose of auditing your server is to find out if there’s any glaring or obscure issue that can affect your server security or stability.

To achieve this, it is important to be as thorough as possible. One way we do it is to build a checklist for each server (the checks can vary), and follow it strictly.

Here’s a general overview of such a checklist we use for shared hosting servers.

Note : It is important to know that the exact checks can vary based on the server type, no: of users, kind of software, etc. will (and should) vary. This list can only act as a loose framework for a customized checklist.

 

Hold on. Can you make me a server audit checklist?

Sure, we can.

In fact, we build and update hundreds of checklists everyday for our customers at the Server Management Services.

For as low as $60, we will audit your server, give you a customized audit report, explain to you what points we checked, and recommend actions to be taken.

To get started, visit our On-demand Server Support page, and request an audit. We are online 24/7 and can get on your case in a few minutes.

 

Now, let’s get back to the server audit framework:

1. Check resource usage (Memory, CPU, I/O, disk space)

Abnormal resource usage is usually an indicator of trouble.

Either you have a performance bottleneck, or someone has exploited a security loophole.

So, the first thing we look during server audits is CPU, Memory and Disk I/O trends.

(You’ll need monitoring tools such as sysstat or atop setup for this).

If we find an anomaly, we identify the service that’s causing the bottle neck, and from there, the user or program that’s causing it.

(Click here for more details on how to troubleshoot high resource usage).

The important thing to look for here is the trend. If you see an unexplained rise or fall in resource usage, investigate.

At the minimum, you need to look at these statistics:

• CPU usage
• Memory usage
• Disk I/O
• Disk space

 

2. Look for unusual log file entries (logins, connections, errors, etc.)

Server logs contain a gold mine of information.

Your goal should be to look through all primary log files, and find out unusual entries.

• Do you see frequent failed connections for one user? Maybe it’s a hacker trying to brute force their password.
• Do you see memory limit error for one user script? Maybe it has a security vulnerability that’s being used to send spam.
• Do you see more frequent mail bounces? Maybe your IP is blacklisted for spamming

The trouble is, log files have thousands of entries, and it is humanly impossibly to read all log entries.

So, look for error entries, and repetitions of error codes.

If you see a new error pop-up in the past few days or weeks, you should investigate.

Here at Bobcares, we have engineers skilled in coding. We cook up custom scripts on the fly to quickly find out anomalies, and zero-in on server issues within minutes.

If you are not sure how to do log analysis, don’t worry, we can take care of that for you.

Our basic server management costs only $69.99/month. It includes 24/7 server monitoring, 24/7 emergency response, server updates, basic auditing and more.

At the minimum, check these logs:

• Login and authentication logs
• System log
• Startup log
• Cron log
• HTTP / Mail log (or whatever is your production service)

 

3. Verify changes to system settings, program files and configuration files

System software vulnerabilities can give attackers administrator access to your servers.

The first thing they’ll do is to replace system files with altered versions of the file that can be used to launch further attacks.

So, it is always a good idea to check if any of your program files or configuration files were recently changed. If you detect an unauthorzed change, investigate. Fast.

Look for file changes under:

• /bin/
• /sbin/
• /etc/
• /boot/
• /dev/

Some automated tools like Lynis can make this process faster.

 

4. Scan your server for malware and rootkits

Ideally, your server should be configured to have your anti-virus automatically update itself, and rescan the server.

The second best option is to perform a scan when you audit the server. While doing this, check if there are any new database sources in the market, and if so, add them to the database update list.

You need at least these scans to be run:

• Anti-malware scan (LMD, CXS, etc.)
• Anti-virus scan (eg. ClamD)
• Rootkit scan (eg. RkHunter, ChkRootKit)

If you find an infection, it is not enough to remove the malware, but you need to find out how the malware got into the server in the first place, and patch the vulnerability.

For as low as $69.99/month, our experts will setup automated scanning, perform periodic manual scanning, and find out the infection route. Click here to know more about our server management services.

 

5. Check for outdated system software and user software

Software vendors find out and patch vulnerabilities all the time. Ideally, your server should be configured to either automatically apply these updates, or notify you of a new update.

If neither of that is possible, you’ll need to cehck for updates manually during the server audit.

Of these the system software like kernel should get the highest priority.

You can use the server’s package manager to list pending updates. For example, in RedHat compatible servers, you can use this command:

# yum list updates

But that wouldn’t show outdated packages that users have installed in their accounts. For that you’ll need to perform a file scan and list all version files that have an older release number.

Here at Bobcares, our engineers use custom built scripts for each server to list out vulnerable software.

If you need help keeping your servers updated, we can do it for you for as low as $69.99/month.

 

6. Re-evaluate adequacy of service, firewall and system settings

A well configured firewall and network will block almost all attacks at the connection level.

But as new kinds of attack evolve, your firewall and network rules need to be updated to be immunized against it.

During your server audit, check if your firewall and network rules are adequate to block newer threats (eg. DNS cache poisoning).

It is also a good idea to check your server performance history, and tweak your service settings to prevent bottlenecks.

 

7. Examine system hardware and network health

Hard disk crashes are a common cause for extended server downtimes.

But before a disk fails, it’ll usually log a lot of error messages in system log.

So, go through the system log for HDD, RAID, Network or Memory errors, and investigate if anything seems amiss.

Here at Bobcares, we’ve saved unexpected server crashes by detecting hardware issues early on, and scheduling a maintenance window which doesn’t affect server users.

 

8. Verify if backups and fail-overs are working right

Backups are your insurance against server crashes.

But how do you know if your backups can be trusted? Did it backup ALL the data? Is the archive corrupted?

You won’t know it unless you test it.

Server audit is a great time to review backups.

During audits, we not only test if all backups are completed, but also test the integrity of archives, confirm enough space is available for backup growth, and more.

It is also a good time to re-consider if you need to increase or decrease your backup frequency in case your business demands have changed.

 

9. Look for outdated user accounts or uncancelled admin privileges

Hackers need not exploit a software if they have the key to your server.

Un-secured user accounts and admin accounts are a common way for hackers to breach servers.

During the server audit, check these logins and accesses of old users:

• SSH logins
• Mail logins
• IP whitelists
• FTP logins
• ..and more

A good way to go about this is to look at your /etc/passwd file and systematically remove all users that are not supposed to be present in your server.

 

10. Dry run emergency alert and response procedures

Let’s say the worst case happened. Your server either crashed or was hacked.

You really don’t have the time to think things through. You’ll need to be notified of the issue as soon as possible, and you’ll need your business back online before your customers notice.

That is why you need a robust emergency reaction process.

It includes 24/7 montioring, alert notification, emergency access to experts, pre-thoughtout process to get the server back online, and more.

Server audit is a good time to review & test this process, and make changes it needed.

 

How to get started?

OK, I’ll not sugar coat this for you. It can be a bit hard to get started if you are not familiar to Linux servers.

The fastest way is to get a professional Linux administration company like Bobcares perform the initial hardening of the server, setup backup & emergency response systems, and put the server on a regular audit channel.

At $69.99/month, you can skip the ardous journey to become a Linux guru, and actually focus on your business.

I’d recommend you take a look at our server management plans. It starts from as low as $24.99/month.