404 bot attack is one of the powerful weapons of hackers.
But, that’s where Fail2ban comes for your help.
We can use Fail2ban to block malicious 404 scans and invalid requests on a web server such as Apache.
At Bobcares, we help server owners setup Fail2ban to block 404 bot attacks on their servers as part of our Server Management Services.
Today, let’s discuss how we setup Fail2ban to block 404 bot attacks on Apache web server.
What’s Apache 404 error and how Fail2ban helps here?
A 404 is an HTTP status code which indicates that the requested web page couldn’t be found on the server. In other words, the requested webpage doesn’t exist or broken.
And, hackers use this as a DDoS tool. They generate thousands of requests in a minute to the non existing web pages which result in 404 errors like this.
xxx.xxx.xx.xx - - [11/Nov/2018:22:25:01 +0100] xxx.xxx.x.x "POST /xx23456.php HTTP/1.1" 404 209 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:57.0) Gecko/20100101 Firefox/57.0" xxx.xxx.xx.xx - - [11/Nov/2018:22:25:03 +0100] xxx.xxx.x.x "POST /xx123.php HTTP/1.1" 404 210 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:57.0) Gecko/20100101 Firefox/57.0" xxx.xxx.xx.xx - - [11/Nov/2018:22:25:05 +0100] xxx.xxx.x.x "POST /xxx.php HTTP/1.1" 404 206 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:57.0) Gecko/20100101 Firefox/57.0"
But, the web server has to process these much requests in a minute putting the server on stress.
That’s where Fail2ban plays it’s role. Fail2ban constantly monitors the Apache logs. Once it identifies an unusual behavior, it blocks such repeating offenders using temporary firewall rules. In other words, there are custom fail2ban jails which monitor the log files for malicious 404 patterns and block those IPs in firewall.
Our Server Experts help customers installing fail2ban and custom jails on their servers. For example, we install fail2ban using the below command on an Ubuntu server.
apt-get install fail2ban
And, we make further modifications in the configuration file /etc/failban/jail.local.
How to setup Fail2ban to detect Apache 404 attacks?
Now, let’s see how our Dedicated Engineers setup Fail2ban to block 404 scans and invalid request methods.
1) Create filter
Firstly, our Support Engineers create a filter in the location /etc/fail2ban/filter.d. Further, we add a set of rules to ban IPs that cause 404 errors.
For example, to monitor the Apache 404 requests, we create a filter file apache-404.conf in the location /etc/fail2ban/filter.d. The filter looks like this.
failregex = ^<HOST> - .* "(GET|POST|HEAD).*HTTP.*" 404 .*$
ignoreregex =.*(robots.txt|favicon.ico|jpg|png)
We define the regular expression to be matched under the failregex parameter. Here, the above regular expression identifies the IP address that is making too many 404 requests. And, the ignoreregex excludes the valid files such as robots.txt, favicon.io and the images.
2) Create a custom jail
Secondly, we add a new jail in the location /etc/fail2ban/jail.conf. This defines the Apache log path to be checked, maxretry, bantime etc.
For example, to create a custom jail that monitors the Apache 404 requests, we add the following code in the file /etc/fail2ban/jail.conf.
[apache-404]
enabled = true
port = http,https
filter = apache-404
logpath = /var/log/httpd/error_log
logpath = /var/log/httpd/access_log
bantime = 3600
findtime = 600
maxretry = 5
Here, we update the apache log file under logpath parameter. Similarly, the bantime, species how many seconds an offending IP is banned for. We always set this value to an optimum level, so that it’s not short to affect the legitimate users, while not long enough favoring malicious users.
Further, the maxretry parameter specifies the total number of connection attempts. So, if a client makes retry attempts more than maxretry value within the time specified in findtime parameter, they will be banned.
Similarly, on Plesk servers, we create new jails from Tools & Settings > IP address Banning > Jails > Add Jail.
How to add a new jail in Plesk?
Finally, we restart fail2ban for the changes to take effect.
3) Test Fail2ban rules
Further, our Server Experts test the new Fail2ban rules on the server to ensure that they work as expected. For example, here we test the fail2ban filter apache-404.conf using the below command.
fail2ban-regex /var/log/httpd/error_log /etc/fail2ban/filter.d/apache-404.conf
fail2ban-regex /var/log/httpd/access_log /etc/fail2ban/filter.d/apache-404.conf
In addition to that, we verify the status of the newly created jail using the fail2ban-client command. For example, we use the following command to check the status of the newly created jail apache-404.
fail2ban-client status apache-404
The output will be like this.
Status for the jail: apache-404
|- filter
| |- File list: /var/log/httpd/access.log
| |- Currently failed: 0
| `- Total failed: 0
`- action
|- Currently banned: 0
| `- IP list:
`- Total banned: 04) Verify firewall rules
Finally, we verify that the rules were added by fail2ban to the server firewall. For example, we confirm whether fail2ban blocks IPs in iptables using the below command.
iptables -nL
Moreover, we check the fail2ban logs /var/log/fail2ban.log to confirm the IPs are banned.
2013-02-28 02:17:21,388 fail2ban.actions: WARNING [apache-404] Ban xxx.xxx.xx.xx
2019-02-28 02:37:21,195 fail2ban.actions: WARNING [apache-404] Unban xxx.xxx.xx.xx
Done! Fail2ban will now protect Apache from 404 bot attacks.
Conclusion
In short, it’s quite easy to setup fail2ban to protect Apache web server from 404 bot attacks. Today, we’ve discussed how our Dedicated Engineers setup fail2ban to protect Apache from 404 attacks.
This is a great article, very concise. My only concern is what would happen in the case of a major website redesign. We’re currently facing that situation. We installed numerous 301s but the number of pages failing are simply too large.
My concern is that with a 404 blocker you might end up blocking search engine crawlers which could spell disaster for the customer.
I guess the solution would be to heavily fine tune findtime and bantime but alas how? And how to avoid disasters in SEO if done incorrectly after a site relaunch that might already have 404s?
I can however say from experience if one use something like:
findtime=60
maxretry = 8
You’d stop some of the aggressive script kiddies.
Anyway, just complimenting you on this article, it’s excellent and will do your hosting business a very well. I would host with you, but alas, I have my own hosting firm.
Hi Eugene,
Thanks for the comment. 🙂
Yes, a major site redeisgn will trigger the rule.
However this article is ideally suited to a shared hosting environment where there won’t be any huge websites with a huge redesign. Such sites are exceptions rather than the rule.
Big sites are ideally hosted in VPS containers where the rules can be tweaked accordingly. Better yet, after a big site redesign, it is ideal to list the old URLs and give then a new target. That can’t be done manually. We do it via scripts for the sites we develop and maintain.
For that we take the list of URLs from the old database, and then generate .htaccess rules using scripts. And to prevent those old URLs being indexed in Google, we submit new sitemaps which will eventually phase out the old URLs.
Eugene, I also want to point out that we are NOT a web host.
We are a technical services provider for web hosts like you.
We take care of web hosting infrastructure, and end-user technical support. We’ll be happy to talk to you on chat (click on icon at right-bottom) if you have any questions about our services.
Have a great day! 🙂
Hello Lakshmi Vijayakumar,
Thank you for your clear and concise fail2ban instructional guide. Is it possible to include more than one log file for in the case where more than one website is hosted?
Hello Kobi,
Thanks for the comment.
Yes, it is possible to include more than one log file in a jail. And the logpath will look like:
logpath = /path/to/website1.com/access.log/path/to/website2.com/access.log
Again, Fail2ban works on per IP address basis. If you need a customized jail configuration, our Fail2ban experts can help. We’ll be happy to talk to you on chat (click on icon at right-bottom).
I believe the nginx-botsearch will cover Apache as well, and comes with Fail2ban now:
https://fossies.org/linux/fail2ban/config/filter.d/nginx-botsearch.conf
Note the slight differences such as “/”
Thank you for the input.
I got this error where 2 log path is not allowed
option ‘logpath’ in section ‘apache-404’ already exists
So, i delete second logpath. Then i got this error when using the conf above
ERROR Failed during configuration: File contains no section headers.
file: ‘/etc/fail2ban/filter.d/apache-404.conf’, line: 1
‘failregex = ^ – .* “(GET|POST|HEAD).*HTTP.*” 404 .*$\n’
So you need to add [Definition] on the top of apache-404.conf file
Now, jail Apache 404 is started but my fail2ban.log
shows many warning like this
http://prntscr.com/p1jrmb
How to solve that?
Hello!
How should we do this for nginx?
Hello Emre,
Fail2ban works well for Nginx as well. But differs in the config file, log location, etc.
Hi, thank you for your article. I am in a situation where I run apache mostly behind a cluster of “Pound” reverse ssl proxies and some behind “Hitch” ssl terminator+”Varnish” web cache for media servers. Blocking at the apache level would leave the reverse proxies unprotected to DDOS.
Varnish has a rich config and could see the response so I could block requests at the varnish level but this will not offload the Hitch ssl reverse proxy. Pound allows to inject some headers but not Hitch. Also, both Hitch and Pound don’t allow to log the response from the backend servers.
Do you have any example for such situations ?
Hello Fathi,
Looks like you have a custom configuration. Our Engineers can provide you advice on the best Fail2ban setup. We’ll be happy to talk to you on chat (click on the icon at right-bottom).
Thanks for the useful post.
I’ve configured a jail.local file and enabled = true for both ssh and apache-auth.
When run fail2ban-client status I only see one jail (sshd).
Should I be seeing a jail for apache (which I want to protect, too)?
Thanks!
After the second time of restarting fail2ban my jails have shown up 🙂
So, a slightly different question – I’m running a basic website (largely html with some php). What jails should I implement?
Thank you!
Hello James,
Glad to know that it worked for you. For a basic website, we recommend using jails for apache auth failure, bad bot check, apache overflow, etc. And since you use PHP, it’s worth to add jails for blocking php-url-fopen or similar dangerous functions. Just in case you need further help, we’ll be happy to talk to you on chat (click on the icon at right-bottom).