Need help?

Our experts have had an average response time of 13.14 minutes in February 2024 to fix urgent issues.

We will keep your servers stable, secure, and fast at all times for one fixed price.

How to use Fail2ban for securing Apache web server from 404 attacks?

by | Mar 5, 2019

404 bot attack is one of the powerful weapons of hackers.

But, that’s where Fail2ban comes for your help.

We can use Fail2ban to block malicious 404 scans and invalid requests on a web server such as Apache.

At Bobcares, we help server owners setup Fail2ban to block 404 bot attacks on their servers as part of our Server Management Services.

Today, let’s discuss how we setup Fail2ban to block 404 bot attacks on Apache web server.

 

What’s Apache 404 error and how Fail2ban helps here?

A 404 is an HTTP status code which indicates that the requested web page couldn’t be found on the server. In other words, the requested webpage doesn’t exist or broken.

And, hackers use this as a DDoS tool. They generate thousands of requests in a minute to the non existing web pages which result in 404 errors like this.

xxx.xxx.xx.xx - - [11/Nov/2018:22:25:01 +0100] xxx.xxx.x.x "POST /xx23456.php HTTP/1.1" 404 209 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:57.0) Gecko/20100101 Firefox/57.0"
xxx.xxx.xx.xx - - [11/Nov/2018:22:25:03 +0100] xxx.xxx.x.x "POST /xx123.php HTTP/1.1" 404 210 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:57.0) Gecko/20100101 Firefox/57.0"
xxx.xxx.xx.xx - - [11/Nov/2018:22:25:05 +0100] xxx.xxx.x.x "POST /xxx.php HTTP/1.1" 404 206 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:57.0) Gecko/20100101 Firefox/57.0"

 

But, the web server has to process these much requests in a minute putting the server on stress.

That’s where Fail2ban plays it’s role. Fail2ban constantly monitors the Apache logs. Once it identifies an unusual behavior, it blocks such repeating offenders using temporary firewall rules. In other words, there are custom fail2ban jails which monitor the log files for malicious 404 patterns and block those IPs in firewall.

Our Server Experts help customers installing fail2ban and custom jails on their servers. For example, we install fail2ban using the below command on an Ubuntu server.

apt-get install fail2ban

 

And, we make further modifications in the configuration file /etc/failban/jail.local.

 

How to setup Fail2ban to detect Apache 404 attacks?

Now, let’s see how our Dedicated Engineers setup Fail2ban to block 404 scans and invalid request methods.

1) Create filter

Firstly, our Support Engineers create a filter in the location /etc/fail2ban/filter.d. Further, we add a set of rules to ban IPs that cause 404 errors.

For example, to monitor the Apache 404 requests, we create a filter file apache-404.conf in the location /etc/fail2ban/filter.d. The filter looks like this.

failregex = ^ - .* "(GET|POST|HEAD).*HTTP.*" 404 .*$
ignoreregex =.*(robots.txt|favicon.ico|jpg|png)

 

We define the regular expression to be matched under the failregex parameter. Here, the above regular expression identifies the IP address that is making too many 404 requests. And, the ignoreregex excludes the valid files such as robots.txt, favicon.io and the images.

 

2) Create a custom jail

Secondly, we add a new jail in the location /etc/fail2ban/jail.conf. This defines the Apache log path to be checked, maxretry, bantime etc.

For example, to create a custom jail that monitors the Apache 404 requests, we add the following code in the file /etc/fail2ban/jail.conf.

[apache-404]
enabled = true
port = http,https
filter = apache-404
logpath = /var/log/httpd/error_log
logpath = /var/log/httpd/access_log
bantime = 3600
findtime = 600
maxretry = 5

 

Here, we update the apache log file under logpath parameter. Similarly, the bantime, species how many seconds an offending IP is banned for. We always set this value to an optimum level, so that it’s not short to affect the legitimate users, while not long enough favoring malicious users.

Further, the maxretry parameter specifies the total number of connection attempts. So, if a client makes retry attempts more than maxretry value within the time specified in findtime parameter, they will be banned.

Similarly, on Plesk servers, we create new jails from Tools & Settings > IP address Banning > Jails > Add Jail.

fail2ban apache 404

How to add a new jail in Plesk?

 

Finally, we restart fail2ban for the changes to take effect.

3) Test Fail2ban rules

Further, our Server Experts test the new Fail2ban rules on the server to ensure that they work as expected. For example, here we test the fail2ban filter apache-404.conf using the below command.

fail2ban-regex /var/log/httpd/error_log /etc/fail2ban/filter.d/apache-404.conf
fail2ban-regex /var/log/httpd/access_log /etc/fail2ban/filter.d/apache-404.conf

 

In addition to that, we verify the status of the newly created jail using the fail2ban-client command. For example, we use the following command to check the status of the newly created jail apache-404.

fail2ban-client status apache-404

 

The output will be like this.

Status for the jail: apache-404
|- filter
| |- File list: /var/log/httpd/access.log
| |- Currently failed: 0
| `- Total failed: 0
`- action
|- Currently banned: 0
| `- IP list:
`- Total banned: 0

4) Verify firewall rules

Finally, we verify that the rules were added by fail2ban to the server firewall. For example, we confirm whether fail2ban blocks IPs in iptables using the below command.

iptables -nL

 

Moreover, we check the fail2ban logs /var/log/fail2ban.log to confirm the IPs are banned.

2013-02-28 02:17:21,388 fail2ban.actions: WARNING [apache-404] Ban xxx.xxx.xx.xx
2019-02-28 02:37:21,195 fail2ban.actions: WARNING [apache-404] Unban xxx.xxx.xx.xx

 

Done! Fail2ban will now protect Apache from 404 bot attacks.

Conclusion

In short, it’s quite easy to setup fail2ban to protect Apache web server from 404 bot attacks. Today, we’ve discussed how our Dedicated Engineers setup fail2ban to protect Apache from 404 attacks.

PREVENT YOUR SERVER FROM CRASHING!

Never again lose customers to poor server speed! Let us help you.

Our server experts will monitor & maintain your server 24/7 so that it remains lightning fast and secure.

GET STARTED

var google_conversion_label = "owonCMyG5nEQ0aD71QM";

16 Comments

  1. Eugene van der Merwe

    This is a great article, very concise. My only concern is what would happen in the case of a major website redesign. We’re currently facing that situation. We installed numerous 301s but the number of pages failing are simply too large.

    My concern is that with a 404 blocker you might end up blocking search engine crawlers which could spell disaster for the customer.

    I guess the solution would be to heavily fine tune findtime and bantime but alas how? And how to avoid disasters in SEO if done incorrectly after a site relaunch that might already have 404s?

    I can however say from experience if one use something like:

    findtime=60
    maxretry = 8

    You’d stop some of the aggressive script kiddies.

    Anyway, just complimenting you on this article, it’s excellent and will do your hosting business a very well. I would host with you, but alas, I have my own hosting firm.

    Reply
    • Visakh S

      Hi Eugene,

      Thanks for the comment. 🙂

      Yes, a major site redeisgn will trigger the rule.

      However this article is ideally suited to a shared hosting environment where there won’t be any huge websites with a huge redesign. Such sites are exceptions rather than the rule.

      Big sites are ideally hosted in VPS containers where the rules can be tweaked accordingly. Better yet, after a big site redesign, it is ideal to list the old URLs and give then a new target. That can’t be done manually. We do it via scripts for the sites we develop and maintain.

      For that we take the list of URLs from the old database, and then generate .htaccess rules using scripts. And to prevent those old URLs being indexed in Google, we submit new sitemaps which will eventually phase out the old URLs.

      Eugene, I also want to point out that we are NOT a web host.

      We are a technical services provider for web hosts like you.

      We take care of web hosting infrastructure, and end-user technical support. We’ll be happy to talk to you on chat (click on icon at right-bottom) if you have any questions about our services.

      Have a great day! 🙂

      Reply
  2. kobi

    Hello Lakshmi Vijayakumar,

    Thank you for your clear and concise fail2ban instructional guide. Is it possible to include more than one log file for in the case where more than one website is hosted?

    Reply
    • Sijin George

      Hello Kobi,

      Thanks for the comment.
      Yes, it is possible to include more than one log file in a jail. And the logpath will look like:
      logpath = /path/to/website1.com/access.log
      /path/to/website2.com/access.log

      Again, Fail2ban works on per IP address basis. If you need a customized jail configuration, our Fail2ban experts can help. We’ll be happy to talk to you on chat (click on icon at right-bottom).

      Reply
    • Sijin George

      Thank you for the input.

      Reply
  3. Adi Hanif

    I got this error where 2 log path is not allowed
    option ‘logpath’ in section ‘apache-404’ already exists

    So, i delete second logpath. Then i got this error when using the conf above

    ERROR Failed during configuration: File contains no section headers.
    file: ‘/etc/fail2ban/filter.d/apache-404.conf’, line: 1
    ‘failregex = ^ – .* “(GET|POST|HEAD).*HTTP.*” 404 .*$\n’

    Reply
    • Adi Hanif

      So you need to add [Definition] on the top of apache-404.conf file

      Reply
      • Adi Hanif

        Now, jail Apache 404 is started but my fail2ban.log
        shows many warning like this
        http://prntscr.com/p1jrmb

        How to solve that?

        Reply
  4. Emre

    Hello!

    How should we do this for nginx?

    Reply
    • Sijin George

      Hello Emre,
      Fail2ban works well for Nginx as well. But differs in the config file, log location, etc.

      Reply
  5. Fathi Ben Nasr

    Hi, thank you for your article. I am in a situation where I run apache mostly behind a cluster of “Pound” reverse ssl proxies and some behind “Hitch” ssl terminator+”Varnish” web cache for media servers. Blocking at the apache level would leave the reverse proxies unprotected to DDOS.
    Varnish has a rich config and could see the response so I could block requests at the varnish level but this will not offload the Hitch ssl reverse proxy. Pound allows to inject some headers but not Hitch. Also, both Hitch and Pound don’t allow to log the response from the backend servers.

    Do you have any example for such situations ?

    Reply
    • Sijin George

      Hello Fathi,
      Looks like you have a custom configuration. Our Engineers can provide you advice on the best Fail2ban setup. We’ll be happy to talk to you on chat (click on the icon at right-bottom).

      Reply
  6. JamesS

    Thanks for the useful post.
    I’ve configured a jail.local file and enabled = true for both ssh and apache-auth.

    When run fail2ban-client status I only see one jail (sshd).
    Should I be seeing a jail for apache (which I want to protect, too)?

    Thanks!

    Reply
    • JamesS

      After the second time of restarting fail2ban my jails have shown up 🙂

      So, a slightly different question – I’m running a basic website (largely html with some php). What jails should I implement?

      Thank you!

      Reply
      • Sijin George

        Hello James,
        Glad to know that it worked for you. For a basic website, we recommend using jails for apache auth failure, bad bot check, apache overflow, etc. And since you use PHP, it’s worth to add jails for blocking php-url-fopen or similar dangerous functions. Just in case you need further help, we’ll be happy to talk to you on chat (click on the icon at right-bottom).

        Reply

Submit a Comment

Your email address will not be published. Required fields are marked *

Privacy Preference Center

Necessary

Necessary cookies help make a website usable by enabling basic functions like page navigation and access to secure areas of the website. The website cannot function properly without these cookies.

PHPSESSID - Preserves user session state across page requests.

gdpr[consent_types] - Used to store user consents.

gdpr[allowed_cookies] - Used to store user allowed cookies.

PHPSESSID, gdpr[consent_types], gdpr[allowed_cookies]
PHPSESSID
WHMCSpKDlPzh2chML

Statistics

Statistic cookies help website owners to understand how visitors interact with websites by collecting and reporting information anonymously.

_ga - Preserves user session state across page requests.

_gat - Used by Google Analytics to throttle request rate

_gid - Registers a unique ID that is used to generate statistical data on how you use the website.

smartlookCookie - Used to collect user device and location information of the site visitors to improve the websites User Experience.

_ga, _gat, _gid
_ga, _gat, _gid
smartlookCookie
_clck, _clsk, CLID, ANONCHK, MR, MUID, SM

Marketing

Marketing cookies are used to track visitors across websites. The intention is to display ads that are relevant and engaging for the individual user and thereby more valuable for publishers and third party advertisers.

IDE - Used by Google DoubleClick to register and report the website user's actions after viewing or clicking one of the advertiser's ads with the purpose of measuring the efficacy of an ad and to present targeted ads to the user.

test_cookie - Used to check if the user's browser supports cookies.

1P_JAR - Google cookie. These cookies are used to collect website statistics and track conversion rates.

NID - Registers a unique ID that identifies a returning user's device. The ID is used for serving ads that are most relevant to the user.

DV - Google ad personalisation

IDE, test_cookie, 1P_JAR, NID, DV, NID
IDE, test_cookie
1P_JAR, NID, DV
NID
hblid

Security

These are essential site cookies, used by the google reCAPTCHA. These cookies use an unique identifier to verify if a visitor is human or a bot.

SID, APISID, HSID, NID, PREF
SID, APISID, HSID, NID, PREF