Need help?

Our experts have had an average response time of 11.7 minutes in August 2021 to fix urgent issues.

We will keep your servers stable, secure, and fast at all times for one fixed price.

EC2 Error Audit: Backlog Limit Exceeded – How to fix it

by | Aug 14, 2021

Generally, we see the EC2 Error Audit: Backlog Limit Exceeded in the Amazon EC2 Linux instance’s screenshot and system logs.

Here, at Bobcares, we assist our customers with several AWS queries as part of our AWS Support Services.

Today, let us discuss why we receive these messages, and how we can prevent them from reoccurring.

 

EC2 Error Audit: Backlog Limit Exceeded

In a Linux system, the audit backlog buffer is to maintain or log audit events. When a new audit event triggers, the system logs the event and adds it to the audit backlog buffer queue.

The backlog_limit parameter value is the number of audit backlog buffers.

By default, the parameter is set to 320:

# auditctl -s
enabled 1
failure 1
pid 2264
rate_limit 0
backlog_limit 320
lost 0
backlog 0

Any event beyond the default number can result in the following errors:

audit: audit_backlog=321 > audit_backlog_limit=320

audit: audit_lost=44393 audit_rate_limit=0 audit_backlog_limit=320

audit: backlog limit exceeded

-or-

audit_printk_skb: 153 callbacks suppressed

audit_printk_skb: 114 callbacks suppressed

An audit buffer queue at or exceeding capacity might also cause the instance to hang or remain unresponsive.

To avoid them, we increase the backlog_limit parameter value. Mostly, increasing buffer space helps avoid error messages.

Here is a calculation of the memory for the auditd backlog. We can use this to determine how large we can make the backlog queue without causing memory stress.

One audit buffer = 8970 Bytes
Default number of audit buffers (backlog_limit parameter) = 320
320 * 8970 = 2870400 Bytes, or 2.7 MiB

The MAX_AUDIT_MESSAGE_LENGTH parameter defines the size of the audit buffer.

 

How to fix further errors?

In case the instance is inaccessible with backlog limit exceeded messages in the system log, we stop and start the instance.

Then, we perform the following steps to change the audit buffer value.

Here we change the backlog_limit parameter value to 8192 buffers.

1. First and foremost, we access the instance via SSH.

2. Then we verify the current audit buffer size.

For Amazon Linux 1 and other OS that don’t have systemd:

$ sudo cat /etc/audit/audit.rules
# This file contains the auditctl rules that are loaded
# whenever the audit daemon is started via the initscripts.
# The rules are simply the parameters that would be passed
# to auditctl.

# First rule - delete all
-D

# Increase the buffers to survive stress events.
# Make this bigger for busy systems
-b 320

# Disable system call auditing.
# Remove the following line if you need the auditing.
-a never,task

# Feel free to add below this line. See auditctl man page

For Amazon Linux 2 and other OS that use systemd:

$ sudo cat /etc/audit/audit.rules
# This file is automatically generated from /etc/audit/rules.d
-D
-b 320
-f 1

3. After that we with an editor we access the audit.rules file:

For Amazon Linux 1 and other OS that doesn’t use systemd:

$ sudo vi /etc/audit/audit.rules

For Amazon Linux 2 and other OS that use systemd:

$ sudo vi /etc/audit/rules.d/audit.rules

4. Now we proceed to edit the -b parameter to a larger value.

$ sudo cat /etc/audit/audit.rules
# This file contains the auditctl rules that are loaded
# whenever the audit daemon is started via the initscripts.
# The rules are simply the parameters that would be passed
# to auditctl.

# First rule - delete all
-D

# Increase the buffers to survive stress events.
# Make this bigger for busy systems
-b 8192

# Disable system call auditing.

# Remove the following line if you need the auditing.

-a never,task

# Feel free to add below this line. See auditctl man page

$ sudo auditctl -s
enabled 1
failure 1
pid 2264
rate_limit 0
backlog_limit 320
lost 0
backlog 0

5. Once done, we restart the auditd service. The new backlog_limit value takes effect.

In addition, as we can see, it updates in auditctl -s:

# sudo service auditd stop
Stopping auditd: [ OK ]
# sudo service auditd start
Starting auditd: [ OK ]
# auditctl -s
enabled 1
failure 1
pid 26823
rate_limit 0
backlog_limit 8192
lost 0
backlog 0

[Stuck in between? We are here for you]

 

Conclusion

In short, we saw how our Support Techs fix the EC2 error.

PREVENT YOUR SERVER FROM CRASHING!

Never again lose customers to poor server speed! Let us help you.

Our server experts will monitor & maintain your server 24/7 so that it remains lightning fast and secure.

GET STARTED

var google_conversion_label = "owonCMyG5nEQ0aD71QM";

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

Privacy Preference Center

Necessary

Necessary cookies help make a website usable by enabling basic functions like page navigation and access to secure areas of the website. The website cannot function properly without these cookies.

PHPSESSID - Preserves user session state across page requests.

gdpr[consent_types] - Used to store user consents.

gdpr[allowed_cookies] - Used to store user allowed cookies.

PHPSESSID, gdpr[consent_types], gdpr[allowed_cookies]
PHPSESSID
WHMCSpKDlPzh2chML

Statistics

Statistic cookies help website owners to understand how visitors interact with websites by collecting and reporting information anonymously.

_ga - Preserves user session state across page requests.

_gat - Used by Google Analytics to throttle request rate

_gid - Registers a unique ID that is used to generate statistical data on how you use the website.

smartlookCookie - Used to collect user device and location information of the site visitors to improve the websites User Experience.

_ga, _gat, _gid
_ga, _gat, _gid
smartlookCookie

Marketing

Marketing cookies are used to track visitors across websites. The intention is to display ads that are relevant and engaging for the individual user and thereby more valuable for publishers and third party advertisers.

IDE - Used by Google DoubleClick to register and report the website user's actions after viewing or clicking one of the advertiser's ads with the purpose of measuring the efficacy of an ad and to present targeted ads to the user.

test_cookie - Used to check if the user's browser supports cookies.

1P_JAR - Google cookie. These cookies are used to collect website statistics and track conversion rates.

NID - Registers a unique ID that identifies a returning user's device. The ID is used for serving ads that are most relevant to the user.

DV - Google ad personalisation

IDE, test_cookie, 1P_JAR, NID, DV, NID
IDE, test_cookie
1P_JAR, NID, DV
NID
hblid

Security

These are essential site cookies, used by the google reCAPTCHA. These cookies use an unique identifier to verify if a visitor is human or a bot.

SID, APISID, HSID, NID, PREF
SID, APISID, HSID, NID, PREF