Need help?

Our experts have had an average response time of 11.7 minutes in August 2021 to fix urgent issues.

We will keep your servers stable, secure, and fast at all times for one fixed price.

Can’t push log data to CloudWatch Logs with awslogs agent

by | Aug 24, 2021

Can’t push log data to CloudWatch Logs with awslogs agent? We can help you.

Here, at Bobcares, we assist our customers with several AWS queries as part of our AWS Support Services.

Today, let us see how we can troubleshoot this.

 

Can’t push log data to CloudWatch Logs with awslogs agent

In order to begin, we need to make sure that the awslogs agent can connect to the CloudWatch Logs API endpoint.

In addition, the configuration must have the following:

  1. Internet connectivity
  2. Valid security group configurations
  3. Valid network ACLs

 

Fingerprint issues

While we configure data to push to CloudWatch we set a file’s path. We review the header lines of the source log file.

Suppose the first few lines are blank or contain non-event data that stays the same. This means, there is an issue with the log-identifying hash.

In this case, we update the file_fingerprint_lines option in the agent configuration file.

In addition, we need to ensure to specify what lines in each file we use to generate the identifying hash.

 

Check the awslogs log file for errors

Initially, we review the /var/log/awslogs.log log file. We might come across the errors below.

  • Permissions errors:
NoCredentialsError: Unable to locate credentials

In case we didn’t add an AWS IAM role to the instance, we need to create and attach it.

However, if we did add an IAM role to the instance, we need to update the IAM user credentials in the /etc/awslogs/awscli.conf file.

ClientError: An error occurred (AccessDeniedException) when calling the PutLogEvents operation: User: arn:aws:iam::012345678910:<role/user>/<iam-user-name> is not authorized to perform: logs:PutLogEvents[…]

Here, we configure the IAM role or user with the necessary permissions for CloudWatch Logs.

  • Timestamp errors:
Fall back to previous event time: {‘timestamp’: 1492395793000, ‘start_position’: 17280L, ‘end_position’: 17389L}, previousEventTime: 1492395793000, reason: timestamp could not be parsed from message.

We make sure that the log events begin with a timestamp.

Then we check if the datetime_format in /etc/awslogs/awslogs.conf matches the timestamp format of the log events.

If not, we change the datetime_format.

No file is found with given path ‘<PATH-TO-FILE>’

All we need to do here is to update the log file path in the agent configuration file to the correct path.

Caught exception: An error occurred (InvalidSequenceTokenException) when calling the PutLogEvents operation: The given sequenceToken is invalid[…] -or- Multiple agents might be sending log events to log stream[…]

It is not possible to push logs from multiple log files to a single log stream. So, we update the configuration to push each log to a log stream-log group combination.

 

Other awslogs issues

Sometimes, the logs may stop pushing after a log rotation. In that case, we check the supported log rotation methods.

If it pushes briefly after the restart, we check for duplicates in the [logstream] section of the agent configuration file.

Suppose the awslogs.log log file takes up too much disk space. Then we need to check the log file for errors and then correct them.

If it contains only informational messages, we specify a lower logging level.

 

Further troubleshooting

For further troubleshooting, we note the instance-id. Then, we collect and review the following based on the configuration.

Yum installations:

  • yum version
    $ yum info awslogs
    $ yum info aws-cli-plugin-cloudwatch-logs
  • /etc/awslogs/awslogs.conf file
  • /etc/awslogs/awscli.conf file
  • Other relevant files in /etc/awslogs/
  • /var/log/awslogs.log file

Script-based installations:

  • The awslogs version:
    $ /var/awslogs/bin/awslogs-version.sh
  • /var/awslogs/etc/awslogs.conf file
  • /var/awslogs/etc/awscli.conf file
  • Other relevant files in /var/awslogs/etc/
  • /var/log/awslogs.log
  • /var/log/awslogs-agent-setup.log

For rotation-related issues, collect and review:

  • A snippet of the source logs
  • A list of the monitoring target directory’s contents:
    $ ls -la <Monitoring-Target-Directory-Path>

[Need help with the troubleshooting? We are here for you]

 

Conclusion

In short, we saw how our Support Techs troubleshoot the log data query for our customers.

PREVENT YOUR SERVER FROM CRASHING!

Never again lose customers to poor server speed! Let us help you.

Our server experts will monitor & maintain your server 24/7 so that it remains lightning fast and secure.

GET STARTED

var google_conversion_label = "owonCMyG5nEQ0aD71QM";

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

Privacy Preference Center

Necessary

Necessary cookies help make a website usable by enabling basic functions like page navigation and access to secure areas of the website. The website cannot function properly without these cookies.

PHPSESSID - Preserves user session state across page requests.

gdpr[consent_types] - Used to store user consents.

gdpr[allowed_cookies] - Used to store user allowed cookies.

PHPSESSID, gdpr[consent_types], gdpr[allowed_cookies]
PHPSESSID
WHMCSpKDlPzh2chML

Statistics

Statistic cookies help website owners to understand how visitors interact with websites by collecting and reporting information anonymously.

_ga - Preserves user session state across page requests.

_gat - Used by Google Analytics to throttle request rate

_gid - Registers a unique ID that is used to generate statistical data on how you use the website.

smartlookCookie - Used to collect user device and location information of the site visitors to improve the websites User Experience.

_ga, _gat, _gid
_ga, _gat, _gid
smartlookCookie

Marketing

Marketing cookies are used to track visitors across websites. The intention is to display ads that are relevant and engaging for the individual user and thereby more valuable for publishers and third party advertisers.

IDE - Used by Google DoubleClick to register and report the website user's actions after viewing or clicking one of the advertiser's ads with the purpose of measuring the efficacy of an ad and to present targeted ads to the user.

test_cookie - Used to check if the user's browser supports cookies.

1P_JAR - Google cookie. These cookies are used to collect website statistics and track conversion rates.

NID - Registers a unique ID that identifies a returning user's device. The ID is used for serving ads that are most relevant to the user.

DV - Google ad personalisation

IDE, test_cookie, 1P_JAR, NID, DV, NID
IDE, test_cookie
1P_JAR, NID, DV
NID
hblid

Security

These are essential site cookies, used by the google reCAPTCHA. These cookies use an unique identifier to verify if a visitor is human or a bot.

SID, APISID, HSID, NID, PREF
SID, APISID, HSID, NID, PREF