Need help?

Our experts have had an average response time of 11.7 minutes in August 2021 to fix urgent issues.

We will keep your servers stable, secure, and fast at all times for one fixed price.

Unified CloudWatch agent not pushing log events

by | Aug 24, 2021

Wondering why unified CloudWatch agent not pushing log events? We can help you.

Here, at Bobcares, we assist our customers with several AWS queries as part of our AWS Support Services.

Today, let us see how our Support Techs assist with this CloudWatch issue.

 

How to resolve unified CloudWatch agent not pushing log events?

Below are some of the reasons that prevent the unified CloudWatch agent from pushing log events:

  • Firstly, out-of-sync metadata caused by creating an Amazon Machine Image (AMI) after the CloudWatch agent is installed
  • Then, using an outdated version of the CloudWatch agent
  • Failure to connect to the CloudWatch Logs endpoint
  • Incorrect account, Region, or log group configurations
  • Insufficient AWS Identity and Access Management (IAM) permissions
  • CloudWatch agent run errors
  • Timestamp issues

 

Today, let us see the steps followed by our Support Techs to resolve this issue:

Review the installation method for the CloudWatch agent

It is better to install the CloudWatch agent at launch using AWS CloudFormation, AWS Systems Manager Agent (SSM Agent), user data scripts, or the AWS CLI.

Creating an AMI with the CloudWatch agent already installed isn’t recommended.

Typically, AMIs capture unique information from the original instance.

Metadata becomes out of sync, and this state can lead to the CloudWatch agent not working as intended.

Out-of-sync metadata is the reason that many Windows instances require Sysprep when working with AMI.

 

Confirm that you’re using the latest version of the CloudWatch agent

Download and review the README files for the CloudWatch agent release notes and latest version number.

If you’re using an older version of the CloudWatch agent, be sure to upgrade.

The latest version might include updates that resolve the issue that you’re experiencing.

 

Test connectivity to your CloudWatch Logs endpoint

Test connectivity to the CloudWatch Logs endpoint using either of the following commands:

telnet logs.<region>.amazonaws.com 443
nc -zv logs.<region>.amazonaws.com 443

If you encounter connectivity failures, be sure that:

  • Firstly, the security group and network access control list (ACL) rules allow connectivity.
  • Then, your instance can reach the public endpoint using an internet gateway or a network address translation (NAT) gateway.
  • If you’re using VPC endpoints, the endpoint resolves to a VPC IP and the endpoint security group allows access from the source instance.

 

Review your account, Region, and log group configurations

In the CloudWatch agent configuration file:

  • Firstly, be sure that the specified Region matches the console Region
  • Then, verify that logs are checked in the correct account

Optionally, you can use the common-config.toml file to override system defaults for the CloudWatch agent.

These system defaults include the proxy, Region, and credential information for the agent.

The file is available in the following locations.

Linux:

/opt/aws/amazon-cloudwatch-agent/etc/common-config.toml

-OR-

/etc/amazon/amazon-cloudwatch-agent/common-config.toml

Windows:

$Env:ProgramData\Amazon\AmazonCloudWatchAgent\common-config.toml

 

Check your IAM permissions

The CloudWatch agent uses credentials from either the IAM user or IAM role policy to push log events to the CloudWatch service.

Before a log event can be published, you must create a log group and log stream.

If there’s no log group or log stream, the CloudWatch agent creates them.

Confirm that your policy includes the following IAM permissions:

"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:DescribeLogStreams"

Add any missing IAM permissions to the user policy or the role policy.

Please note when creating IAM roles and users, it’s a best practice to use the CloudWatchAgentServerPolicy and CloudWatchAgentAdminPolicy policies created by Amazon rather than custom policies.

 

Resolve CloudWatch agent run errors

Verify that the CloudWatch agent is running. If the agent isn’t running, check the log file for errors and resolve them.

Log files are located in the following locations.

Linux:

/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

Windows:

$Env:ProgramData\Amazon\AmazonCloudWatchAgent\Logs\amazon-cloudwatch-agent.log

Please note logs might be specified in a custom logfile location.

Check the agent configuration file to identify any custom log locations.

In the agent configuration file, enable verbose debug logging using the debug parameter.

If you’re using the run_as_user parameter, confirm that the user has permissions to the log location path.

Without the necessary permissions, the CloudWatch agent can’t write logs to the location.

 

Resolve timestamp issues

Firstly, check for log event timestamps that are older than 14 days or more than two hours in the future.

The PutLogEvents command doesn’t allow log batches in either time frame.

Also, verify that the system time service on the instance is correctly configured.

 

[Need help with the procedures? We’d be happy to assist you]

 

Conclusion

In short, we saw troubleshooting steps followed by our Support Techs to resolve unified CloudWatch agent not pushing log events.

PREVENT YOUR SERVER FROM CRASHING!

Never again lose customers to poor server speed! Let us help you.

Our server experts will monitor & maintain your server 24/7 so that it remains lightning fast and secure.

GET STARTED

var google_conversion_label = "owonCMyG5nEQ0aD71QM";

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

Privacy Preference Center

Necessary

Necessary cookies help make a website usable by enabling basic functions like page navigation and access to secure areas of the website. The website cannot function properly without these cookies.

PHPSESSID - Preserves user session state across page requests.

gdpr[consent_types] - Used to store user consents.

gdpr[allowed_cookies] - Used to store user allowed cookies.

PHPSESSID, gdpr[consent_types], gdpr[allowed_cookies]
PHPSESSID
WHMCSpKDlPzh2chML

Statistics

Statistic cookies help website owners to understand how visitors interact with websites by collecting and reporting information anonymously.

_ga - Preserves user session state across page requests.

_gat - Used by Google Analytics to throttle request rate

_gid - Registers a unique ID that is used to generate statistical data on how you use the website.

smartlookCookie - Used to collect user device and location information of the site visitors to improve the websites User Experience.

_ga, _gat, _gid
_ga, _gat, _gid
smartlookCookie

Marketing

Marketing cookies are used to track visitors across websites. The intention is to display ads that are relevant and engaging for the individual user and thereby more valuable for publishers and third party advertisers.

IDE - Used by Google DoubleClick to register and report the website user's actions after viewing or clicking one of the advertiser's ads with the purpose of measuring the efficacy of an ad and to present targeted ads to the user.

test_cookie - Used to check if the user's browser supports cookies.

1P_JAR - Google cookie. These cookies are used to collect website statistics and track conversion rates.

NID - Registers a unique ID that identifies a returning user's device. The ID is used for serving ads that are most relevant to the user.

DV - Google ad personalisation

IDE, test_cookie, 1P_JAR, NID, DV, NID
IDE, test_cookie
1P_JAR, NID, DV
NID
hblid

Security

These are essential site cookies, used by the google reCAPTCHA. These cookies use an unique identifier to verify if a visitor is human or a bot.

SID, APISID, HSID, NID, PREF
SID, APISID, HSID, NID, PREF