Wondering why unified CloudWatch agent not pushing log events? We can help you.

Here, at Bobcares, we assist our customers with several AWS queries as part of our AWS Support Services.

Today, let us see how our Support Techs assist with this CloudWatch issue.

How to resolve unified CloudWatch agent not pushing log events?

Below are some of the reasons that prevent the unified CloudWatch agent from pushing log events:

• Firstly, out-of-sync metadata caused by creating an Amazon Machine Image (AMI) after the CloudWatch agent is installed
• Then, using an outdated version of the CloudWatch agent
• Failure to connect to the CloudWatch Logs endpoint
• Incorrect account, Region, or log group configurations
• Insufficient AWS Identity and Access Management (IAM) permissions
• CloudWatch agent run errors
• Timestamp issues

Today, let us see the steps followed by our Support Techs to resolve this issue:

Review the installation method for the CloudWatch agent

It is better to install the CloudWatch agent at launch using AWS CloudFormation, AWS Systems Manager Agent (SSM Agent), user data scripts, or the AWS CLI.

Creating an AMI with the CloudWatch agent already installed isn’t recommended.

Typically, AMIs capture unique information from the original instance.

Metadata becomes out of sync, and this state can lead to the CloudWatch agent not working as intended.

Out-of-sync metadata is the reason that many Windows instances require Sysprep when working with AMI.

Confirm that you’re using the latest version of the CloudWatch agent

Download and review the README files for the CloudWatch agent release notes and latest version number.

If you’re using an older version of the CloudWatch agent, be sure to upgrade.

The latest version might include updates that resolve the issue that you’re experiencing.

Test connectivity to your CloudWatch Logs endpoint

Test connectivity to the CloudWatch Logs endpoint using either of the following commands:

telnet logs.<region>.amazonaws.com 443

nc -zv logs.<region>.amazonaws.com 443

If you encounter connectivity failures, be sure that:

• Firstly, the security group and network access control list (ACL) rules allow connectivity.
• Then, your instance can reach the public endpoint using an internet gateway or a network address translation (NAT) gateway.
• If you’re using VPC endpoints, the endpoint resolves to a VPC IP and the endpoint security group allows access from the source instance.

Review your account, Region, and log group configurations

In the CloudWatch agent configuration file:

• Firstly, be sure that the specified Region matches the console Region
• Then, verify that logs are checked in the correct account

Optionally, you can use the common-config.toml file to override system defaults for the CloudWatch agent.

These system defaults include the proxy, Region, and credential information for the agent.

The file is available in the following locations.

Linux:

/opt/aws/amazon-cloudwatch-agent/etc/common-config.toml

-OR-

/etc/amazon/amazon-cloudwatch-agent/common-config.toml

Windows:

$Env:ProgramData\Amazon\AmazonCloudWatchAgent\common-config.toml

Check your IAM permissions

The CloudWatch agent uses credentials from either the IAM user or IAM role policy to push log events to the CloudWatch service.

Before a log event can be published, you must create a log group and log stream.

If there’s no log group or log stream, the CloudWatch agent creates them.

Confirm that your policy includes the following IAM permissions:

"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:DescribeLogStreams"

Add any missing IAM permissions to the user policy or the role policy.

Please note when creating IAM roles and users, it’s a best practice to use the CloudWatchAgentServerPolicy and CloudWatchAgentAdminPolicy policies created by Amazon rather than custom policies.

Resolve CloudWatch agent run errors

Verify that the CloudWatch agent is running. If the agent isn’t running, check the log file for errors and resolve them.

Log files are located in the following locations.

Linux:

/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

Windows:

$Env:ProgramData\Amazon\AmazonCloudWatchAgent\Logs\amazon-cloudwatch-agent.log

Please note logs might be specified in a custom logfile location.

Check the agent configuration file to identify any custom log locations.

In the agent configuration file, enable verbose debug logging using the debug parameter.

If you’re using the run_as_user parameter, confirm that the user has permissions to the log location path.

Without the necessary permissions, the CloudWatch agent can’t write logs to the location.

Resolve timestamp issues

[Need help with the procedures? We’d be happy to assist you]

Conclusion

In short, we saw troubleshooting steps followed by our Support Techs to resolve unified CloudWatch agent not pushing log events.