Need help?

Our experts have had an average response time of 11.7 minutes in August 2021 to fix urgent issues.

We will keep your servers stable, secure, and fast at all times for one fixed price.

CloudWatch logs fails to export to S3 buckets – Let’s Troubleshoot

by | Jul 25, 2021

CloudWatch logs fails to export to S3 buckets? We can help you.

Recently, one of our customers tried to export Amazon CloudWatch log data to Amazon S3 buckets. However, the export task failed.

Here, at Bobcares, we assist our customers with several AWS queries as part of our AWS Support Services.

Today, let us see why the CloudWatch logs fail to export to S3 buckets.

 

CloudWatch logs fails to export to S3 buckets

Our Support Techs recommend the following to troubleshoot tasks that fail during creation:

  • Region

We need to confirm that the CloudWatch Logs log streams and S3 buckets are in the same Region.

  • S3 bucket policies

Since all S3 buckets and objects are private, only the resource owner can access them.

However, the resource owner can grant access permissions to other resources and users by writing an access policy.

To do so, in the Amazon S3 console, we select the bucket.

Then we go to Permissions > Bucket policy.

Here, we add one of the following policies.

If the bucket is in our account, we add:

{
"Version": "2012-10-17",
"Statement": [
{
"Action": "s3:GetBucketAcl",
"Effect": "Allow",
"Resource": "arn:aws:s3:::my-exported-logs", "Principal": { "Service": "logs.us-west-2.amazonaws.com" } }, { "Action": "s3:PutObject" , "Effect": "Allow", "Resource": "arn:aws:s3:::my-exported-logs/random-string/*", "Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control" } }, "Principal": { "Service": "logs.us-west-2.amazonaws.com" } } ] }

If the bucket is in a different account:

{
"Version": "2012-10-17",
"Statement": [
{
"Action": "s3:GetBucketAcl",
"Effect": "Allow",
"Resource": "arn:aws:s3:::my-exported-logs", "Principal": { "Service": "logs.us-west-2.amazonaws.com" } }, { "Action": "s3:PutObject" , "Effect": "Allow", "Resource": "arn:aws:s3:::my-exported-logs/random-string/*", "Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control" } }, "Principal": { "Service": "logs.us-west-2.amazonaws.com" } }, { "Action": "s3:PutObject" , "Effect": "Allow", "Resource": "arn:aws:s3:::my-exported-logs/random-string/*", "Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control" } }, "Principal": { "AWS": "arn:aws:iam::SendingAccountID:user/CWLExportUser" } } ] }

Eventually, we select Save to set the policy. It enables CloudWatch Logs to export log data to the Amazon S3 bucket.

  • S3 bucket prefixes

When we set the policy, our Support Techs recommend including a random string as the prefix for the bucket. Hence, the only intended log streams export to the bucket.

If we use a prefix, we must also specify it in the S3 bucket prefix settings when we create the export task. Otherwise, the export task creation fails.

  • AWS Identity and Access Management (IAM) policies

Confirm that the IAM user (IAM role) who created the export task has full access to Amazon S3 and CloudWatch Logs.

To do so, our Support Techs suggest the following steps:

  1. Initially, we open the IAM console.
  2. Then we go to Users > Add Users.
  3. Once we enter a user name, we select both Programmatic access and AWS Management Console access.
  4. We need to select either Autogenerated password or a Custom password.
  5. Then we select Next: Permissions.
  6. Here, we select Attach existing policies directly, and attach the AmazonS3FullAccess and CloudWatchLogsFullAccess policies to the user.
  7. After that, we select Next: Tags, Next: Review, and then Create user.
  • Resource quotas

There are CloudWatch Logs service quotas. It restricts the number of running or pending export tasks per account per Region.

We need to make sure that we operate within the allowed quotas.

  • Type of server-side encryption

Similarly, we need to make sure we use a supported type of server-side encryption.

We cannot export to S3 buckets encrypted with SSE-KMS. However, it supports exporting to S3 buckets encrypted with AES-256.

To troubleshoot tasks that fail after creation, we check the Time Range setting.

Export of log streams with large amounts of data and a long time range might fail.

In this case, we specify a shorter time range.

[Stuck with the error? We are here for you]

 

Conclusion

In short, we saw how our Support Techs fix the export error.

PREVENT YOUR SERVER FROM CRASHING!

Never again lose customers to poor server speed! Let us help you.

Our server experts will monitor & maintain your server 24/7 so that it remains lightning fast and secure.

GET STARTED

var google_conversion_label = "owonCMyG5nEQ0aD71QM";

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

Privacy Preference Center

Necessary

Necessary cookies help make a website usable by enabling basic functions like page navigation and access to secure areas of the website. The website cannot function properly without these cookies.

PHPSESSID - Preserves user session state across page requests.

gdpr[consent_types] - Used to store user consents.

gdpr[allowed_cookies] - Used to store user allowed cookies.

PHPSESSID, gdpr[consent_types], gdpr[allowed_cookies]
PHPSESSID
WHMCSpKDlPzh2chML

Statistics

Statistic cookies help website owners to understand how visitors interact with websites by collecting and reporting information anonymously.

_ga - Preserves user session state across page requests.

_gat - Used by Google Analytics to throttle request rate

_gid - Registers a unique ID that is used to generate statistical data on how you use the website.

smartlookCookie - Used to collect user device and location information of the site visitors to improve the websites User Experience.

_ga, _gat, _gid
_ga, _gat, _gid
smartlookCookie

Marketing

Marketing cookies are used to track visitors across websites. The intention is to display ads that are relevant and engaging for the individual user and thereby more valuable for publishers and third party advertisers.

IDE - Used by Google DoubleClick to register and report the website user's actions after viewing or clicking one of the advertiser's ads with the purpose of measuring the efficacy of an ad and to present targeted ads to the user.

test_cookie - Used to check if the user's browser supports cookies.

1P_JAR - Google cookie. These cookies are used to collect website statistics and track conversion rates.

NID - Registers a unique ID that identifies a returning user's device. The ID is used for serving ads that are most relevant to the user.

DV - Google ad personalisation

IDE, test_cookie, 1P_JAR, NID, DV, NID
IDE, test_cookie
1P_JAR, NID, DV
NID
hblid

Security

These are essential site cookies, used by the google reCAPTCHA. These cookies use an unique identifier to verify if a visitor is human or a bot.

SID, APISID, HSID, NID, PREF
SID, APISID, HSID, NID, PREF