CloudWatch logs fails to export to S3 buckets? We can help you.

Recently, one of our customers tried to export Amazon CloudWatch log data to Amazon S3 buckets. However, the export task failed.

Here, at Bobcares, we assist our customers with several AWS queries as part of our AWS Support Services.

Today, let us see why the CloudWatch logs fail to export to S3 buckets.

CloudWatch logs fails to export to S3 buckets

Our Support Techs recommend the following to troubleshoot tasks that fail during creation:

• Region

We need to confirm that the CloudWatch Logs log streams and S3 buckets are in the same Region.

• S3 bucket policies

Since all S3 buckets and objects are private, only the resource owner can access them.

However, the resource owner can grant access permissions to other resources and users by writing an access policy.

To do so, in the Amazon S3 console, we select the bucket.

Then we go to Permissions > Bucket policy.

Here, we add one of the following policies.

If the bucket is in our account, we add:

{
"Version": "2012-10-17",
"Statement": [
{
"Action": "s3:GetBucketAcl",
"Effect": "Allow",
"Resource": "arn:aws:s3:::my-exported-logs", "Principal": { "Service": "logs.us-west-2.amazonaws.com" } }, { "Action": "s3:PutObject" , "Effect": "Allow", "Resource": "arn:aws:s3:::my-exported-logs/random-string/*", "Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control" } }, "Principal": { "Service": "logs.us-west-2.amazonaws.com" } } ] }

If the bucket is in a different account:

{
"Version": "2012-10-17",
"Statement": [
{
"Action": "s3:GetBucketAcl",
"Effect": "Allow",
"Resource": "arn:aws:s3:::my-exported-logs", "Principal": { "Service": "logs.us-west-2.amazonaws.com" } }, { "Action": "s3:PutObject" , "Effect": "Allow", "Resource": "arn:aws:s3:::my-exported-logs/random-string/*", "Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control" } }, "Principal": { "Service": "logs.us-west-2.amazonaws.com" } }, { "Action": "s3:PutObject" , "Effect": "Allow", "Resource": "arn:aws:s3:::my-exported-logs/random-string/*", "Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control" } }, "Principal": { "AWS": "arn:aws:iam::SendingAccountID:user/CWLExportUser" } } ] }

Eventually, we select Save to set the policy. It enables CloudWatch Logs to export log data to the Amazon S3 bucket.

• S3 bucket prefixes

When we set the policy, our Support Techs recommend including a random string as the prefix for the bucket. Hence, the only intended log streams export to the bucket.

If we use a prefix, we must also specify it in the S3 bucket prefix settings when we create the export task. Otherwise, the export task creation fails.

• AWS Identity and Access Management (IAM) policies

Confirm that the IAM user (IAM role) who created the export task has full access to Amazon S3 and CloudWatch Logs.

To do so, our Support Techs suggest the following steps:

1. Initially, we open the IAM console.
2. Then we go to Users > Add Users.
3. Once we enter a user name, we select both Programmatic access and AWS Management Console access.
4. We need to select either Autogenerated password or a Custom password.
5. Then we select Next: Permissions.
6. Here, we select Attach existing policies directly, and attach the AmazonS3FullAccess and CloudWatchLogsFullAccess policies to the user.
7. After that, we select Next: Tags, Next: Review, and then Create user.

• Resource quotas

There are CloudWatch Logs service quotas. It restricts the number of running or pending export tasks per account per Region.

We need to make sure that we operate within the allowed quotas.

• Type of server-side encryption

Similarly, we need to make sure we use a supported type of server-side encryption.

We cannot export to S3 buckets encrypted with SSE-KMS. However, it supports exporting to S3 buckets encrypted with AES-256.

To troubleshoot tasks that fail after creation, we check the Time Range setting.

Export of log streams with large amounts of data and a long time range might fail.

In this case, we specify a shorter time range.

[Stuck with the error? We are here for you]

Conclusion

In short, we saw how our Support Techs fix the export error.