AWS CloudWatch Logs Cost Optimization Through Automated Lifecycle Management
The client had 42 accounts on AWS that created 3,847 CloudWatch Log Groups and were logging roughly 48 TB of logs per month. Unmanaged retention for years had grown the amount of stored log data to approximately 680 TB, driving up storage costs and making many logs unoperational.
Bobcares implemented a tiered retention framework with automated enforcement using EventBridge and Lambda, supported by AWS Config compliance monitoring and FinOps cost governance. The solution applied retention policies according to environment and log purpose while providing continuous controls for newly created log groups.
Business Challenge
Uncontrolled CloudWatch Logs retention was driving substantial storage costs across the client's AWS organization. Most log groups had no expiration policy, while orphaned and duplicated logs continued accumulating. The client needed consistent retention standards, automated enforcement, and ongoing cost governance without compromising operational or compliance requirements.
Technology Used
Blueprint
Applications and AWS services published logs to CloudWatch Logs, where log groups were classified into retention tiers. EventBridge detected new log groups and invoked Lambda to apply the appropriate policy. AWS Config continuously checked compliance, while S3 provided long-term archival for selected logs and Cost Explorer supported ongoing FinOps monitoring.
Key Outcomes
- Reduced monthly CloudWatch Logs cost from $467,000 to $106,520.
- Reduced stored log volume from 680 TB to approximately 55 TB.
- Applied automated retention policies across 3,847 log groups.
- Removed unmanaged Never Expire retention from 2,999 log groups.
- Established continuous retention compliance monitoring with AWS Config.
- Automated retention enforcement for newly created log groups.
- Introduced structured cost attribution and FinOps governance.
Customer Profile
The client had a large and established AWS environment with 42 accounts and 6 organizational units. CloudWatch Logs was the out-of-the-box observability and audit logging platform for workloads running on EC2, ECS, EKS, Lambda, API Gateway, RDS, VPC Flow Logs, and CloudTrail. Logging infrastructure had been growing for several years, without any centralised lifecycle management in place. The client wanted to reduce the costs of their logs and yet retain the logs for operational and compliance requirements.
The Challenges
- 77.9% of log groups had no retention policy.
- Approximately 680 TB of log data had accumulated under uncapped retention.
- 1,016 log groups were identified as orphaned.
- Logs from retired resources continued generating storage costs.
- Some workloads stored logs in both CloudWatch Logs and S3.
- High-volume Lambda and container workloads generated substantial log volumes.
- Development environments produced high-volume DEBUG-level logs.
- There was no organization-wide retention standard.
- Log groups lacked consistent ownership, application, and cost-center tags.
- No preventive control stopped new log groups from being created without retention policies.
Why Bobcares?
- Assessed the complete CloudWatch Logs estate before applying retention changes.
- Used cost attribution and log inventory data to identify the primary cost drivers.
- Designed retention tiers around operational value, environment, and compliance requirements.
- Combined preventive automation with continuous compliance monitoring.
- Worked with the existing AWS Organizations structure instead of requiring broad infrastructure changes.
- Established governance controls to prevent the retention problem from recurring.
What We Delivered
- Audited CloudWatch Log Groups across all 42 AWS accounts.
- Established a five-tier retention framework covering 14, 30, 90, 180, and 365 days.
- Classified existing log groups according to environment and naming patterns.
- Deployed a Lambda-based retention policy enforcement mechanism.
- Configured EventBridge to trigger enforcement when new log groups were created.
- Applied bulk retention remediation to existing log groups.
- Implemented AWS Config compliance evaluation and remediation.
- Established structured log group naming and tagging standards.
- Configured S3 archival for selected long-term retention requirements.
- Added SNS notifications for retention enforcement and compliance events.
- Established ongoing cost monitoring through Cost Explorer and AWS Budgets.
Results
CloudWatch Logs Cost
- Before: Monthly CloudWatch Logs cost was $467,000.
- After: Monthly cost decreased to $106,520, resulting in $360,480 in monthly savings.
Stored Log Volume
- Before: Approximately 680 TB of log data was stored with uncapped retention.
- After: Steady-state stored volume decreased to approximately 55 TB, representing a 92% reduction.
Retention Governance
- Before: 2,999 of 3,847 log groups had no retention policy.
- After: Tiered retention policies were applied according to environment, operational value, and compliance requirements.
Retention Enforcement
- Before: Teams created log groups without a centralized mechanism to apply lifecycle policies.
- After: EventBridge and Lambda automatically applied retention policies to newly created log groups.
Compliance Monitoring
- Before: There was no organization-wide automated retention compliance control.
- After: AWS Config continuously evaluated retention compliance and supported automated remediation.
FinOps Visibility
- Before: CloudWatch Logs costs lacked consistent ownership and cost attribution.
- After: Cost monitoring, tagging standards, account-level analysis, and budget alerts provided an ongoing FinOps governance framework.
The Business Impact
- Reduced the financial impact of long-term CloudWatch Logs storage.
- Lowered infrastructure pressure caused by unnecessary historical log retention.
- Reduced the administrative effort required to maintain retention policies.
- Improved visibility into CloudWatch Logs spending across accounts and business units.
- Reduced the volume of data scanned by CloudWatch Logs Insights queries.
- Established a consistent approach to balancing operational access with retention costs.
- Reduced the risk of future cost growth caused by unmanaged log group creation.
- Improved governance over logs with operational, audit, and compliance value.
Ongoing Collaboration
- Monitor CloudWatch Logs costs through monthly FinOps reviews.
- Review cost anomalies and retention tier classifications.
- Maintain AWS Budgets alerts for CloudWatch Logs spending.
- Review orphaned log groups periodically.
- Check retention compliance through AWS Config.
- Assess high-volume logging sources for more cost-efficient storage or delivery options.
- Review retention requirements to balance operational needs, compliance, and storage costs.
Conclusion
The client's CloudWatch Logs environment had accumulated substantial storage costs because retention was largely unmanaged across its 42 AWS accounts. Bobcares added tiered retention policies, automated enforcement, compliance monitoring and archival controls, as well as FinOps governance. These changes reduced monthly CloudWatch Logs costs from $467,000 to $106,520 and reduced stored log volume from approximately 680 TB to 55 TB.
