How a Misconfigured AWS Security Group Took Down Production
The client operated a production application on AWS using an Application Load Balancer, EC2 application servers, Amazon RDS MySQL, an Auto Scaling Group, and a bastion host. The environment had been running reliably with minimal downtime.
During a routine infrastructure update, an inbound rule on the database Security Group was removed before its replacement was validated. This blocked application servers from reaching RDS, causing application failures and a production outage. The original rule was restored, followed by stronger change controls, infrastructure-as-code management, validation, and monitoring.
Business Challenge
A database Security Group change unintentionally blocked MySQL traffic from the application servers. This caused database connection failures, application errors, and customer transaction failures. The client needed to restore production connectivity quickly while reducing the risk of similar configuration-related outages during future infrastructure changes.
Technology Used
Blueprint
Client requests reached the Application Load Balancer and were forwarded to EC2 application servers. The application servers connected to the Amazon RDS MySQL database through Security Group rules. Restoring the required database access rule re-established connectivity, while infrastructure-as-code, validation, monitoring, and change controls were introduced to prevent recurrence.
Key Outcomes
- Restored application-to-database connectivity.
- Recovered production services without database restoration or EC2 replacement.
- Improved control over production Security Group changes.
- Added validation and monitoring for network and application failures.
- Reduced the risk of similar configuration-related outages.
Customer Profile
The client operated a production application hosted entirely on AWS. Its environment included EC2 application servers, an Application Load Balancer, an Amazon RDS MySQL database, an Auto Scaling Group, and a bastion host. Security Groups controlled communication between the infrastructure components. The environment required reliable database connectivity to support application operations and customer transactions.
The Challenges
- The database Security Group's inbound MySQL rule was removed during a routine infrastructure change.
- EC2 application servers could no longer establish connections to Amazon RDS.
- API requests and web pages began returning errors.
- Database connection pools became exhausted.
- New EC2 instances launched by Auto Scaling experienced the same connectivity restriction.
- Customer transactions failed during the outage.
- The configuration change required rapid investigation and rollback.
Why Bobcares?
- Focused troubleshooting on the network access path between the application and database.
- Used CloudTrail, CloudWatch, VPC Flow Logs, and database metrics to isolate the configuration issue.
- Restored the required Security Group rule without unnecessary database or EC2 replacement.
- Applied infrastructure-as-code and validation practices to reduce configuration risk.
- Strengthened production change controls and monitoring around critical infrastructure.
What We Delivered
- Investigated the production connectivity failure.
- Reviewed CloudTrail records to identify the Security Group modification.
- Examined CloudWatch metrics and application failures.
- Reviewed VPC Flow Logs and Security Group configuration.
- Restored the inbound rule permitting TCP port 3306 from the application Security Group.
- Implemented infrastructure-as-code management for Security Groups.
- Added peer review requirements for production network changes.
- Introduced pre-deployment validation for required Security Group rules.
- Added CloudWatch monitoring for database connection failures, application errors, network connectivity, and health checks.
- Validated security hardening changes in staging before production deployment.
Results
Production Connectivity
- Before: EC2 application servers could not establish the required connections to the Amazon RDS database.
- After: Restoring the original Security Group rule re-established database connectivity.
Application Availability
- Before: API requests and web pages were failing, affecting production services and customer transactions.
- After: Application servers reconnected to the database and production services fully recovered.
Infrastructure Recovery
- Before: Auto Scaling launched additional EC2 instances, but the new instances experienced the same database connectivity restriction.
- After: The network configuration was corrected without requiring EC2 replacement or database restoration.
Change Management
- Before: A manual Security Group modification removed a required database access rule before the replacement was validated.
- After: Production network changes were placed under peer review, validation, infrastructure-as-code management, and rollback procedures.
The Business Impact
- Restored production application availability after a configuration-related outage.
- Prevented unnecessary database restoration or EC2 replacement during recovery.
- Reduced operational risk from manual Security Group changes.
- Improved visibility into application, database connectivity, and network failures.
- Established stronger controls for future production infrastructure changes.
Ongoing Collaboration
- Monitor database connection failures and application error rates.
- Monitor network connectivity and health-check failures.
- Validate Security Group changes before production deployment.
- Review infrastructure changes through peer approval.
- Test security hardening changes in staging environments.
- Maintain documented rollback procedures for infrastructure changes.
Conclusion
A single Security Group change disconnected the client's EC2 application servers from Amazon RDS and caused a production outage. Restoring the required database access rule recovered the application without database restoration or EC2 replacement. Infrastructure-as-code, change approval, pre-deployment validation, and enhanced monitoring were then introduced to reduce the risk of similar incidents.
