Learn what caused the October 2025 AWS outage and how DNS failures, service dependencies, and recovery planning affect cloud resilience.
The AWS outage in October 2025 showed how a failure in one service can affect workloads that depend on it. The incident began with DNS resolution failures for Amazon DynamoDB in the US-East-1 region. Problems then spread to other AWS services, including EC2, Lambda, and Network Load Balancer. For businesses running critical applications in the cloud, the incident raises an important question: how well can their systems handle a regional service failure?
An Overview of the October 2025 AWS Outage
The disruption began late on October 19, 2025, and continued into October 20. AWS reported increased DynamoDB API errors in its Northern Virginia region, also known as US-East-1. The incident affected customers and AWS services that depended on DynamoDB.
The disruption developed in three main phases. First, applications could not establish new connections to the regional DynamoDB endpoint. Next, EC2 experienced failures when launching new instances. Network Load Balancer then experienced connection errors because of problems with its health-check system.
AWS reported that all services had returned to normal operations by 2:20 PM PDT on October 20. However, different services recovered at different times. This distinction matters because restoring one dependency does not necessarily restore every system that relies on it.
For the official technical account, read the AWS post-event summary.
What Caused the AWS Outage?
The initial failure came from a defect in DynamoDB’s automated DNS management system. Understanding the sequence helps explain why the incident affected more than a database endpoint.
A race condition disrupted DNS records
DNS translates a service hostname into the IP addresses clients use to connect to it. AWS uses automated systems to maintain DNS records as its services scale, change capacity, and recover from infrastructure problems.
DynamoDB’s DNS management system used separate components to create and apply DNS plans. During the incident, an unusual timing interaction between two DNS Enactors caused an older plan to overwrite a newer one. A cleanup process then deleted that older plan.
This sequence removed the IP addresses from the regional DynamoDB endpoint. The system also entered an inconsistent state that prevented the automation from applying subsequent updates. AWS engineers had to intervene to restore the DNS information.
The failure was not simply a case of DNS being unavailable. It was a race condition in the automation responsible for maintaining the endpoint’s DNS state.
Why the failure affected dependent services
Applications and internal AWS services use DynamoDB for different operations. When the regional endpoint stopped resolving, those systems could not establish new connections through it.
This affected more than customer applications that directly used DynamoDB. Some AWS services relied on DynamoDB for internal workflows, so their operations also failed or slowed down.
The incident illustrates a key reliability risk: a service can become a critical dependency even when customers do not interact with it directly. Architecture reviews need to account for these indirect dependencies, not just the services visible in an application’s design.
How the AWS Outage Spread Across Services
Restoring DynamoDB’s DNS records resolved the primary failure, but it did not immediately restore all affected services. Some internal systems had accumulated backlogs or entered states that required additional recovery work.
EC2 instance launches remained impaired
Existing EC2 instances that had launched before the incident remained healthy. However, customers experienced errors and delays when launching new instances.
An internal EC2 subsystem called DropletWorkflow Manager depended on DynamoDB to maintain state about the physical servers supporting EC2 instances. During the outage, its state checks failed and leases began to expire.
Once DynamoDB recovered, the subsystem attempted to re-establish those leases across a large fleet. The volume of queued work prevented it from making progress quickly enough, creating a congestive collapse.
AWS engineers throttled incoming work and selectively restarted affected hosts. These actions cleared queues and helped the subsystem recover. EC2 launch operations returned to normal later that day.
This sequence shows why recovery planning must account for backlogs and internal state, not just the availability of the original dependency.
Network Load Balancer experienced further errors
The EC2 recovery process also affected network configuration. Newly launched instances needed their network state propagated before they could communicate correctly.
Delays in this process meant that some load balancer health checks failed even when the underlying nodes and backend targets were healthy. The health-check system then removed some capacity from service, causing additional connection errors for affected customers.
AWS temporarily disabled automatic health-check failover to restore available capacity while engineers addressed the underlying issues.
The broader lesson is that automated recovery mechanisms need careful design. A health check should help remove unhealthy capacity, but incorrect or unstable health signals can also remove healthy capacity and worsen an incident.
What the AWS Outage Reveals About Cloud Resilience
The incident provides several practical lessons for teams responsible for production systems.
1. Map critical service dependencies
A cloud architecture diagram should show more than application components and databases. It should also identify dependencies used for identity, DNS, configuration, deployment, scaling, and recovery.
For each critical workflow, identify which services it needs to complete a request. Include dependencies that operate behind managed cloud services when that information is available.
Then ask what happens if each dependency becomes slow, unreachable, or unable to accept new connections. This exercise helps reveal single points of failure and unexpected dependency chains.
2. Distinguish Availability Zones from Regions
Availability Zones help protect workloads against certain localized infrastructure failures. They do not provide complete isolation from every regional service dependency.
A multi-AZ application can still experience problems when it relies on a regional endpoint or a control-plane operation affected by a regional incident. A multi-region design can provide another recovery path, but only if the application and its dependencies support that path.
Review the actual failure boundaries of each AWS service in use. Do not assume that placing compute resources in multiple Availability Zones automatically makes the entire application region-independent.
3. Decide which workloads need multi-region recovery
Running a workload in multiple Regions adds cost and operational complexity. It may require data replication, cross-region routing, identity planning, configuration management, and a process for handling conflicting writes.
Start by identifying the business functions that cannot tolerate a prolonged regional outage. Define recovery time objectives (RTOs) and recovery point objectives (RPOs) for those functions.
Then choose an appropriate recovery strategy. Some applications may need a warm standby in another Region. Others may be able to restore from backups or operate in a reduced-capacity mode.
The goal is not to duplicate every resource. It is to provide a recovery path that meets the application’s actual requirements.
4. Make retry behaviour safe
Retries can help an application recover from temporary errors. However, uncontrolled retries can increase load when a dependency is already struggling.
Use exponential backoff and jitter to spread retry attempts over time. Set limits on the number of attempts, apply timeouts, and use circuit breakers where they suit the application. Avoid retrying operations that are not safe to repeat unless the system supports idempotency.
Retries should also be observable. Track retry counts, timeouts, and downstream error rates so engineers can identify when retry behaviour is contributing to load rather than helping recovery.
These are general resilience practices. They should not be treated as the confirmed cause of the October 2025 AWS incident.
5. Monitor applications independently
Cloud-provider status information is important, but it does not show whether every customer workflow is working as expected.
Use application-level monitoring to track successful transactions, response times, error rates, and critical business operations. Independent checks can help determine whether users can reach the application and complete important tasks.
For critical services, consider monitoring from a separate provider or network path. Keep the monitoring system itself simple enough to remain useful during an incident.
Monitoring should help answer two different questions: is the infrastructure reporting healthy, and can the application actually perform its required work?
6. Test recovery procedures
A documented disaster recovery plan is not proof that recovery will work. Teams need to test whether they can restore data, redirect traffic, rebuild infrastructure, and verify application behaviour under realistic conditions.
Run controlled exercises for regional outages, unavailable dependencies, failed deployments, and delayed data replication. Record the recovery time and any manual steps required.
Also test access to the tools needed during an incident. Recovery can become harder if the same affected services are required to sign in, change infrastructure, or coordinate the response.
A Practical Cloud Resilience Checklist
Use this checklist during an architecture review or disaster recovery exercise.
- Map direct and indirect dependencies for critical application workflows.
- Identify services and endpoints that depend on a single Region.
- Document RTO and RPO targets for each critical workload.
- Confirm how the application behaves when a dependency times out or becomes unavailable.
- Configure bounded retries, exponential backoff, and jitter where appropriate.
- Verify that repeated operations are safe or idempotent.
- Monitor real application transactions, not only infrastructure status.
- Test backup restoration and regional recovery procedures.
- Check whether identity, DNS, deployment, and monitoring tools remain accessible during an incident.
- Record test results, recovery times, and unresolved failure scenarios.
The checklist should lead to specific actions. If a critical service has no tested recovery path, assign an owner and define the next exercise rather than treating the review as complete.
The Cost of Resilience and the Limits of Redundancy
Redundancy can reduce the impact of some failures, but it does not eliminate every dependency. Two application instances may still rely on the same regional database endpoint. A standby Region may also be ineffective if it lacks current data, valid credentials, working DNS, or sufficient capacity.
Multi-region infrastructure also requires ongoing testing and maintenance. Replication can introduce lag, failover can create conflicting writes, and standby resources can become outdated. These risks need to be addressed as part of the design.
For some workloads, tested backups and a reliable restoration process may be sufficient. Others may require automated failover and a second Region. The right choice depends on business impact, recovery targets, technical dependencies, and cost.
For more on planning recovery, see Bobcares’ guide to cloud disaster recovery strategies.
Frequently Asked Questions
What caused the October 2025 AWS outage?
A race condition in DynamoDB’s automated DNS management system removed the IP addresses from its regional endpoint in US-East-1. This prevented new connections to DynamoDB and affected dependent AWS services.
Which AWS services were affected?
The incident affected DynamoDB and contributed to problems with EC2 instance launches, Network Load Balancer, Lambda, and other services that depended on the affected systems. The impact and recovery time differed by service.
Did the outage affect every AWS Region?
The initial DynamoDB endpoint failure occurred in US-East-1. However, some features and services outside that Region were affected because they depended on endpoints or identity operations associated with US-East-1. Customers with DynamoDB Global Tables could access replica tables in other Regions, although replication lag affected synchronization with the impaired Region.
Does deploying across multiple Availability Zones prevent regional outages?
No. Multiple Availability Zones can reduce exposure to localized failures, but they do not protect against every regional service dependency. Applications that require regional recovery need a tested strategy for their data, routing, identity, and application components.
How can businesses prepare for another AWS outage?
Map critical dependencies, define recovery targets, monitor application-level health, control retries, and test recovery procedures. Consider multi-region deployment when the business impact of a regional outage justifies the added cost and complexity.
Conclusion
The October 2025 AWS outage began with a DNS automation defect in DynamoDB and developed into a broader disruption involving dependent services. Restoring the original endpoint did not immediately restore every affected system. EC2 launch backlogs and Network Load Balancer health-check problems required additional recovery work.
For businesses, the central lesson is to design and test for dependency failure. Multi-AZ deployment, multi-region recovery, independent monitoring, and controlled retries can all help, but each must fit the application’s real failure modes.
Cloud resilience depends on more than choosing a major provider. It requires knowing what your systems depend on, defining acceptable recovery targets, and proving that the recovery process works.
