8:11 PM PDT We are investigating degraded performance for some EBS volumes in a single Availability Zone (USE1-AZ2) in the US-EAST-1 Region. Some new EC2 instance launches, within the affected Availability Zone, are also impacted by this issue. We are working to resolve the issue.
8:41 PM PDT We can confirm degraded performance for some EBS volumes within a single Availability Zone (USE1-AZ2) in the US-EAST-1 Region. Existing EC2 instances within the affected Availability Zone that use EBS volumes may also experience impairment due to stuck IO to the attached EBS volume(s). Newly launched EC2 instances within the affected Availability Zone may fail to launch due to the degraded volume performance. We continue to work toward determining root cause and mitigating impact but recommend that you fail out of the affected Availability Zone (USE1-AZ2) if you are able to do so. Other Availability Zones within the US-EAST-1 Region are not affected by this issue.
9:17 PM PDT We are making progress in determining the root cause and have isolated it to a subsystem within the EBS service. We are working through multiple steps to mitigate the issue and will continue to provide updates as we make progress. Other Availability Zones remain unaffected by this issue and affected EBS volumes and EC2 instances within the affected Availability Zone have plateaued at this stage. We continue to recommend that you fail out of the affected Availability Zone (USE1-AZ2) if you are able to do so.
9:47 PM PDT We continue to make progress in determining the root cause of the issue causing degraded performance for some EBS volumes in a single Availability Zone (USE1-AZ2) in the US-EAST-1 Region. A subsystem within the larger EBS service that is responsible for coordinating storage hosts is currently degraded due to increased resource contention. We continue to work to understand the root cause of the elevated resource contention, but are actively working to mitigate the issue. Once mitigated, we expect performance for the affected EBS volumes to return to normal levels. We will continue to provide you with updates on our progress. For immediate recovery, we continue to recommend that you fail out of the affected Availability Zone (USE1-AZ2) if you are able to do so.
10:23 PM PDT We continue to make progress in determining the root cause of the issue causing degraded performance for some EBS volumes in a single Availability Zone (USE1-AZ2) in the US-EAST-1 Region. We have made several changes to address the increased resource contention within the subsystem responsible for coordinating storage hosts with the EBS service. While these changes have led to some improvement, we have not yet seen full recovery for the affected EBS volumes. We continue to expect full recovery of the affected EBS volumes once the subsystem issue has been addressed. For immediate recovery, we continue to recommend that you fail out of the affected Availability Zone (USE1-AZ2) if you are able to do so.
11:19 PM PDT We continue to work to address the increased resource contention within the EBS service subsystem responsible for coordinating EBS storage hosts. We have applied several mitigations, and while we have seen some improvements, we have not yet seen performance for affected volumes return to normal levels. We are currently rolling out another mitigation that we believe addresses the root cause and has shown promising signs in testing. We will know within the next 30-45 minutes as to whether this restores normal operations for current affected volumes. For immediate recovery, we continue to recommend that you fail out of the affected Availability Zone (USE1-AZ2) if you are able to do so.
11:43 PM PDT We can confirm that the deployed mitigation has worked and we have started to see recovery for some affected EBS volumes within the affected Availability Zone (USE1-AZ2). We are still finishing the deployment of the mitigation, but expect performance of affected EBS volumes in this single Availability Zone to return to normal levels over the next 60 minutes.
Sep 27, 12:30 AM PDT We can confirm that the performance has returned to normal levels for the majority of the affected EBS volumes within the affected Availability Zone (USE1-AZ2). We continue to work towards full recovery for the remaining EBS volumes.
Sep 27, 1:15 AM PDT We can confirm that the performance has returned to normal levels for the majority of the affected EBS volumes within the affected Availability Zone (USE1-AZ2). Starting at 12:12 AM PDT, we saw recovery slow down some affected EBS volumes as well as some degraded performance for a small number of additional volumes in the affected Availability Zone. We have investigated the root cause and mitigations are underway to complete the performance recovery of the affected EBS volumes. We continue to work towards full resolution for all affected EBS volumes. In some cases, customers may be experiencing volume state transition delays, which we expect to clear up once volumes have fully recovered.
Sep 27, 2:26 AM PDT We continue to apply mitigations to address the degraded performance for the smaller set of remaining EBS volumes affected by this issue. While the vast majority of affected EBS volumes were operating normally by 12:05 AM PDT, we have been working to recover a smaller set of affected volumes since 12:12 AM PDT. We continue to make progress on restoring performance for affected volumes, and expect full recovery to take another 2 hours. In some cases, customers may also be experiencing volume state transition delays, which will resolve when the underlying volume has fully recovered.
Sep 27, 3:36 AM PDT We had restored performance for the vast majority of affected EBS volumes within the affected Availability Zone in the US-EAST-1 Region at 12:05 AM PDT and have been working to restore a remaining smaller set of EBS volumes. EC2 instances affected by this issue have now also recovered and new EC2 instance launches with attached EBS volumes have been succeeding since 1:30 AM PDT. Other services - including Redshift, OpenSearch, and Elasticache - are seeing recovery. Some RDS databases are still experiencing connectivity issues, but we’re working towards full recovery. We are in the process of restoring performance for the remaining small number of EBS volumes and EC2 instances that are still affected by this issue.
Sep 27, 4:21 AM PDT Starting at 6:41 PM PDT on September 26th, we experienced degraded performance for some EBS volumes in a single Availability Zone (USE1-AZ2) in the US-EAST-1 Region. The issue was caused by increased resource contention within the EBS subsystem responsible for coordinating EBS storage hosts. Engineering worked to identify the root cause and resolve the issue within the affected subsystem. At 11:20 PM PDT, after deploying an update to the affected subsystem, IO performance for the affected EBS volumes began to return to normal levels. By 12:05 AM on September 27th, IO performance for the vast majority of affected EBS volumes in the USE1-AZ2 Availability Zone were operating normally. However, starting at 12:12 AM PDT, we saw recovery slow down for a smaller set of affected EBS volumes as well as seeing degraded performance for a small number of additional volumes in the USE1-AZ2 Availability Zone. Engineering investigated the root cause and put in place mitigations to restore performance for the smaller set of remaining affected EBS volumes. These mitigations slowly improved the performance for the remaining smaller set of affected EBS volumes, with full operations restored by 3:45 AM PDT. While almost all of EBS volumes have fully recovered, we continue to work on recovering a remaining small set of EBS volumes. We will communicate the recovery status of these volumes via the Personal Health Dashboard. While the majority of affected services have fully recovered, we continue to recover some services, including RDS databases and Elasticache clusters. We will also communicate the recovery status of these services via the Personal Health Dashboard. The issue has been fully resolved and the service is operating normally.