AWS updated us at 1:17 ET to report this issue as resolved, and that the service is operating normally: "Between 8:20 AM and 9:57 AM PDT, we experienced elevated packet loss, impacting AWS Direct Connect connectivity to a single Availability Zone (use1-az6) in the US-EAST-1 Region."
We have now re-enabled our servers in the affected AZ and are no longer seeing any errors.
Lastly, our job processing latency has returned to normal, and we are considering this incident resolved.
Posted Aug 22, 2020 - 13:30 EDT
Update
AWS has identified the root cause of increased packet loss, impacting AWS Direct Connect connectivity to a single Availability Zone (AZ) (use1-az6) in the US-EAST-1 Region.
We have been able to successfully rebalance our traffic and job processing to other, non-affected AZs, and have brought additional servers online in those AZs, to keep our performance consistent.
We are no longer seeing any errors, and our job processing latency is trending down.
We'll provide another update once AWS provides us one, or once job processing latency returns to normal.
Posted Aug 22, 2020 - 13:16 EDT
Identified
We have identified an issue affecting one availability zone for US clusters hosted on Amazon Web Services. This issue is causing degraded performance for data processing and message sending for 1/3 of traffic in US AWS Clusters. AWS has reported an issue causing packet loss on AWS Direct Connect which is affecting our dedicated network connectivity to one of our databases. We are working on redirecting traffic away from the affected availability zone.
Posted Aug 22, 2020 - 12:27 EDT
This incident affected: US 01 Cluster (Data Processing, Outbound Messaging), US 02 Cluster (Data Processing, Outbound Messaging), US 03 Cluster (Data Processing, Outbound Messaging), US 04 Cluster (Data Processing, Outbound Messaging), and US 06 Cluster (Data Processing, Outbound Messaging).