All data processing and message sending in the affected data center caught up within a few minutes of the last status update, and all has remained stable since then. API calls from approximately 2:04pm until 2:09pm ET had elevated error rates, and data processing and message sending was queued for approximately 20 minutes starting at 2:13pm ET while we took backups and failed over services. As mentioned in the last status update, we believe this to have been a hypervisor issue with one of our machines.
Posted Nov 05, 2016 - 15:51 EDT
Monitoring
We have failed over to this instance's secondary database and processing has resumed. Before resuming service, we took complete backups which increased resolution time. Job processing has resumed in all data centers and should be fully caught up in a few minutes. From our investigation, we believe that this was caused by a hypervisor issue with this machine on Amazon.
Posted Nov 05, 2016 - 14:47 EDT
Investigating
We are currently looking into a database failure. Job processing may be delayed in one of our US-EAST data centers.