This incident has been resolved and processing has fully returned to normal.
Posted Feb 03, 2020 - 19:25 EST
Update
Data processing and message sending on US 01 have returned to normal for the vast majority of US 01 customers. We are continuing to monitor.
Posted Feb 03, 2020 - 18:52 EST
Update
US 03 is back to normal for all systems. US 01 message sending and data processing are mostly caught up with its backlog. We anticipate they will be back to normal in the next 15 minutes.
Posted Feb 03, 2020 - 18:20 EST
Monitoring
Systems are returning to normal and we are spinning up more servers to process the data processing and message sending backlog on US 01 and US 03.
Posted Feb 03, 2020 - 17:49 EST
Identified
We are continuing to resolve but many services are coming back online at this time. Delayed messages which had been queued are being sent.
Posted Feb 03, 2020 - 17:38 EST
Investigating
The fix implemented by the DBAs did not resolve the issue, our teams are continuing to investigate.
Posted Feb 03, 2020 - 17:09 EST
Update
Our DBAs have implemented a fix on their end. We are working through restarting connections to our databases. Dashboards no longer appear to be erroring out, but other processes are still impacted.
Posted Feb 03, 2020 - 16:56 EST
Identified
Our DBAs believe they have identified an issue that may be contributing and are working toward resolution.
Posted Feb 03, 2020 - 16:37 EST
Update
We are continuing to investigate this issue.
Posted Feb 03, 2020 - 16:24 EST
Update
Our DBAs are investigating a loss of connectivity to a set of load balancers which manage connections to some of our databases.
Posted Feb 03, 2020 - 16:20 EST
Update
We are continuing to investigate this issue.
Posted Feb 03, 2020 - 16:19 EST
Investigating
We are currently investigating a connection issue to some databases which is affecting US clusters. Customers may be seeing intermittent errors on REST APIs and dashboard in the US. We are working with our DBA team on resolution.
Posted Feb 03, 2020 - 16:16 EST
This incident affected: US 01 Cluster (Dashboard, SDK Data Collection, Data Processing, REST APIs, Outbound Messaging), US 03 Cluster (Dashboard, SDK Data Collection, Data Processing, REST APIs, Outbound Messaging), and US 04 Cluster (SDK Data Collection, REST APIs).