This incident has been resolved. Processing latency for all components has returned to normal levels.
Posted Aug 02, 2022 - 06:29 EDT
Update
We continue to monitor for recurrence: Dashboard, APIs and Data Processing have returned to normal processing for all customers. Message Sending processing has returned to real-time processing for the majority of customers.
Posted Aug 02, 2022 - 05:47 EDT
Update
We continue to monitor for recurrence and are seeing processing return to normal for Dashboard and Data Processing for most customers. Message sending continues to be latent as we work through the backlog.
Posted Aug 02, 2022 - 05:34 EDT
Monitoring
We've identified the root cause of this issue. We're seeing error rates decrease, and are continuing to troubleshoot and monitor.
Posted Aug 02, 2022 - 05:02 EDT
Update
We continue to work to identify the root cause of this issue. All components that require access to our underlying database are impacted, including message composition & sending, canvas & campaign processing, and navigating the dashboard.
Posted Aug 02, 2022 - 04:03 EDT
Identified
We are currently investigating a database issue resulting in increased latency and timeouts in our US 03 cluster. Our DevOps and DBA teams are investigating.
Posted Aug 02, 2022 - 03:29 EDT
This incident affected: US 03 Cluster (Dashboard, Data Processing, REST APIs, Outbound Messaging).