We were able to work with our DBAs to resolve this issue, and all systems are operating normally. Analytics on the dashboard should appear and are no longer delayed.
We have escalated the support case with the database vendor to assist in root cause analysis.
Posted Oct 02, 2019 - 02:05 EDT
Update
Data processing is back to normal for all customers and the backlog has been processed. At this time, the dashboard on US 01 still has delays in showing statistics for events and campaigns. We are working to restore processing for those statistics.
Posted Oct 02, 2019 - 00:45 EDT
Update
We are continuing to work with our DBAs and have escalated to the database vendor for further support. The majority of customers on US 01 have full systems operational except that some dashboard analytics may not be updated.
Posted Oct 02, 2019 - 00:23 EDT
Update
We are continuing to work with our DBAs on this issue. At this time, message sending has remained fully operational since 11:05pm ET.
Posted Oct 01, 2019 - 23:24 EDT
Update
Message sending is currently caught up and processing normally right now. Data processing for the majority of customers has also returned to normal. We are continuing to work with our DBAs toward this issue.
Posted Oct 01, 2019 - 23:10 EDT
Update
We are continuing to work with our DBAs on the issue.
Posted Oct 01, 2019 - 22:34 EDT
Update
Data processing has returned to normal levels for a subset of customers. Campaign backlog is processing but still slower than usual on US 01. The dashboard on US 01 may not show updated analytics for message sending due to the data processing delay, but messages are sending properly.
Posted Oct 01, 2019 - 21:56 EDT
Investigating
We are investigating data processing delays and some slower than expected message sending on US 01.
Posted Oct 01, 2019 - 20:58 EDT
This incident affected: US 01 Cluster (Data Processing, Outbound Messaging).