At this point, all components are operating within normal latency and performance ranges, and we consider the incident to be resolved.
Our MSP has committed to providing a Technical Root Cause Analysis next week, and we will update this status page once we're informed of that.
Thank you.
Posted Oct 04, 2019 - 16:58 EDT
Monitoring
Our systems are continuing to process backlogged work and more components are operational. We are fully in a monitoring state at this point. We will be updating to operational as the latency on each component drops back into a normal range.
Posted Oct 04, 2019 - 16:38 EDT
Update
Our systems are continuing to process backlogged work and more components are operational. We will update this page again soon as more systems are fully caught up.
Posted Oct 04, 2019 - 16:08 EDT
Update
We believe our managed service provider's DNS issue has been identified and resolved at their data center. On our side, data is currently being processed and messages are sending successfully. No data has been lost but it will take some time to process the backlog of work. We'll continue to update the status page as everything recovers.
Posted Oct 04, 2019 - 16:02 EDT
Update
We are continuing to investigate this issue.
Posted Oct 04, 2019 - 15:49 EDT
Update
Because the issue is DNS-related, we're seeing intermittent errors with the Dashboard and REST API as well. Our managed service provider's data center and network teams are fully deployed working on the issue. We will relay information we get from them to our status page as the issue is resolved.
Posted Oct 04, 2019 - 15:38 EDT
Update
As of 5:20 PM (Eastern) (2:20 PM Pacific) our managed service provider for our Mongo Database hosting has identified it to be an issue with the network layer at their datacenter. http://status.objectrocket.com/ We are actively working with them to resolve the issue.
Posted Oct 04, 2019 - 15:28 EDT
Update
We are continuing work on resolving the issue.
Posted Oct 04, 2019 - 15:21 EDT
Update
We're still working on identifying the root cause. We've narrowed it down to a database issue and are actively working with our DBAs on a resolution. Next update in ~30 minutes or a meaningful status change.
Posted Oct 04, 2019 - 14:49 EDT
Investigating
We are investigating data processing delays and some slower than expected message sending on US 01, US 03, and US 04.
Posted Oct 04, 2019 - 14:30 EDT
This incident affected: US 01 Cluster (Dashboard, SDK Data Collection, Data Processing, REST APIs, Outbound Messaging), US 03 Cluster (Dashboard, SDK Data Collection, Data Processing, REST APIs, Outbound Messaging), and US 04 Cluster (Dashboard, Data Processing, REST APIs, Outbound Messaging).