Resolved -
This incident is resolved. As of 15:18 ET / 19:18 UTC, we have fully processed the backlog of jobs, and our EU02 Currents connectors are all again processing within normal latency ranges. No data was lost, and all Currents events have been sent to their destinations.
Jul 31, 15:25 EDT
Update -
We continue to make progress clearing the remaining backlog on EU02, improving our performance, and decreasing overall latency. We expect to be caught up within the next hour, although a small number of events may take slightly longer to fully process. The remaining impact is limited to less than half a percent of our overall traffic.
We'll provide another update once we're fully caught up.
Jul 31, 14:04 EDT
Identified -
We continue to investigate degraded performance in EU02. We have identified the root cause of the underlying latency, which stemmed from multiple Currents connectors failing to gracefully restart after routine maintenance, but appearing in a healthy state in our monitoring system.
Those specific servers build significant backlogs while in this state. We have completed a graceful restart, and have spun up additional resources to process through the backlogged items more rapidly.
At this point ~95% of data is flowing normally, while up to ~5% of our connectors, impacting a subset of customers, continue to experience latency.
We see throughput significantly improved due to config changes, and will provide an update in 1 hour or when we have an updated timeline.
Jul 31, 12:14 EDT
Investigating -
We are currently investigating degraded performance in our EU02 cluster, which is causing delays in Currents processing for a small subset of customers. No data has been lost, and the majority of Currents data continues to flow normally. Our team is actively working toward a resolution and will provide updates as they become available.
Jul 31, 10:09 EDT