All systems in EU 01 have remained stable since the previous update. We have continued to remediate the underlying issue and are continuing to work through our remediation plan with our DBA team.
Posted Feb 07, 2019 - 19:45 EST
Update
Latencies are back to normal. We are continuing to work with our DBAs on full remediation.
Posted Feb 07, 2019 - 14:56 EST
Update
Data processing latencies for the remaining affected EU 01 customers is coming down after our most recent patch with our DBA team. We are continuing to troubleshoot.
Posted Feb 07, 2019 - 14:09 EST
Update
Data processing is back to real-time for over 95% of the affected EU 01 customers. The code patch made to the dashboard with the last update has been successful in removing occurrences of timeouts since then. We are continuing to work with our DBAs on full remediation.
Posted Feb 07, 2019 - 13:29 EST
Update
We are continuing to work on the issue, which affects approximately 20% of Braze EU 01 customers. The current status is that message sending is operating by and large nominally with some occasional delays for messages which may last for a few minutes. Of the 20% affected, about 50% of those customers have had data processing delays that at times are up to one hour. The other 50% is processing in real-time. Some of the affected customers have experienced timeouts on the dashboard while saving or duplicating campaigns. A code patch was deployed moments ago which should lessen occurrences of those dashboard errors.
The overall issue is with some of our secondary read replica databases that serve 20% of our EU 01 cluster. Our DBAs are continuing to investigate. We have multiple teams working on the issue.
Posted Feb 07, 2019 - 12:42 EST
Update
We are continuing to work with our DBAs on a fix for this issue.
Posted Feb 07, 2019 - 10:59 EST
Identified
We are seeing the issue re-appear for a few of the initially affected customers and are digging in with our DBA team.
Posted Feb 07, 2019 - 09:42 EST
Monitoring
A fix has been implemented for all affected customers and we are monitoring the results. All messaging that had been queued has been sent. Data processing is still delayed for the affected customers as we spin up more capacity to process the backlog.
Posted Feb 07, 2019 - 08:58 EST
Identified
We have identified the database issue that is causing degraded performance for approximately 20% of Braze EU 01 customers and are working with our DBA team.
Posted Feb 07, 2019 - 08:34 EST
Update
We are continuing to investigate this issue.
Posted Feb 07, 2019 - 08:01 EST
Investigating
We are investigating degraded performance on our EU-01 cluster
Posted Feb 07, 2019 - 08:00 EST
This incident affected: EU 01 Cluster (Dashboard, Data Processing, REST APIs, Outbound Messaging).