Beginning at approximately 2:10am Eastern Time on June 4, 2020 and lasting until approximately 5:19am ET, Braze experienced a database issue affecting US 01 that resulted in increased error rates to our APIs and dashboard, as well as degraded message sending and data processing.
No data was lost, all data that was received by Braze did process, and all messages that were scheduled to send were sent. Any SDK API attempt that failed would have stored the data on the end user's device and re-sent it upon the next communication with Braze's servers. Any REST API call that failed will need to be retried.
Our database service provider regularly runs a compaction process to reclaim disk storage and optimize the database. This process is continuously running and iterates through our database secondary read-replicas and performs the maintenance operation on them one at a time. In this incident, an error occurred which removed multiple secondary read-replicas from the database on US 01 which stores information such as app group and campaign metadata, segment definitions and analytics. Because of this, there were widespread error rates and degraded performance on US 01 occurring in this approximately 3 hour window.
We are awaiting a full incident report and root cause analysis from our database service provider and will provide more information upon request once we receive that information.
This incident only affected US 01. Customers who are on other Braze clusters were unaffected.