High error rates in US 01 and delayed processing

Incident Report for Braze, Inc.

Postmortem

Beginning at approximately 2:10am Eastern Time on June 4, 2020 and lasting until approximately 5:19am ET, Braze experienced a database issue affecting US 01 that resulted in increased error rates to our APIs and dashboard, as well as degraded message sending and data processing.

No data was lost, all data that was received by Braze did process, and all messages that were scheduled to send were sent. Any SDK API attempt that failed would have stored the data on the end user's device and re-sent it upon the next communication with Braze's servers. Any REST API call that failed will need to be retried.

Our database service provider regularly runs a compaction process to reclaim disk storage and optimize the database. This process is continuously running and iterates through our database secondary read-replicas and performs the maintenance operation on them one at a time. In this incident, an error occurred which removed multiple secondary read-replicas from the database on US 01 which stores information such as app group and campaign metadata, segment definitions and analytics. Because of this, there were widespread error rates and degraded performance on US 01 occurring in this approximately 3 hour window.

We are awaiting a full incident report and root cause analysis from our database service provider and will provide more information upon request once we receive that information.

This incident only affected US 01. Customers who are on other Braze clusters were unaffected.

Posted Jun 04, 2020 - 10:33 EDT

Resolved

All systems and processing remain operational for all customers on US 01.
Posted Jun 04, 2020 - 05:56 EDT

Update

Data processing latencies have returned to normal for the vast majority of customers on US 01, and we have additional servers coming online to fully process the backlog. We are continuing to monitor as all systems remain healthy.
Posted Jun 04, 2020 - 05:39 EDT

Monitoring

The fix our DBAs put in place has remained stable, and all systems are returning to normal. Data processing remains delayed for some customers but is rapidly catching up to real-time.
Posted Jun 04, 2020 - 05:29 EDT

Update

Timeseries analytics on the US 01 dashboard should be updating and returning to normal.
Posted Jun 04, 2020 - 05:21 EDT

Update

Our DBA team has restored secondary read-replicas and expect to be able to increase the processing speed of our data processing backlog shortly.
Posted Jun 04, 2020 - 05:16 EDT

Update

We are continuing to work on a fix for this issue.
Posted Jun 04, 2020 - 05:01 EDT

Identified

Our DBAs are working through a possible solution. Message sending has caught up for the majority of customers and campaigns on US 01. Timeseries analytics are still delayed and will not show updated statistics on the dashboard.
Posted Jun 04, 2020 - 04:30 EDT

Update

Timeseries analytics on the dashboard, such as sessions or message sends, are heavily delayed. However, messages are being sent and data processed in US 01. The dashboard will show inaccurate stats while we continue to work to resolve this issue.
Posted Jun 04, 2020 - 04:15 EDT

Update

Our DBAs are continuing to work on the issue. Initial investigation appears to be that a database maintenance task has taken multiple secondary read-replicas offline.
Posted Jun 04, 2020 - 04:04 EDT

Investigating

We are investigating an issue with the US 01 environment that is causing high error rates in US 01 and degraded data processing and message sending.
Posted Jun 04, 2020 - 02:10 EDT
This incident affected: US 01 Cluster (Dashboard, SDK Data Collection, Data Processing, REST APIs, Outbound Messaging).