Issue Description

On July 14 starting at approximately 2:14am ET, the US 03 Cluster of the Braze Services became unavailable, affecting the Braze Dashboard and APIs of customers on that Cluster. Data processing and message sending was delayed until service was restored at 2:51am ET. No data was lost, however any REST API calls which received an error during the incident window need to be resent by customers.

Cause

The root cause of this error is that a database ran out of disk space which caused both the primary and one of its secondary servers to crash. While the hard disk space was automatically extended via automation, the crash prevented the processes from automatically restarting without manual intervention. While the database has two read-replicas, since both the primary and one secondary had crashed, that prevented automatic failover of the remaining online secondary to the primary database. After manual intervention, all Services returned to normal operations.

Remediation

• DBAs are modifying thresholds for alerting for disk space and automatic disk extension
• Braze will be improving resiliency around failures of this particular database, which stores customer entitlements.

We apologize for the inconvenience.