Postmortem
Braze added more information and clarifying language to the historical timeline in response to questions we've received since the incident. This was additive only; no previously posted information was removed or modified.
Customers can expect future incidents to use this more detailed format.
As the impact affected multiple Braze clusters and customers differently, a general-purpose RCA will not be posted. However, RCAs are available; please reach out to your customer success manager or support@braze.com.
Posted Oct 02, 2024 - 15:09 EDT
Resolved
We have been monitoring since our last update and are declaring this incident as resolved.
REST and SDK API performance remains stable in all clusters.
Dashboard login remains fully operational in all clusters.
Braze's Transactional Message sending APIs continue to operate within SLA bound latency.
Posted Sep 30, 2024 - 17:24 EDT
Update
Both Data Processing and Outbound Messaging backlogs in US01 have been fully processed.
Both Data Processing and Outbound Messaging backlogs in US06 have been fully processed.
For services still identified as Degraded Performance, we are still working through the job backlogs for a small subset of customers.
At this point, less than 1% of customers have messaging latency or data processing queues greater than 10 minutes.
We will update in 1 hour or sooner when we have a material update.
Posted Sep 30, 2024 - 16:50 EDT
Update
All job processing backlogs in US04 have been fully processed.
The data processing backlogs in US06 and US01 are fully processed. We are still working through the messaging backlogs for a small subset of customers.
We will continue to update as clusters fully clear their backlogs.
At this point, less than 5% of customers have a backlog of messages to send that are showing latency in excess of 10 minutes.
We will update in 1 hour, or when we have a material update, whichever is sooner.
REST and SDK API performance remains stable in all clusters.
Dashboard login remains fully operational in all clusters.
Braze's Transactional Message sending APIs continue to operate within SLA bound latency.
Posted Sep 30, 2024 - 16:06 EDT
Update
At this point, approximately 90% of customers either no longer have any backlog of messages to send, or the remaining messages in the backlog have delays fewer than 10 minutes. As of this update, backlog processing on other US clusters continues, and we are adding more resources to maximize throughput.
US04 remains on track to shortly have the backlog fully processed.
In the last 45 minutes, we have processed through more than a third of the remaining backlog across customers still impacted.
REST and SDK API performance remains stable in all clusters.
Dashboard login remains fully operational in all clusters.
Braze's Transactional Message sending APIs continue to operate within SLA bound latency.
Posted Sep 30, 2024 - 15:26 EDT
Update
We have completed processing through the backlog on US02, US08.
US04 is on track to be resolved in the next 30 minutes. As of this update, backlog processing on other US clusters continues and we are continuing to add more resources to maximize throughput. At this point, approximately 85% of customers either no longer have any backlog of messages to send, or the remaining messages in the backlog are delayed by less than 10 minutes.
REST API errors have recovered in all clusters.
SDK API errors have recovered in all clusters.
REST and SDK API performance remains stable in all clusters.
Message Sending and Data Processing remains in real time, on US07.
Dashboard in US01, US03, and US05 is now performant for all customers.
Dashboard login remains fully operational in US02, US04, US06, US07, US08, and US09.
Braze's Transactional Message sending APIs continue to operate within SLA bound latency.
Posted Sep 30, 2024 - 14:48 EDT
Update
We have completed processing through the backlog of all delayed messages, for all customers on US07, confirming our resolution steps are effective.
As a minor correction to our last post, Dashboard logins had a less than 1% error rate, but our US01, US03, and US05 clusters continue to have some sporadic timeouts for some customers.
We are still scaling up processing to ensure we rapidly process through the backlogs in other clusters, our next update should provide insight into latency and backlog processing completion targets.
REST and SDK error rates have returned to normal levels in all clusters.
Dashboard login remains fully operational in US02, US04, US06, US07, US08, and US09.
Braze's Transactional Message sending APIs are now executing within SLA bound latency for all customers.
Posted Sep 30, 2024 - 14:07 EDT
Update
We are continuing to work on a fix for this issue.
Posted Sep 30, 2024 - 13:36 EDT
Update
Dashboard login is now fully operational in all clusters.
The error rates for the REST and SDK APIs are now in normal ranges. Customers should retry their REST API calls at this point.
We are actively processing the backlog and are actively making progress against that.
Message sending and data processing remain latent.
From now on, we will update the Statuspage every 30 minutes. Our next update will be at 2 p.m. ET.
Posted Sep 30, 2024 - 13:34 EDT
Update
While network connectivity to the database layer has been restored, we continue to experience errors in services while cycling through restarts at the database layer. We expect no SDK data loss and all messages to enqueue and send.
We continue to process the backlog, however, recovery is taking longer than originally anticipated.
From now on, we will update the Statuspage every 30 minutes. Our next update will be at 1:30 p.m. ET.
Posted Sep 30, 2024 - 12:56 EDT
Update
We have recovered most network connections to our databases and are in the process of scaling up backlog processing - prioritizing message sending. We have begun to process the message-sending backlog on impacted clusters. Customers have a backlog of queued messages on a customer-by-customer basis. As we process these backlogs and communicate broadly, we will speak on a cluster-wide or customer-percentage basis. However, please note that these are worst-case numbers, and for most customers, your specific backlogs will have been processed much earlier.
Our REST and SDK API error rates have dropped below 10% and are close to fully recovered.
Posted Sep 30, 2024 - 11:18 EDT
Update
A fix to the network issue has been implemented, and we are now restoring connections to the database layer.
Posted Sep 30, 2024 - 10:13 EDT
Identified
We have become aware of a networking issue at our database layer that is impacting services across our US clusters. Our engineers and DBAs are currently engaged and investigating.
Posted Sep 30, 2024 - 10:04 EDT
This incident affected: US 01 Cluster (Dashboard, SDK Data Collection, Data Processing, REST APIs, Outbound Messaging), US 02 Cluster (Dashboard, SDK Data Collection, Data Processing, REST APIs, Outbound Messaging), US 03 Cluster (Dashboard, SDK Data Collection, Data Processing, REST APIs, Outbound Messaging), US 04 Cluster (Dashboard, SDK Data Collection, Data Processing, REST APIs, Outbound Messaging), US 06 Cluster (Dashboard, SDK Data Collection, Data Processing, REST APIs, Outbound Messaging), US 08 Cluster (Dashboard, SDK Data Collection, Data Processing, REST APIs, Outbound Messaging), US 05 Cluster (Dashboard, SDK Data Collection, Data Processing, REST APIs, Outbound Messaging), and US 07 Cluster (Dashboard, SDK Data Collection, Data Processing, REST APIs, Outbound Messaging).