Across all clusters and services, we've confirmed latency and performance have returned to normal operating levels.
Posted Jan 19, 2024 - 15:17 EST
Update
We are continuing to monitor for any further issues.
Posted Jan 19, 2024 - 14:35 EST
Update
Latency continues to improve. Dashboard usage and SDK data collection are operating as normal in all environments. Data Processing and Outbound Messaging remain latent in US01, and US03, and have resolved in all other environments.
Posted Jan 19, 2024 - 13:19 EST
Monitoring
The change has been successfully reverted, and we're in the process of scaling up to process through the backlog.
Posted Jan 19, 2024 - 12:35 EST
Update
The networking team supporting our data center has identified the specific configuration change affecting our performance, and are in the process of reverting that change.
Posted Jan 19, 2024 - 11:53 EST
Identified
Braze has identified an issue causing latency across multiple clusters. An issue has been identified with the networking infrastructure in one of our data centers. This networking issue has been identified to cause increased latency in connecting to our Mongo databases, resulting in degraded performance & latency of our Data Processing and Message Sending capabilities.
Posted Jan 19, 2024 - 11:16 EST
This incident affected: US 01 Cluster (Dashboard, SDK Data Collection, Data Processing, Outbound Messaging), US 03 Cluster (Dashboard, SDK Data Collection, Data Processing, Outbound Messaging), US 06 Cluster (Dashboard, SDK Data Collection, Data Processing, REST APIs, Outbound Messaging, Currents), and US 05 Cluster (Dashboard, SDK Data Collection, Data Processing, Rest APIs, Outbound Messaging, Currents).