We have completed our internal remediation plan. All systems are fully operational. We will be sending out a communication tomorrow to customers on the US 01 cluster summarizing our work and providing more information. Customers who are on the US 01 cluster log in to their dashboard at https://dashboard.braze.com or https://dashboard-01.braze.com. Customers who log in using other URLs were not impacted by either the AWS patching or by today's work.
Posted Jan 05, 2018 - 01:58 EST
Update
Message sending and data processing has resumed in US 01 and is catching up on the backlog.
Posted Jan 05, 2018 - 01:48 EST
Update
We are updating the last round of servers on our internal remediation plan. Some data processing and message sending in US 01 has been temporarily paused for this final round of server upgrades. We anticipate restoring this processing in the next few minutes
Posted Jan 05, 2018 - 01:33 EST
Update
Message sending and data processing has resumed in US 01 and is catching up on the backlog.
Posted Jan 05, 2018 - 00:16 EST
Update
Some data processing and message sending in US 01 has been temporarily paused for the next round of server upgrades. We anticipate restoring this processing in the next few minutes.
Posted Jan 04, 2018 - 23:52 EST
Update
All systems are fully operational in a similar state prior to when the patches were applied by AWS. We are still continuing to perform maintenances on our servers according to our internal remediation plan and will keep this status page updated if there is downtime from such maintenance.
Posted Jan 04, 2018 - 23:02 EST
Update
We are continuing to monitor changes from our ongoing maintenance in response to AWS' patching of the Intel security vulnerability. At this time, all systems on US 01 are fully functional, though approximately 10% of customers on US 01 may experience data processing delays of a few minutes while we continue mitigation. Our team is continuing to make updates to our infrastructure, and will keep this status page updated with progress.
Posted Jan 04, 2018 - 18:34 EST
Update
Message sending and data processing has resumed in US 01 and is catching up on the backlog.
Posted Jan 04, 2018 - 14:37 EST
Update
Some data processing and message sending in US 01 has been temporarily paused for emergency server upgrades. We anticipate restoring this processing in the next few minutes.
Posted Jan 04, 2018 - 14:21 EST
Update
Message sending and data processing has resumed in US 01 and is catching up on the backlog.
Posted Jan 04, 2018 - 11:29 EST
Update
Some data processing and message sending in US 01 has been temporarily paused for emergency server upgrades.
Posted Jan 04, 2018 - 11:04 EST
Monitoring
Yesterday, a security flaw in Intel processors was announced that has forced an emergency redesign of the underlying operating system code for Linux and Windows machines. Cloud providers such as AWS, Microsoft Azure, and Google Cloud have been rushing to patch their servers since yesterday. As a side effect of the redesign, most server workloads are experiencing degraded performance and several reports of performance problems are already starting to populate around the internet.
Braze has been monitoring the situation for impacts on our servers which is being caused by both the patching process itself and the ongoing performance penalty thereafter. At this time, customer impact is limited to some delays in data processing on the US 01 cluster. We have seen periodic elevated error rates on the APIs lasting a few seconds at a time. No data in our queues has been lost, and all messages are sending as scheduled.
We are in conversation with our infrastructure providers regarding their efforts, and also at the same time are strategizing for how to best implement a long-term solution for the ongoing impact the security patches are having on our infrastructure. Please be aware that there may be some emergency maintenance performed today in response to the security patches, and this maintenance could result in downtime. This downtime will not result in any data loss on our servers, and Braze SDKs will cache any recorded data in the event that our SDK API is brought down for maintenance.
We will keep this status page updated throughout the day as we determine the best course of action.