Investigating connectivity issues caused by Fastly issue

Incident Report for Braze, Inc.

Resolved

All systems have remained operational. While Fastly has not resolved their status page issue, CenturyLink which owns Level 3 has confirmed that their outage is resolved. As such, we are going to resolve this issue on our side.
Posted Aug 30, 2020 - 12:50 EDT

Update

All Braze services remain operational. As the Level 3 incident remains ongoing, we are continuing to monitor and will resolve this on our status page when Fastly does at https://status.fastly.com/incidents/p1lwmmv2j2gq.
Posted Aug 30, 2020 - 10:24 EDT

Update

Error rates for Braze Services have been at normal levels since 7:36am ET, as Fastly has indicated they are re-routing traffic across the affected regions. We will continue to monitor their status page and post updates.
Posted Aug 30, 2020 - 08:37 EDT

Update

Upon our teams' investigation, it appears there is a global internet outage caused by the Tier 1 Network, Level 3. This outage may also prevent certain customers from loading various webpages unrelated to Braze. If you receive an error, please retry your request.
Posted Aug 30, 2020 - 07:49 EDT

Monitoring

Our upstream network and CDN service provider, Fastly, is reporting an incident on their status page that they are "observing increased errors and latency across multiple regions due to a common IP transit provider experiencing a widespread event." Braze's inbound networking graphs show an increased error rate for US services starting at approximately 6:55am Eastern Time. If you receive a 50x error communicating with Braze's Services, please retry your request.

Fastly is posting updates at https://status.fastly.com/incidents/p1lwmmv2j2gq. We will continue to monitor and post updates.
Posted Aug 30, 2020 - 07:39 EDT
This incident affected: US 01 Cluster (Dashboard, SDK Data Collection, REST APIs), US 02 Cluster (Dashboard, SDK Data Collection, REST APIs), US 03 Cluster (Dashboard, SDK Data Collection, REST APIs), US 04 Cluster (Dashboard, SDK Data Collection, REST APIs), US 06 Cluster (Dashboard, SDK Data Collection, REST APIs), and US 08 Cluster (Dashboard, SDK Data Collection, REST APIs).