Elevated Error Rates on Message Sending and Dashboard Services Due to AWS S3 Incident

Incident Report for Braze, Inc.

Resolved

AWS has resolved their S3 issue. All our systems are operating normally.
Posted Feb 28, 2017 - 17:34 EST

Monitoring

The vast majority of backlogged messages have gone out, and new message sends are not delayed. Writes to Amazon S3 remain unavailable, meaning that image and CSV uploads on the Appboy Dashboard remain impacted.
Posted Feb 28, 2017 - 16:47 EST

Update

Amazon has posted an update indicating that they have a partial solution, and will have a full solution within the hour. https://www.appboy.com, documentation, and the Appboy Academy are back online due to their fix. Message sending is still delayed but is now draining the backlog of things that were unable to send due to the S3 outage.
Posted Feb 28, 2017 - 15:59 EST

Update

The Amazon S3 outage is also affecting sites behind Amazon's content delivery network. As a result, if you are in the US northeast region, you may be unable to load https://www.appboy.com or our documentation and academy pages until Amazon resolves the issue on their end.

We will continue to update periodically. You may also refer to Amazon's status page for more information: https://status.aws.amazon.com/.
Posted Feb 28, 2017 - 14:17 EST

Update

Amazon S3 API errors has also started to affect our dashboard services as campaign editing at creation requires images hosted on Amazon S3
Posted Feb 28, 2017 - 13:16 EST

Identified

An increase in error rates on Amazon's S3 API in the US-EAST-1 region is causing some delays in message sending, we are monitoring the situation and will provide updates as Amazon provides updates to us.
Posted Feb 28, 2017 - 13:07 EST