Appboy is investigating connectivity issues to our dashboard and APIs

Incident Report for Braze, Inc.

Postmortem

CPU on both aggregation switches at our data center spiked to high utilization levels and lost OSPF adjacency to upstream routers, which caused network failures. Our data center technician team escalated to Cisco, who identified critical bugs in the Cisco NX-OS software release on the aggregation switches.

OSPF adjacency transitioned back to FULL, restoring connectivity due to the high CPU load on the aggregation switches. After a reboot of the aggregation router, network connectivity was restored to normal.

Our data center team will be performing emergency maintenance next week to upgrade the Cisco router software on the aggregation switches.

Posted Apr 03, 2015 - 17:25 EDT

Resolved

All systems remain online and are fully operational. All messaging that was scheduled to go out finished as soon as we posted the last update. We are awaiting a full RCA from Rackspace and will post it when we have it.

Posted Mar 30, 2015 - 16:42 EDT

Update

All campaigns and messages which were scheduled to go out during the outage are currently sending.

Posted Mar 30, 2015 - 16:24 EDT

Monitoring

We are continuing to monitor the situation, but all services are back online.

Posted Mar 30, 2015 - 16:19 EDT

Identified

Service is sporadically returning as Rackspace continues to troubleshoot with Cisco. Evidence indicates that the network aggregation switches are maxing out their CPU which leads into a loss of connectivity across the board at the data center.

Posted Mar 30, 2015 - 16:21 EDT

Update

Rackspace has escalated the issue to Cisco and is continuing to troubleshoot.

Posted Mar 30, 2015 - 16:12 EDT

Update

Rackspace is continuing to investigate, as some servers are still unreachable from the Internet.

Posted Mar 30, 2015 - 15:53 EDT

Update

Rackspace's backbone team is currently troubleshooting the network aggregators in the data center, believed to be the source of the issue. We do not currently have an ETA except that Rackspace has confirmed it to be an issue affecting a large number of their customers.

Posted Mar 30, 2015 - 15:21 EDT

Update

Rackspace appears to be experiencing global connectivity issues across all data centers, which is affecting our services. We are working with their team to resolve the issue.

Posted Mar 30, 2015 - 15:06 EDT

Investigating

We are currently investigating this issue.

Posted Mar 30, 2015 - 14:59 EDT