Systems have remained operational since the earlier incident. Due to the recent continued incidence of AWS chassis instability for the dedicated instance types that our database clusters run on, we have accelerated a number of remediation plans to remove the impact on Appboy systems due to this class of failure.
We also identified an issue during recovery with some message sending jobs that were scheduled for future delivery. We've notified affected customers and are working toward resolution and recovery.
Posted Aug 23, 2017 - 21:10 EDT
Monitoring
Systems are operational. We are continuing to monitor and investigate.
Posted Aug 23, 2017 - 15:35 EDT
Update
We have removed the affected availability zone's APIs from load balancers. REST API traffic is returning to normal. Message sending and data processing in unaffected zones are processing at a degraded pace.
Posted Aug 23, 2017 - 15:18 EDT
Identified
We appear to have a recurrence of the AWS issue which has taken out a dozen databases with failing health checks in our US EAST environment. We are working on database failover.