On Sunday, January 22nd at approximately 05:21 AM Eastern Time (ET), the Appboy Services became unavailable for customers in the US data center. Services were restored for most customers by 06:40 AM ET, with all service restored at approximately 7:55 AM ET.
SDK Data Collection Delays
From 05:21 AM ET to 07:55 AM ET, devices attempting to send Appboy information were unable to send that information to our servers via any of our SDKs. During this time, these devices continued storing data locally, and any incomplete data would have been re-sent to Appboy during their first transmission after 07:55 AM ET.
REST API Data Collection Delays
From 05:28 AM ET, until 05:34 AM ET and again from 05:45 AM ET to 06:01 AM ET the REST API was unavailable as well. Any data transmission that was not automatically retried outside of this window will need to be resent to Appboy.
Message Sending & Data Processing Delays
From 05:21 AM ET to 06:40 AM ET, message sending either slowed or ceased for all customers in Appboy’s US region. After 06:40 AM ET any delayed messaging resumed sending, and all subsequent messaging went out appropriately.
An error in our alerting protocol this morning failed to notify our team that a configuration server had failed at approximately 03:00 AM ET. Our team did receive an alert that load was increasing, but absent alerting of the root cause, it was incorrectly determined that our automatic scaling processes would handle the increased load. Automatic failovers and redundancies also failed to account for the error, which caused the outage to occur at 05:41 AM ET.
Appboy’s DevOps staff and CTO Jon Hyman have reviewed the incident timeline and alerting failures which led to the ultimate outage. In response we’ve corrected the error, and have made holistic improvements to automated escalation alerting protocol to involve senior escalation more often and earlier in an incident’s lifecycle. Furthermore, we are working through a number of follow-ups, both procedural and infrastructural, which will prevent recurrence of this type of outage in the future.
Please let support@appboy.com know if you have any questions.
Best,
The Appboy Team