Message sending in EU 01 has returned to normal processing times. The message sending backlog caused by the Google Firebase Cloud Messaging incident has been sent. We are continuing to monitor. All systems look normal at this time.
Posted Nov 12, 2021 - 06:32 EST
Monitoring
We have been able to fully restore Android push sending capabilities. Backlogs for other non Android push channels are mostly finished sending and have turned to normal sending speeds. We are scaling up servers to more quickly process the backlog of messages to be sent. No messages have been lost.
Posted Nov 12, 2021 - 06:27 EST
Update
We are slowly re-enabling Android push sending in EU 01.
Posted Nov 12, 2021 - 06:09 EST
Identified
Android push sending in EU seems unavailable because our systems cannot communicate with Google's Firebase Cloud Messaging. Google has reported an issue on their status page specific to Firebase at https://status.firebase.google.com/.
Delays sending Android push notifications caused cascading message sending delays for other channels beginning at approximately 3:35am ET. We have been able to unblock other channels and those messages are currently sending their backlog. Android Push sending in EU will remain unavailable until Firebase Cloud Messaging fixes their incident.
Posted Nov 12, 2021 - 05:52 EST
Investigating
We are investigating issues affecting delays in EU. The source of the issue appears to be an outage with Google Cloud affecting Android Push sending (https://status.cloud.google.com/incidents/1xkAB1KmLrh5g3v9ZEZ7), which is causing cascading message sending delays.
Posted Nov 12, 2021 - 05:44 EST
This incident affected: EU 01 Cluster (Outbound Messaging).