Our data center team has resolved the issue. The incident was caused by human error at our data center. After the network maintenance we performed on January 5th when we upgraded our load balancers, a decommission order was placed for the old load balancers. Because this was an in-place upgrade, the physical device location was the same on the decommission form. This morning, the data center technicians accidentally removed the new, live load balancers from the server cabinet. The data center operations team is working internally to determine how this mixup happened and implement processes to prevent it.
Posted Jan 22, 2015 - 08:40 EST
Identified
Our data center operations team believes they have identified the issue and are working toward a resolution.
Posted Jan 22, 2015 - 08:40 EST
Update
Our networking team continues to investigate, as it appears that our load balancers are not responding. We have failed over the external API to a failover service so requests are being captured.