At 5:36pm ET on August 26, 2020 Braze Engineering was performing maintenance on the database schema underlying Content Cards in EU 01. As part of this maintenance, some unused database columns were dropped. The maintenance plan involved removing and recreating a database view which previously used these columns. The view was incorrectly recreated which caused the database query planner to change the indexes used to respond to queries, which impacted Content Cards sending performance in EU 01 severely. Braze engineering paused the sending of Content Cards on EU 01 from 6:57pm ET until 9:45pm ET while we identified and debugged the issue of correcting the view. Content Cards sending was fully restored at 9:50pm ET.

This maintenance had been run on staging as well as other smaller production environments successfully prior to this change. This was not caught in other environments because the lesser Card volume there did not cause the impacted query to meaningfully change its performance. We apologize for this issue and are implementing tests to ensure any other database schema updates don’t impact content cards sending in this manner.