At 8:00 AM Pacific this morning, our monitoring reported severely degraded performance of the Substack website. All instances of our web application were serving requests very slowly. By 8:20 AM performance had degraded so much that up to 30% of all requests to our web application were timing out rather than delivering useful responses.

Our team addressed the issue by pulling load off of our database, which is the common critical element shared by individual application instances. By 8:50 performance had improved significantly but was still far below our target speeds. Performance remained uneven and slow from 8:50 until 9:30, when we disabled a feature of our application that was known to run expensive and frequent database queries. After this the performance of our application returned to normal.