Unable to load webpages

Incident Report for Substack

Resolved

From 5:30am to 7:07am PT we had a full website outage. During that time we experienced increased load that our spam detection system failed to appropriately scale to match. During this time the spam detection system continued to work correctly (blocking spammers) but lead to cascading issues across the service. We had a confounding issue with our alerting system that delayed our team's progress.

We've since addressed the root cause and are working to minimize the cross-dependencies in these systems. We have also improved our alerting system to help us navigate these issues quicker in the future.

Posted Mar 02, 2020 - 05:30 PST