Unable to load webpages

Incident Report for Substack

Postmortem

We experienced website outages and connectivity issues this morning between 6:12am - 7:25am (PT) and again between 9:25am - 9:57am. We apologize for the inconvenience. We know how many of you rely on Substack on a daily basis and we're working to improve the stability and performance of the platform.

The root cause of the outage was database misconfigurations that were introduced with recent features for teams of writers. Coupled with extremely high load, these issues brought the website down at 6:12am. Our engineers were notified and began investigating at 6:14am, temporarily resolving some of the database issues at 7:25am when service was restored. At 9:25am we attempted to deploy instrumentation and optimizations to better handle the situation but this had cascading effects, causing the issue to return and disrupting website traffic once again. At 9:57am we were able to fully restore the service and website performance slowly returned to normal over the next half hour.

To prevent these issues from happening in the future, our engineers have released improvements to the system and have begun the process of upgrading our databases.

Posted Jan 28, 2020 - 02:01 PST

Resolved

This incident has been resolved.
Posted Jan 27, 2020 - 10:30 PST