Incident
Elevated API errors and slow Web App performance
- Started
- Resolved
- (20h 42m)
- Affected
- API, Web App
Status of the affected services
The status for each minute, from 2 hours before the start to 2 hours after the end.
API
Degraded performance- Partial outage – (20h 43m)
- Degraded performance – (1m)
- OperationalFrom
Web App
Operational- Partial outage – (20h 43m)
- Degraded performance – (1m)
- OperationalFrom
Updates
Resolved
This incident is resolved. After rolling back the configuration change that limited the API's database connections, error rates and response times have stayed normal, and the API and Web App are fully operational. We're adding a check so a change like this can't reach production without enough connection capacity. We're sorry for the disruption.
Monitoring
The rollback is complete and the API has its full set of database connections again. Error rates are back to normal and response times are recovering, and the Web App should feel normal for most people. We're watching closely to make sure things stay stable. Next update within 30 minutes.
Identified
We've identified the cause: a configuration change deployed earlier today reduced the number of database connections available to the API, so some requests were failing or timing out under normal load. We're rolling back that change now. The Web App may stay slow until the rollback completes. Next update within 30 minutes.
Investigating
We're investigating elevated error rates and increased latency on the API, starting around 2:20 PM PT. Some requests are failing with 502 errors, and the Web App may feel slow as a result. We'll post another update within 30 minutes.