Incident
Login failures on API and Web App
- Started
- Resolved
- (7h 41m)
- Affected
- API, Web App
Status of the affected services
The status for each minute, from 2 hours before the start to 2 hours after the end.
API
Degraded performance- Partial outage – (3h 1m)
- Degraded performance – (2h 57m)
- Operational – (12m)
- Degraded performance – (1h 32m)
- OperationalFrom
Web App
Operational- Partial outage – (3h 1m)
- Degraded performance – (2h 57m)
- OperationalFrom
Updates
Resolved
This incident is resolved. All access tokens have refreshed and the API and Web App are both operating normally. Sign-ins and API authentication are working as expected. We're sorry for the disruption and will share a follow-up on how we're preventing expired signing keys from causing this again.
Monitoring
The Web App has fully recovered and sign-ins are working normally. Some API clients are still seeing occasional authentication errors while older access tokens are refreshed. We're continuing to monitor the API and will update within 30 minutes.
Monitoring
This incident is resolved. After we rotated the expired authentication signing key, sign-ins and API requests have stayed normal, and the API and Web App are fully operational. We're adding automatic alerts and rotation ahead of key expiry so this can't happen again. We're sorry for the disruption.
Monitoring
We've rotated to a new signing key and sign-ins are succeeding again. Some users may still see slower responses or need to sign in again while sessions refresh. We're monitoring closely and will update within 30 minutes.
Identified
We've identified the cause. An authentication signing key expired, so the API began rejecting sign-ins and access tokens. We're rotating to a new key now. Users who are already signed in may still be affected until their session refreshes. We'll share another update within 30 minutes.
Investigating
We're investigating reports that users can't sign in to the Web App and that API requests are failing authentication. Users who are already signed in may not be affected. We'll share another update within 30 minutes.