Tilda Corporation
← Back to status

Incident

Login failures on API and Web App

ResolvedMajor impact
Started
Resolved
(7h 41m)
Affected
API, Web App

Status of the affected services

The status for each minute, from 2 hours before the start to 2 hours after the end.

  • API

    Degraded performance
    • Partial outage – (3h 1m)
    • Degraded performance – (2h 57m)
    • Operational – (12m)
    • Degraded performance – (1h 32m)
    • OperationalFrom
  • Web App

    Operational
    • Partial outage – (3h 1m)
    • Degraded performance – (2h 57m)
    • OperationalFrom

Updates

  1. Resolved

    This incident is resolved. All access tokens have refreshed and the API and Web App are both operating normally. Sign-ins and API authentication are working as expected. We're sorry for the disruption and will share a follow-up on how we're preventing expired signing keys from causing this again.

  2. Monitoring

    The Web App has fully recovered and sign-ins are working normally. Some API clients are still seeing occasional authentication errors while older access tokens are refreshed. We're continuing to monitor the API and will update within 30 minutes.

  3. Monitoring

    This incident is resolved. After we rotated the expired authentication signing key, sign-ins and API requests have stayed normal, and the API and Web App are fully operational. We're adding automatic alerts and rotation ahead of key expiry so this can't happen again. We're sorry for the disruption.

  4. Monitoring

    We've rotated to a new signing key and sign-ins are succeeding again. Some users may still see slower responses or need to sign in again while sessions refresh. We're monitoring closely and will update within 30 minutes.

  5. Identified

    We've identified the cause. An authentication signing key expired, so the API began rejecting sign-ins and access tokens. We're rotating to a new key now. Users who are already signed in may still be affected until their session refreshes. We'll share another update within 30 minutes.

  6. Investigating

    We're investigating reports that users can't sign in to the Web App and that API requests are failing authentication. Users who are already signed in may not be affected. We'll share another update within 30 minutes.