Tilda Corporation
← Back to status

Incident

Elevated API errors and slow Web App performance

ResolvedMajor impact
Started
Resolved
(20h 42m)
Affected
API, Web App

Status of the affected services

The status for each minute, from 2 hours before the start to 2 hours after the end.

  • API

    Degraded performance
    • Partial outage – (20h 43m)
    • Degraded performance – (1m)
    • OperationalFrom
  • Web App

    Operational
    • Partial outage – (20h 43m)
    • Degraded performance – (1m)
    • OperationalFrom

Updates

  1. Resolved

    This incident is resolved. After rolling back the configuration change that limited the API's database connections, error rates and response times have stayed normal, and the API and Web App are fully operational. We're adding a check so a change like this can't reach production without enough connection capacity. We're sorry for the disruption.

  2. Monitoring

    The rollback is complete and the API has its full set of database connections again. Error rates are back to normal and response times are recovering, and the Web App should feel normal for most people. We're watching closely to make sure things stay stable. Next update within 30 minutes.

  3. Identified

    We've identified the cause: a configuration change deployed earlier today reduced the number of database connections available to the API, so some requests were failing or timing out under normal load. We're rolling back that change now. The Web App may stay slow until the rollback completes. Next update within 30 minutes.

  4. Investigating

    We're investigating elevated error rates and increased latency on the API, starting around 2:20 PM PT. Some requests are failing with 502 errors, and the Web App may feel slow as a result. We'll post another update within 30 minutes.