Cubbit - Quota events are not aggregated – Incident details

Report and Quota API experiencing degraded performance

Quota events are not aggregated

Monitoring
Major outage
Started 17 days ago

Affected

Report and Quota API

Major outage from 10:15 AM to 3:43 PM, Partial outage from 3:43 PM to 10:08 AM, Degraded performance from 10:08 AM to 10:40 AM, Partial outage from 10:40 AM to 4:41 PM, Degraded performance from 4:41 PM to 12:00 AM

Updates
  • Update
    Update

    Quota aggregation has been reactivated. We are monitoring the situation. We thank you for your patience.

  • Update
    Update

    Quota aggregation is temporarily paused to ensure all events are collected before processing. We are monitoring the situation and will resume aggregation soon. Thank you for your patience.

  • Update
    Update

    The system is up and running, quotas are aggregated correctly though with more latency than usual

  • Update
    Update

    The problem has been identified and solved. We are currently monitoring the disposal of event queues that accumulated during the period when the service was affected by the bug.

    Quota aggregation will be reactivated as soon as the queue has been consumed.

  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Identified
    Identified

    Transaction events are excluded from the aggregation of the resources quota.

    We are currently investigating this incident.