Cubbit - Quota events are not aggregated – Incident details

All systems operational

Quota events are not aggregated

Resolved
Major outage
Started about 1 month agoLasted 23 days

Affected

Report and Quota API

Major outage from 10:15 AM to 3:43 PM, Partial outage from 3:43 PM to 10:08 AM, Degraded performance from 10:08 AM to 10:40 AM, Partial outage from 10:40 AM to 4:41 PM, Degraded performance from 4:41 PM to 10:43 AM

Updates
  • Resolved
    Resolved

    All problems have been resolved, and automation has been restored.

    The quotas have been aggregated correctly for the entire period during which service was affected by the incident.

    The current aggregation delay is approximately 12 hours

  • Update
    Update

    Quota aggregation has been reactivated. We are monitoring the situation. We thank you for your patience.

  • Update
    Update

    Quota aggregation is temporarily paused to ensure all events are collected before processing. We are monitoring the situation and will resume aggregation soon. Thank you for your patience.

  • Update
    Update

    The system is up and running, quotas are aggregated correctly though with more latency than usual

  • Update
    Update

    The problem has been identified and solved. We are currently monitoring the disposal of event queues that accumulated during the period when the service was affected by the bug.

    Quota aggregation will be reactivated as soon as the queue has been consumed.

  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Identified
    Identified

    Transaction events are excluded from the aggregation of the resources quota.

    We are currently investigating this incident.