Cubbit - Quota events are not aggregated – Incident details

All systems operational

Quota events are not aggregated

Resolved
Major outage
Started over 1 year agoLasted 552 hours 28 minutes

Affected

Coordinator [Europe]

Major outage from 10:15 AM to 3:43 PM, Partial outage from 3:43 PM to 10:08 AM, Degraded performance from 10:08 AM to 10:40 AM, Partial outage from 10:40 AM to 4:41 PM, Degraded performance from 4:41 PM to 10:43 AM

Report and Quota API

Major outage from 10:15 AM to 3:43 PM, Partial outage from 3:43 PM to 10:08 AM, Degraded performance from 10:08 AM to 10:40 AM, Partial outage from 10:40 AM to 4:41 PM, Degraded performance from 4:41 PM to 10:43 AM

Updates
  • Resolved
    UTC
    Resolved

    All problems have been resolved, and automation has been restored.

    The quotas have been aggregated correctly for the entire period during which service was affected by the incident.

    The current aggregation delay is approximately 12 hours

  • Update
    UTC
    Update

    Quota aggregation has been reactivated. We are monitoring the situation. We thank you for your patience.

  • Update
    UTC
    Update

    Quota aggregation is temporarily paused to ensure all events are collected before processing. We are monitoring the situation and will resume aggregation soon. Thank you for your patience.

  • Update
    UTC
    Update

    The system is up and running, quotas are aggregated correctly though with more latency than usual

  • Update
    UTC
    Update

    The problem has been identified and solved. We are currently monitoring the disposal of event queues that accumulated during the period when the service was affected by the bug.

    Quota aggregation will be reactivated as soon as the queue has been consumed.

  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Identified
    UTC
    Identified

    Transaction events are excluded from the aggregation of the resources quota.

    We are currently investigating this incident.