Resolved -
FireHydrant experienced a prolonged delay of runbook executions due to a sudden outlier amount of traffic to our services. This traffic caused a shared worker queue to flood and go into a "spin lock" preventing other work from executing, such as runbook attachment. Existing incidents were not impacted from this incident, and our systems have been scaled to handle traffic spikes like the one experienced in the future.
Jul 23, 19:17 UTC
Update -
All backlogs have successfully cleared and Runbooks are functioning as expected
Jul 23, 18:22 UTC
Update -
We've processed the backlog of Runbook attachments and are seeing Runbooks execute as expected
Jul 23, 17:43 UTC
Update -
We've identified a backlog of runbook attachments and are working to process them
Jul 23, 17:35 UTC
Update -
We are continuing to investigate an issue with runbooks not attaching to incidents automatically, and manually attached runbooks not executing
Jul 23, 17:15 UTC
Update -
We're investigating an issue with Runbooks failing to attach to incidents
Jul 23, 16:40 UTC
Investigating -
We are currently investigating this issue.
Jul 23, 16:39 UTC