FireHydrant experienced a prolonged delay of runbook executions due to a sudden outlier amount of traffic to our services. This traffic caused a shared worker queue to flood and go into a "spin lock" preventing other work from executing, such as runbook attachment. Existing incidents were not impacted from this incident, and our systems have been scaled to handle traffic spikes like the one experienced in the future.
Posted Jul 23, 2024 - 19:17 UTC
Update
All backlogs have successfully cleared and Runbooks are functioning as expected
Posted Jul 23, 2024 - 18:22 UTC
Update
We've processed the backlog of Runbook attachments and are seeing Runbooks execute as expected
Posted Jul 23, 2024 - 17:43 UTC
Update
We've identified a backlog of runbook attachments and are working to process them
Posted Jul 23, 2024 - 17:35 UTC
Update
We are continuing to investigate an issue with runbooks not attaching to incidents automatically, and manually attached runbooks not executing
Posted Jul 23, 2024 - 17:15 UTC
Update
We're investigating an issue with Runbooks failing to attach to incidents