AU - Incident with Actions
Incident
Report for GitHub Enterprise Cloud Australia
Resolved
On September 24, between 15:00 and 17:25 UTC, customers using the Australia stamp of GitHub Enterprise Cloud experienced failures and delays when running workflows on GitHub Actions. During this period, processing of Actions webhook events and scheduled invocations was paused. After the incident was resolved, delayed webhooks were delivered, and approximately 98% of Actions events were eventually processed.
The incident was caused by an existing misconfiguration in our stream processing system preventing automated recovery when multiple brokers went unavailable. When these failures occurred simultaneously, the misconfigured threshold prevented jobs from being enqueued.
We have corrected this configuration across all environments, and implemented tests which ensure future configuration conformance and prevent similar issues.
The incident was caused by an existing misconfiguration in our stream processing system preventing automated recovery when multiple brokers went unavailable. When these failures occurred simultaneously, the misconfigured threshold prevented jobs from being enqueued.
We have corrected this configuration across all environments, and implemented tests which ensure future configuration conformance and prevent similar issues.
Update
We have identified the issue with some hosts in our infrastructure and are working on mitigating the issue.
Update
We are investigating reports of degraded performance in regards to starting Actions workflows.
Investigating
We are investigating reports of degraded performance for Actions
This incident affected: Actions.
Subscribe to our developer newsletter
Get tips, technical guides, and best practices. Twice a month. Right in your inbox.
Subscribe