Resolved -
Between March 27, 2025, 23:45 UTC and March 28, 2025, 01:40 UTC the Pull Requests service was degraded and failed to update refs for repositories with higher traffic activity. This was due to a large repository migration that resulted in a larger than usual number of enqueued jobs; while simultaneously impacting git fileservers where the problematic repository was hosted. This resulted in an increase in queue depth due to retries on failures to perform those jobs causing delays for non-migration sourced jobs.
We declared an incident once we confirmed that this issue was not isolated to the problematic migration and other repositories were also failing to process ref updates.
We mitigated the issue by stopping the migration and short circuiting the remaining jobs. Additionally, we increased the worker pool of this job to reduce the time required to recover.
As a result of this incident, we are revisiting our repository migration process and are working to isolate potentially problematic migration workloads from non-migration workloads.
Mar 28, 01:40 UTC
Update -
This issue has been mitigated and we are operating normally.
Mar 28, 01:40 UTC
Update -
We are continuing to monitor for recovery.
Mar 28, 00:54 UTC
Update -
We believe we have identified the source of the issue and are monitoring for recovery.
Mar 28, 00:20 UTC
Investigating -
We are investigating reports of degraded performance for Pull Requests
Mar 27, 23:52 UTC