Repository navigation
fleet: keep Linux runners off the CPUs the kernel isolates - #6485
Merged
Merged
Conversation
A host booted with isolcpus keeps those CPUs out of the scheduler's
balancing, but systemd starts every unit on all CPUs, so a runner's job
threads could land on an isolated CPU and stay there, every thread of the
job sharing one core (measured on a 32-CPU host with isolcpus=1-15: mypy,
go and Runner.Worker on CPU 1, jobs of 26-57 s taking 118-133 s and failed
by the 2-minute rule).
The role reads /sys/devices/system/cpu/{online,isolated} (in --check too)
and the unit sets CPUAffinity= to the online CPUs less the isolated ones.
A host that isolates none gets no affinity line, so its units are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A host booted with
isolcpuskeeps those CPUs out of the scheduler's balancing, but systemd starts every unit on all CPUs, so a runner's job threads could land on an isolated CPU and stay there, every thread of that job sharing one core. Measured on a 32-CPU host with isolcpus=1-15 (2026-11-17): mypy, go and Runner.Worker on CPU 1; jobs that take 26-57 s took 118-133 s and were failed by the 2-minute rule.The github-runner role reads
/sys/devices/system/cpu/{online,isolated}(in--checktoo) and the unit setsCPUAffinity=to the online CPUs less the isolated ones (github_runner_cpus). A host that isolates none gets no affinity line, so its units are unchanged (checked: amd-1/2/3, intel, hetzner1/2, vision isolate none). A host that isolates every CPU is refused with a remedy.Tests:
TestRunnerUnitNamesTheBalancedCPUs(unit, structural) andTestRunnerServiceKeepsOffIsolatedCPUs(functional, renders through ansible's templar: 0-31 less 1-15, a mixed list, none isolated, nothing read). README github-runners section documents it.Dry run on the affected host with a build of this branch showed only the unit diff (plus the CPUWeight/IOWeight/Nice lines already released in v2.0.73 that host had not been re-run with).
🤖 Generated with Claude Code