Skip to content

fleet: keep Linux runners off the CPUs the kernel isolates - #6485

Merged
gafferongames merged 1 commit into
mainfrom
fleet/runner-cpu-affinity
Nov 17, 2026
Merged

gafferongames merged 1 commit into
mainfrom
fleet/runner-cpu-affinity

Conversation

@gafferongames

Copy link
Copy Markdown
Contributor

A host booted with isolcpus keeps those CPUs out of the scheduler's balancing, but systemd starts every unit on all CPUs, so a runner's job threads could land on an isolated CPU and stay there, every thread of that job sharing one core. Measured on a 32-CPU host with isolcpus=1-15 (2026-11-17): mypy, go and Runner.Worker on CPU 1; jobs that take 26-57 s took 118-133 s and were failed by the 2-minute rule.

The github-runner role reads /sys/devices/system/cpu/{online,isolated} (in --check too) and the unit sets CPUAffinity= to the online CPUs less the isolated ones (github_runner_cpus). A host that isolates none gets no affinity line, so its units are unchanged (checked: amd-1/2/3, intel, hetzner1/2, vision isolate none). A host that isolates every CPU is refused with a remedy.

Tests: TestRunnerUnitNamesTheBalancedCPUs (unit, structural) and TestRunnerServiceKeepsOffIsolatedCPUs (functional, renders through ansible's templar: 0-31 less 1-15, a mixed list, none isolated, nothing read). README github-runners section documents it.

Dry run on the affected host with a build of this branch showed only the unit diff (plus the CPUWeight/IOWeight/Nice lines already released in v2.0.73 that host had not been re-run with).

🤖 Generated with Claude Code

A host booted with isolcpus keeps those CPUs out of the scheduler's
balancing, but systemd starts every unit on all CPUs, so a runner's job
threads could land on an isolated CPU and stay there, every thread of the
job sharing one core (measured on a 32-CPU host with isolcpus=1-15: mypy,
go and Runner.Worker on CPU 1, jobs of 26-57 s taking 118-133 s and failed
by the 2-minute rule).

The role reads /sys/devices/system/cpu/{online,isolated} (in --check too)
and the unit sets CPUAffinity= to the online CPUs less the isolated ones.
A host that isolates none gets no affinity line, so its units are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gafferongames
gafferongames merged commit d9cef86 into main Nov 17, 2026
21 of 25 checks passed
@gafferongames
gafferongames deleted the fleet/runner-cpu-affinity branch November 17, 2026 15:52
@gafferongames gafferongames mentioned this pull request Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants