Skip to content

Reap the bounded runner's leader only after its process group is retired - #315

Merged
Deicyde merged 2 commits into
mainfrom
fix/skeleton-reap-leader-last
Oct 14, 2026
Merged

Deicyde merged 2 commits into
mainfrom
fix/skeleton-reap-leader-last

Conversation

@Deicyde

@Deicyde Deicyde commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Fixes overnight-review finding PB1 (item 10 of the overnight review report): the skeleton's bounded command runner signalled Lake's process group after reaping Lake, which #168 forbids.

The bug

_run_registered_command waited for Lake with Popen.poll(), which reaps it, and then still signalled Lake's group:

  • after the run, _process_group_is_alive(pid) called killpg(pid, 0) to see whether stragglers were left;
  • in _terminate_process_tree, the SIGKILL phase called killpg(pid, SIGKILL) after the SIGTERM phase's Popen.wait(), or Popen.terminate() (which polls first), had reaped the leader.

Once the leader is reaped and its group is empty, the group id can pass to an unrelated group. killpg(pid, 0) could then report stragglers that are not there, and the cleanup that follows sends SIGTERM and SIGKILL to that unrelated group. The servers' process code (servers/repl/core.py) keeps the leader unreaped until its group is retired. #214 runs probes concurrently, which makes the reuse more likely, so this lands first.

The fix

The runner leaves the leader unreaped until the group is retired, as the servers do:

  • _leader_has_exited sees the exit without reaping: os.waitid(P_PID, pid, WEXITED | WNOHANG | WNOWAIT) where Python has it, and psutil's zombie status otherwise (macOS on Python 3.10). On Linux a zombie status alone is not enough: a leader whose main thread exits while other threads run shows as a zombie, while waitid waits for every thread. If something else already reaped the leader, for example the kernel when the launcher ignores SIGCHLD, poll() records it on either path (ECHILD from waitid, NoSuchProcess from psutil), so nothing signals the group again; once a reap is recorded, the success path skips the group check, since the id is free.
  • _process_group_has_live_members counts the group's members without zombies, like the servers' helper of the same name, and fails closed when the process table cannot be read.
  • The group is signalled only while the leader is unreaped. On POSIX the leader is reached through killpg alone (Windows keeps Popen.terminate and kill), and it is reaped last.
  • The success path reaps the leader after the group check, with the signal guard disarmed for the reap, so no signal lands between the reap and the recorded status.
  • _remember_descendants no longer reads a reaped leader's PID, and the token scan runs after the group scan, so a child forked during the scan is still seen.

Validation

Push run 38629621769 on 4f051572 passed every job: Windows inspection, subprocess, and publication hardening, real Lean (fixture toolchain), real REPL (fixture toolchain), test (3.10), test (3.13).

  • New test_bounded_command_signals_its_group_only_while_the_leader_is_unreaped: it wraps os.killpg and fails if the group is signalled once the leader is reaped, checking with the system that the leader still exists rather than trusting Popen's recorded status. It covers a command that exits, one a child outlives, and one that times out, each with and without os.waitid (as on macOS before Python 3.13) and with SIGCHLD at its default or ignored: 12 cases.
  • New test_bounded_command_defers_a_signal_that_arrives_while_the_leader_is_reaped raises SIGTERM inside the success-path reap; dropping either the guard's disarm or its arm there fails it.
  • The bounded-command tests pass on Python 3.13 and 3.10 locally, and the full suite passes locally.
  • Before opening, two adversarial reviewers checked the change, and a third checked the revision against their findings. A later adversarial review (two reviewers, each finding checked twice) found the psutil path's unrecorded reap and the untested reap window; 4f05157 fixes both, and each of four mutants (the old psutil branch, the group check after a recorded reap, and dropping the disarm or the arm) fails a test.

Merges cleanly with the #212-#219/#284 stack (#213, #214 and the stack tip).

The skeleton's bounded runner waited for Lake with Popen.poll(), which
reaps it, and then still signalled Lake's process group: killpg(pid, 0)
to see whether stragglers were left, and, between the SIGTERM and
SIGKILL phases of the cleanup, killpg after Popen.wait() or
Popen.terminate() (which polls first) had reaped the leader. Once the
leader is reaped and the group is empty, its id can pass to an unrelated
group, which the next killpg would signal. #168 forbids signalling a
group after its leader is reaped, and the servers' process code follows
that rule; concurrent probes make the reuse more likely.

The runner now leaves the leader unreaped until the group is retired. It
sees the exit with waitid(WNOWAIT) where Python has it, which, unlike a
zombie status on Linux, waits for every thread, and with psutil's zombie
status otherwise, and if something else reaped the leader, poll records
it so nothing signals the group again; it counts the group's members
without zombies, failing closed when the process table cannot be read;
it signals the group only while the leader is unreaped and reaches the
leader through killpg alone on POSIX; and it reaps the leader last, with
no signal allowed between the reap and the recorded status. Descendants
are no longer read from a reaped leader's PID, and the token scan runs
after the group scan, so a child forked during it is still seen.

A test wraps os.killpg and fails if the group is signalled after the
leader was reaped: for a command that exits, one a child outlives, and
one that times out. All three fail on main. Another runs a command with
os.waitid removed, as on macOS before Python 3.13.
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Oct 14, 2026
… reap window

Follow-up from an adversarial review of this PR.

- Without os.waitid (macOS before Python 3.13), _leader_has_exited
  answered True for a leader something else had reaped, such as the
  kernel when the launcher ignores SIGCHLD, without recording it. The
  group check and cleanup then signalled the freed id. psutil's
  NoSuchProcess now records the reap through poll, as ECHILD does on the
  waitid path; a zombie still counts as exited and unreaped.
- Once a reap is recorded, the success path no longer checks the group:
  its id is free.
- The group-signalling test runs with and without os.waitid and with
  SIGCHLD ignored, and its oracle asks the system whether the leader is
  still there instead of trusting Popen's recorded status. It replaces
  the test that only ran a passing command without os.waitid.
- A new test raises SIGTERM inside the success-path reap: dropping
  either the guard's disarm or its arm there now fails it.
@Deicyde

Deicyde commented Oct 14, 2026

Copy link
Copy Markdown
Contributor Author

Review: ready to merge. Land it before #214.

I took this PR over from its author and reviewed it adversarially with two reviewers, covering necessity, architecture, simpler alternatives, style and correctness. Two more checked every blocker or should-fix.

  • Design. Both found it necessary and right: keeping the leader unreaped until its group is retired is what servers/repl/core.py does.
  • No simpler option is as safe. On Linux a zombie status alone is not enough, so waitid is needed there.

They confirmed three should-fix items, all fixed in 4f05157:

  1. Unrecorded reap on the psutil path. Without os.waitid (macOS before Python 3.13), the runner treated a leader reaped elsewhere as exited without recording the reap. The kernel does this when the launcher ignores SIGCHLD, for example. The group check and cleanup could then still signal the freed id, which is the PB1 hazard itself; a reviewer reproduced it.
    • NoSuchProcess now records the reap through poll(), as ECHILD already did on the waitid path. A zombie still counts as exited and unreaped.
    • Once a reap is recorded, the success path skips the group check.
  2. Untested reap window (N4). A new test raises SIGTERM inside the success-path reap. Deleting either the guard's disarm() or its arm() now fails it.
  3. Fallback coverage. The group-signalling test now runs 12 cases: each command with and without os.waitid, and with SIGCHLD at its default or ignored.
    • Its oracle asks the system whether the leader still exists, instead of trusting Popen's recorded status.
    • It replaces the test that only ran a passing command without os.waitid.

Verification.

  • Four mutants each fail a test: the old psutil branch, the group check after a recorded reap, dropping the disarm, and dropping the arm.
  • tests/test_skeleton.py passes locally (173 passed, without Lean).
  • CI passes every job on 4f05157 (push run, pull request run).
  • It merges cleanly with current main, and the description is updated.

Left as is, not blocking:

  • N3, a fork chain during the group scan, which no finite scan can close;
  • a small golf: replacing _process_group_is_alive with an os.name check.

I'm marking it ready for review. It should land before #214, whose concurrent probes make PID reuse likelier.

@Deicyde
Deicyde marked this pull request as ready for review October 14, 2026 20:43
@Deicyde
Deicyde merged commit e108ee0 into main Oct 14, 2026
11 checks passed
Deicyde added a commit that referenced this pull request Oct 17, 2026
When SIGCHLD is ignored, the kernel reaps every child as it exits and
CPython's Popen reports each exit status as 0. Linux keeps an ignored
SIGCHLD across exec, so a launcher can leave it that way. The bounded
runner then read Lake's stale-artifact exit 3, and a failed `lake env
lean` run, as successes, and killing stragglers did not make those runs
trustworthy.

- `autoform` restores the default SIGCHLD disposition at startup.
  Autoform waits for its own children, so nothing relied on the ignored
  one.
- A library caller that runs a bounded command while SIGCHLD is ignored
  gets a SkeletonError instead of made-up exit statuses.
- signal.getsignal can't see a disposition set from C, so the runner
  also refuses a command that something else reaped, and sends its
  group no signal.
- The reaped-leader branch goes. #315's rule stands, matching the
  servers (#168): a reaped leader's group is never signalled.
- Tests: a refusal test for each of those two paths, a CLI test, and the
  group-only child test kept at the default disposition as a regression
  guard for the group path. The group-signalling test no longer runs
  with SIGCHLD ignored.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant