Repository navigation
Reap the bounded runner's leader only after its process group is retired - #315
Merged
Merged
Conversation
The skeleton's bounded runner waited for Lake with Popen.poll(), which reaps it, and then still signalled Lake's process group: killpg(pid, 0) to see whether stragglers were left, and, between the SIGTERM and SIGKILL phases of the cleanup, killpg after Popen.wait() or Popen.terminate() (which polls first) had reaped the leader. Once the leader is reaped and the group is empty, its id can pass to an unrelated group, which the next killpg would signal. #168 forbids signalling a group after its leader is reaped, and the servers' process code follows that rule; concurrent probes make the reuse more likely. The runner now leaves the leader unreaped until the group is retired. It sees the exit with waitid(WNOWAIT) where Python has it, which, unlike a zombie status on Linux, waits for every thread, and with psutil's zombie status otherwise, and if something else reaped the leader, poll records it so nothing signals the group again; it counts the group's members without zombies, failing closed when the process table cannot be read; it signals the group only while the leader is unreaped and reaches the leader through killpg alone on POSIX; and it reaps the leader last, with no signal allowed between the reap and the recorded status. Descendants are no longer read from a reaped leader's PID, and the token scan runs after the group scan, so a child forked during it is still seen. A test wraps os.killpg and fails if the group is signalled after the leader was reaped: for a command that exits, one a child outlives, and one that times out. All three fail on main. Another runs a command with os.waitid removed, as on macOS before Python 3.13.
This was referenced Oct 14, 2026
… reap window Follow-up from an adversarial review of this PR. - Without os.waitid (macOS before Python 3.13), _leader_has_exited answered True for a leader something else had reaped, such as the kernel when the launcher ignores SIGCHLD, without recording it. The group check and cleanup then signalled the freed id. psutil's NoSuchProcess now records the reap through poll, as ECHILD does on the waitid path; a zombie still counts as exited and unreaped. - Once a reap is recorded, the success path no longer checks the group: its id is free. - The group-signalling test runs with and without os.waitid and with SIGCHLD ignored, and its oracle asks the system whether the leader is still there instead of trusting Popen's recorded status. It replaces the test that only ran a passing command without os.waitid. - A new test raises SIGTERM inside the success-path reap: dropping either the guard's disarm or its arm there now fails it.
Contributor
Author
|
Review: ready to merge. Land it before #214. I took this PR over from its author and reviewed it adversarially with two reviewers, covering necessity, architecture, simpler alternatives, style and correctness. Two more checked every blocker or should-fix.
They confirmed three should-fix items, all fixed in 4f05157:
Verification.
Left as is, not blocking:
I'm marking it ready for review. It should land before #214, whose concurrent probes make PID reuse likelier. |
Deicyde
marked this pull request as ready for review
October 14, 2026 20:43
This was referenced Oct 14, 2026
Deicyde
added a commit
that referenced
this pull request
Oct 17, 2026
When SIGCHLD is ignored, the kernel reaps every child as it exits and CPython's Popen reports each exit status as 0. Linux keeps an ignored SIGCHLD across exec, so a launcher can leave it that way. The bounded runner then read Lake's stale-artifact exit 3, and a failed `lake env lean` run, as successes, and killing stragglers did not make those runs trustworthy. - `autoform` restores the default SIGCHLD disposition at startup. Autoform waits for its own children, so nothing relied on the ignored one. - A library caller that runs a bounded command while SIGCHLD is ignored gets a SkeletonError instead of made-up exit statuses. - signal.getsignal can't see a disposition set from C, so the runner also refuses a command that something else reaped, and sends its group no signal. - The reaped-leader branch goes. #315's rule stands, matching the servers (#168): a reaped leader's group is never signalled. - Tests: a refusal test for each of those two paths, a CLI test, and the group-only child test kept at the default disposition as a regression guard for the group path. The group-signalling test no longer runs with SIGCHLD ignored.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes overnight-review finding PB1 (item 10 of the overnight review report): the skeleton's bounded command runner signalled Lake's process group after reaping Lake, which #168 forbids.
The bug
_run_registered_commandwaited for Lake withPopen.poll(), which reaps it, and then still signalled Lake's group:_process_group_is_alive(pid)calledkillpg(pid, 0)to see whether stragglers were left;_terminate_process_tree, the SIGKILL phase calledkillpg(pid, SIGKILL)after the SIGTERM phase'sPopen.wait(), orPopen.terminate()(which polls first), had reaped the leader.Once the leader is reaped and its group is empty, the group id can pass to an unrelated group.
killpg(pid, 0)could then report stragglers that are not there, and the cleanup that follows sends SIGTERM and SIGKILL to that unrelated group. The servers' process code (servers/repl/core.py) keeps the leader unreaped until its group is retired. #214 runs probes concurrently, which makes the reuse more likely, so this lands first.The fix
The runner leaves the leader unreaped until the group is retired, as the servers do:
_leader_has_exitedsees the exit without reaping:os.waitid(P_PID, pid, WEXITED | WNOHANG | WNOWAIT)where Python has it, and psutil's zombie status otherwise (macOS on Python 3.10). On Linux a zombie status alone is not enough: a leader whose main thread exits while other threads run shows as a zombie, whilewaitidwaits for every thread. If something else already reaped the leader, for example the kernel when the launcher ignores SIGCHLD,poll()records it on either path (ECHILD fromwaitid,NoSuchProcessfrom psutil), so nothing signals the group again; once a reap is recorded, the success path skips the group check, since the id is free._process_group_has_live_memberscounts the group's members without zombies, like the servers' helper of the same name, and fails closed when the process table cannot be read.killpgalone (Windows keepsPopen.terminateandkill), and it is reaped last._remember_descendantsno longer reads a reaped leader's PID, and the token scan runs after the group scan, so a child forked during the scan is still seen.Validation
Push run 38629621769 on
4f051572passed every job:Windows inspection, subprocess, and publication hardening,real Lean (fixture toolchain),real REPL (fixture toolchain),test (3.10),test (3.13).test_bounded_command_signals_its_group_only_while_the_leader_is_unreaped: it wrapsos.killpgand fails if the group is signalled once the leader is reaped, checking with the system that the leader still exists rather than trusting Popen's recorded status. It covers a command that exits, one a child outlives, and one that times out, each with and withoutos.waitid(as on macOS before Python 3.13) and with SIGCHLD at its default or ignored: 12 cases.test_bounded_command_defers_a_signal_that_arrives_while_the_leader_is_reapedraises SIGTERM inside the success-path reap; dropping either the guard's disarm or its arm there fails it.Merges cleanly with the #212-#219/#284 stack (#213, #214 and the stack tip).