cthread teardown force-DETACHes a live worker without proof of termination (@@cmwshu.c:64 / @@ctdet.c)

[mgrossmann]

Summary

The cthread worker teardown issues DETACH ...,STAE=YES against a worker subtask that has
not proven it terminated. On the escalation path, cthread_worker_shutdown
(@@cmwshu.c) waits up to ~5 s (50 × 0.10 s) for the worker's ATTACH ECB
(task->termecb & 0x40000000) and then, if it has not posted, force-DETACHes anyway:

/* @@cmwshu.c */
for (i = 0; i < 50; i++) {
    if (task->termecb & 0x40000000) { if (task->tcb) cthread_detach(task); goto quit; }
    ...
    __asm__("STIMER WAIT,BINTVL==F'10'");
}
/* unable to shutdown subtask, abend the subtask */
if (task->tcb) cthread_detach(task);   /* <-- line 64: no termecb gate */

cthread_detach (@@ctdet.c) then issues DETACH (%1),STAE=YES whenever
task->tcb != 0, with no check that the subtask terminated and no owner check:

/* @@ctdet.c */
if (!task)      goto quit;
if (!task->tcb) goto quit;
rc = try(detach, task);   /* -> DETACH ...,STAE=YES unconditionally */

Force-DETACHing a still-active subtask abnormally terminates it → S33E.

Relationship to #6 / PR #7 (this is a residual gap, NOT a regression)

#6 was fixed by PR #7 (8ce34b3), which rewrote cthread_manager_term (@@cmterm.c)
to join the dispatch thread before free(mgr), removing the use-after-free / nested
S0C4. PR #7 did not touch @@cmwshu.c or @@ctdet.c, and it explicitly left
dispatch_thread_term unchanged on the bet that a healthy worker drains inside the
escalation window.

That bet loses when a worker is genuinely wedged (e.g. after repeated CGI faults): the 5 s
window expires and line 64 force-DETACHes it → S33E. This is a residual gap PR #7 did not
cover.

Confirmed on a 2026-07-22 httpd STC shutdown running libc.a dated 5 Jul 2026
(after PR #7, 8ce34b3, 2 Jul): the S33E recurred (httpd#122). A relink cannot fix it —
the un-gated DETACH exists in every version because PR #7 never touched this path.

(Filing a new issue rather than reopening #6: PR #7 landed and did what it claimed — the
manager-teardown join. Reopening would blur what #7 accomplished.)

Proposed fix

• Treat task->termecb & 0x40000000 as the single source of truth for "this TCB is
  gone." Issue DETACH only after it posts.
• If a worker will not drain, mark it STUCK, leave its storage, and let address-space
  termination reclaim it (log the leak) — the same "retain rather than risk use-after-
  free" tradeoff cthread_manager_term already makes for the dispatch thread.
• Make cthread_detach idempotent and owner/termination-checked: verify the TCB is
  still a subtask of the caller and has posted termecb before detaching; skip otherwise.
  (Today it is only weakly idempotent — it zeroes tcb so a sequential re-entry
  no-ops, but it is not owner-checked and does not protect against a stale / already-
  reclaimed tcb.)
• Never STAE=YES-detach a live worker.

Why a worker can miss the 5 s window (OPEN question — do not treat as explained)

Why the 2026-07-22 workers did not drain within the 5 s window is not explained. The
earlier failed()-unconditional-retry hypothesis is withdrawn: RTM most plausibly
denies a retry forbidden by SDWACLUP and proceeds with termination rather than stalling
(see #9). So the drain failure is an open question again.

One code-supported plausible mechanism (offered as a lead, not asserted as the
cause): a worker only observes POST_SHUTDOWN when it re-enters
cthread_wait(&work->wait) in cthread_worker_wait (@@cmwwat.c) — i.e. between
requests. While it is executing a handler, cthread_worker_shutdown's POST merely sets the
ECB; the worker will not act on it until it returns to the wait loop. A handler that is
slow, blocked, or wedged/faulting (httpd#122 shows repeated MVSMF99E on one URI — a
faulting endpoint) never re-enters the wait, blows through the 5 s window, and reaches the
force-DETACH at @@cmwshu.c:64. This is consistent with the log, not a confirmed root
cause; the wedged-handler behaviour itself belongs to the httpd/mvsmf side.

Impact / release scoping

The S33E occurs during shutdown of an address space that is terminating anyway. It is
ugly and produces an SVC dump, but no in-flight request or persistent data is lost —
the workers have stopped accepting new work and the region is coming down. That suggests a
robustness/cleanliness fix that can ship after v1.0.0, rather than a v1.0.0 blocker.
(Contrast #9, which stops the recovery exits from running the C runtime under a torn-down
CRT during that same teardown.)

Consumer impact

@@cmwshu.c / @@ctdet.c are shared cthread code — ufsd, ufsd-utils, ftpd, httpd,
mvsmf. The change only affects the teardown escalation path (a non-draining worker);
healthy workers that post termecb are unaffected.

References

• Residual of cthread manager teardown races on shutdown: active worker force-DETACHed (S33E) + nested abend-reporter S0C4 #6 / PR fix(thdmgr): join the dispatch thread on manager teardown (S33E/S0C4 shutdown race) #7 (@@cmterm.c join).
• Observed: httpd#122 (2026-07-22 STC shutdown, S33E on TCB=009B29E8).
• Epic: Epic: v1.0.0 release readiness — console S0C4, worker teardown, recovery exits mvsmf#177 (Workstream B).
• Related: the recovery-exit hardening in Recovery exits run the C runtime at termination — check SDWACLUP, emit one WTO, RC=0 #9 removes the secondary fallout (recovery
  exits running the C runtime under a torn-down CRT) but not this S33E — the two are
  independent. Sequence: Recovery exits run the C runtime at termination — check SDWACLUP, emit one WTO, RC=0 #9 first, this second (separate PRs).

[mgrossmann]

Status: still open, not fixed. #11 is the detach-gating half — teardown
force-DETACHes a worker without proof of termination (@@cmwshu.c:64 /
@@ctdet.c). Neither #6 (closed) nor the recovery-exit work (#9/#10, #16)
addresses it: the recovery exits only make the secondary fault safe, while the
force-DETACH that raises the S33E is untouched. Deferred to post-holiday.