[arekay-nv]

What does this PR do?

Makes test_run_timeout_during_metrics_drain_interrupts deterministic. It is a test-only change and touches no product code.

Root cause. run_benchmark computes the run deadline before services launch. Service startup takes about 2.4–3.2 s, longer than the test's 2.5 s run_timeout_s.

Where the watchdog fired	Path	Outcome
Service launch (most CI failures)	task.cancel() → MetricsPipeline.start → terminate_all SIGTERM, which arrives before the aggregator installs its handler	No final_snapshot.json → aggregator must still write a final snapshot
HTTP client setup (accidental passes)	_kill_services SIGTERM → the aggregator's handler	interrupted snapshot, but the drain path was never exercised

What changed

• A wall-clock deadline can't reliably land in the drain window, so the test no longer uses one. A timer-less RunWatchdog subclass queues the real _fire at the drain's first suspension. It only fires if the run hasn't called watchdog.cancel(), so the test still checks that the watchdog stays armed through the drain.
• The assertions now follow the documented run-outcome precedence (AGENTS.md "Run-outcome precedence"; the ^C twin is test_sigint_in_drain_window_keeps_summary_authoritative):
  • the error matches Run timeout .*report marked INTERRUPTED;
  • the fire reached the drain as an abort (abort_event set);
  • result_summary.json reads state=interrupted with complete: false;
  • final_snapshot.json reads state=complete with n_pending_tasks == 0, because ENDED had already gone out without the INTERRUPTED marker.
• The dataset is cut from 100 to 30 samples, and interrupted_teardown_grace_s=120 leaves room for the drain to finish.

Out of scope (separate follow-ups):

• The aggregator's pre-handler SIGTERM window, and the orphaned shard workers it leaves behind (issue options 3 and 7).
• Sending SIGTERM before SIGKILL when the grace expires (option 2). This would change the precedence contract.
• test_run_timeout_produces_interrupted_report has a 6 s budget, which is close to startup time.

Type of change

• Bug fix (flaky test)
• New feature
• Documentation update
• Refactor/cleanup

Related issues

Closes #553

Testing

• Tests added/updated
• All tests pass locally (macOS; CPU affinity patched off)
• Manual testing completed

Check	Result
Original test	Fails with the CI signature (watchdog fires about 2.4 s after launch, no snapshot)
New test	Passed 4/4 (~11 s each)
Mutation: watchdog.cancel() before the drain	Caught (DID NOT RAISE)
Mutation: drop abort_event.set() in _fire	Caught ([False] == [True])
tests/integration/commands/test_run_timeout.py	5 passed

Linux CI (affinity on, --cov) is still pending.

Checklist

• Code follows project style
• Pre-commit hooks pass (apart from mypy's known macOS-only sched_*affinity errors in src/)
• Documentation updated (if needed): n/a, the contract is already documented in AGENTS.md

🤖 Generated with Claude Code

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅