fix(cache): include scheduler occupancy in owner pressure

[DavidBellamy]

Description

Problem

Cache-aware engine-load routing currently defines pressure as the larger of KV token usage and utilization. A Kimi K3 engine can therefore remain a suitable cached owner while its scheduler is full and has a large waiting queue, so cache affinity keeps feeding that owner while other healthy engines stay idle.

On the observed M2 workload, hot owners reached the 37-request cap with up to 116 waiting requests while most of the 50-engine pool was idle. This PR only changes routing pressure. It does not change adaptive-admission ordering or enable the feature in production.

Solution

Include scheduler occupancy in per-worker pressure when every healthy candidate reports a positive max-running cap for every DP rank:

pressure = max(token_usage, utilization, (running + waiting) / max_running)

The existing 90% high watermark, 10-point owner slack, replication ceiling, cooldown, and provisional-owner coalescing remain unchanged. If any candidate has missing, partial, or stale cap telemetry, the entire decision falls back to the existing backend pressure. This avoids false spills from incomplete multi-DP caps and from SGLang Prometheus fallback counts that may be duplicated across TP ranks.

Refs #11.

Changes

• Add scheduler occupancy to the cache-aware engine pressure plan.
• Sum complete request counts and caps across DP ranks.
• Fail open fleet-wide to backend pressure when scheduler caps are incomplete.
• Add regression coverage for below/above-watermark behavior, multi-DP aggregation, partial telemetry, TP-duplicated fallback counts, existing suitable owners, and saturated spill candidates.

Test Plan

Local verification:

• cargo +nightly fmt --all -- --check
• git diff --check
• cargo test -p smg policies::cache_aware --lib (53 passed)
• cargo test -p smg --lib (1,477 passed, 5 ignored)
• Changed-package clippy with unrelated Rust 1.94 baseline lints allowed

The repository-wide cargo clippy --all-targets --all-features -- -D warnings remains blocked on the unchanged base by existing Rust 1.94 lint failures and this Mac's missing pkg-config/OpenCV dependency. Repository-wide cargo test also fails compiling existing openai-protocol builder tests that omit the other field. Neither failure touches this diff.

Before production use, run an isolated M2 routing canary on the same TP16 engines and growing-prefix trace:

1. Compare engine-load routing disabled versus enabled with adaptive admission in shadow mode.
2. Promote only the passing arm to admission enforcement.
3. Require at least 3 of 4 engines active, bounded prefix-owner replication, full load/cap telemetry coverage, no increase in terminal errors or retractions, and no more than a 10 percentage-point cache-reuse loss.
4. Use successful output tokens per node and completed rollouts per hour as the primary gates. The target is at least 2x over the current owner-trapped baseline.

Checklist

• cargo +nightly fmt passes
• cargo clippy --all-targets --all-features -- -D warnings passes (blocked by unchanged base and local OpenCV tooling, detailed above)
• (Optional) Documentation updated
• (Optional) Please join us on Slack #sig-smg to discuss, review, and merge PRs

[DavidBellamy]

Canary update from the isolated two-engine Kimi K3 run:

• A valid production-SMG control completed at an exact 76-slot scheduler capacity: 228/228 turns, 228,000 output tokens, 104.969 output tok/s/node, zero terminal failures, 408s, retractions, OOMs, or restarts.
• Routing stayed 100:0 for the full trace. The cached owner handled every selection while the second healthy engine stayed idle.
• Control cache reuse was 98.957%. Admission generated 50,322 retried 429s, exactly 47,999 running_limit plus 2,323 engine_waiting decisions.
• The patched owner-pressure arm was not measured. Its gateway startup exposed a separate worker lifecycle race: both 38-slot workers were healthy and ready, but aggregate capacity remained 38 instead of 76. Strict pre-traffic gates aborted the arm.

The capacity race is tracked in #56 and fixed with deterministic regressions in draft PR #57. This PR should remain draft until a combined canary image includes #57 and the same 76-way A/B completes. The balanced-engine estimate is promising, but there is no treatment throughput claim yet.

[DavidBellamy]

Canary correction (2026-08-07): the first isolated M1 A/B did not execute this change. The control and nominal treatment images contained the same compiled smg_rs.abi3.so (sha256:0f93632c7a0f905c4b52edc83383c4949fd69ba063ac6309cb7d936de6455f23). A shared CARGO_TARGET_DIR reused the control Rust artifact while wheel metadata changed, so distinct wheel/image hashes were misleading.

The later legacy /get_load fallback hypothesis was also disproved: the canary logs show successful rich /v1/loads polling, and the pinned SGLang build populates max_running_requests. I reverted that unrelated fallback in 11c6a2f6; the eight focused scheduler-pressure tests pass.

I am rebuilding the original treatment with a source-specific Cargo target and will require the embedded .so digest to differ from control both before installation and after sqsh export. Until that passes and the A/B is rerun, the cancelled job is packaging-negative evidence only, not a behavioral validation of this PR.

[DavidBellamy]

Corrected isolated M1 canary result (job 1191549)

The earlier A/B artifact was invalid because its nominal treatment and control images loaded the same compiled smg_rs binary. I rebuilt the treatment with a unique Cargo target and required wheel, installed-container, exported-image, and fresh-container hashes to agree before launch.

Runtime provenance in this run:

• treatment: this PR plus fix(worker): emit activation status events #57, loaded smg_rs SHA-256 8b4ae47918b8ec304673bcd2aadff55fe9ec52ca0ae205f8a07b2fc72ef6fa02
• control: fix(worker): emit activation status events #57 only, loaded smg_rs SHA-256 0f93632c7a0f905c4b52edc83383c4949fd69ba063ac6309cb7d936de6455f23
• both runtime-loaded hashes matched their embedded provenance and differed from each other

Workload and isolation:

• two independent TP16/EP16 Kimi K3 engines, two H200 nodes per engine, plus one isolated CPU/router node
• 38 running-request slots per engine, 76 trajectories offered concurrently
• 76 growing-prefix trajectories x 3 turns x 1,000 output tokens
• 16,078,104 submitted prompt tokens / 228,000 output tokens, ratio 70.518:1
• both arms were forced to the same pre-release state: cached owner 35 running, peer 0 running, both 0 waiting, then the remaining 41 trajectories were released
• adaptive admission remained shadow-only, so this isolates cache-aware routing pressure

Matched result:

metric	#57 control	#55 + #57 treatment	change
successful output tok/s, fleet	438.090	730.382	+66.72%
successful output tok/s per 4 serving nodes	109.523	182.595	1.667x
successful output tok/s per all 5 allocated nodes	87.618	146.076	1.667x
whole-arm wall time	520.441 s	312.166 s	-40.02%
completed trajectories/hour	525.708	876.458	+66.72%
latency p50 / p95	162.462 / 170.627 s	85.184 / 115.135 s	lower
fleet running max	38	73	+35
generation share	0% / 100%	51.32% / 48.68%	balanced
cached prompt fraction	98.29%	98.58%	+0.29 pp

Both arms completed 228/228 turns with zero terminal failures, 429 retries, retractions, or CUDA OOMs. The treatment made 10 bounded owner_pressure_spill decisions, 207 cached-owner holds, and 11 conservative no_safe_spill_hold decisions. Control made 228/228 cached-owner holds, reached 38 running plus up to 40 queued on one engine, and left the peer completely idle.

The fail-closed analyzer intentionally exited nonzero on three caveats, so I am keeping this PR in draft and am not recommending production activation yet:

1. Exact output-token hashes matched for 156/228 turns (52/76 trajectories) and differed for 72/228 turns under temperature 0. Prompt-token counts and the common prefix were identical. The deployed SGLang profile does not enable batch-invariant deterministic inference, so changed batch shapes can change greedy output, but this still needs a fixed-trace or quality-parity canary before rollout.
2. The analyzer classified the 11 no_safe_spill_hold decisions as an unsafe-branch failure. The code path is conservative: when no unpressured replica exists, it preserves affinity on the least-pressured owner rather than creating another owner. It produced no retractions here, but admission enforcement should be tested with it.
3. Each engine reached near-maximum concurrency (38 and 35), fleet max was 73, and simultaneous high concurrency persisted for 17 aligned samples. One engine's nonzero-running p50 was 24 rather than the analyzer's 30 threshold; the other was 31.

The complete analyzer is at /lustrefs/users/david.bellamy/k3-owner-pressure-routing/runs/1191549/analysis.json, SHA-256 8aab1f496308fcb0b0929ddba1564f54951a4ff46233933a141f2cce306e2cf0. All five nodes returned idle and production Comet was not changed.

Next gates before merge/activation: code-owner review, fixed-trace or task-quality parity, then a multi-engine canary with admission enforcement rather than shadow mode.

[DavidBellamy]

Enforced-admission follow-up (M1 job 1191552)

The 182.595 successful output tok/s/node shadow result above is real, but this PR is not sufficient for the current production admission path by itself.

Source audit shows that HTTP adaptive admission runs before route_typed_request_once and worker selection. In enforce mode, tracker.should_reject() returns 429 immediately, so cache-aware selection and owner_pressure_spill never run for that attempt.

The five-node M1 follow-up reproduced the exact live state in two consecutive one-second gateway samples using the provenance-verified treatment binary:

• cached owner: 38 running, 36 waiting, hard cap 38
• peer: 0 running, 0 waiting, hard cap 38
• load coverage: 1.0
• max-running coverage: 1.0
• aggregate hard cap: 76
• aggregate mean/max token usage: 0.06 / 0.12
• waiting allowance: 0

Snapshot artifacts:

• /lustrefs/users/david.bellamy/k3-enforced-admission-repro/runs/1191552/owner-pressure-treatment-hot-owner-confirmation-1.prom, SHA-256 0f803b4ac7c49af3a90383fa92a5b70e573baa68a488827aa491ae43521107df
• /lustrefs/users/david.bellamy/k3-enforced-admission-repro/runs/1191552/owner-pressure-treatment-hot-owner-confirmation-2.prom, SHA-256 eeecda300e4f1437b7e48e0ee6ef158499ff386234db911aa8cbb3a0dea1bbba

At that state the current EngineFeedback decision is deterministically engine_waiting: token pressure is below threshold, waiting is 36 > 0, and that branch precedes running_limit. Enforced HTTP handling then returns 429 before smg_worker_selection_total or any cache-policy decision can increment. This explains the production loop in which an idle peer cannot acquire its first prefix owner.

I am deliberately classifying this as source-plus-live-state evidence, not an endpoint 429 measurement. The one-shot probe wrapper rejected a zero-retry CLI argument and the fail-closed cleanup stopped the gateway before a replacement raw probe could be sent. The harness is corrected for a future rerun; no production process was touched, and all five nodes returned idle.

I also prototyped a distribution-headroom admission rule in an unpublished worktree. It bypasses both the waiter gate and a skewed learned knee only with complete per-replica load/cap telemetry, a distinct clean peer, and aggregate hard-cap headroom. Formatting/diff checks and 24 targeted Rust tests pass, including hot-owner/idle-peer, learned-knee, full-saturation, and incomplete-telemetry cases. It is not merge-ready: the admission controller cannot yet prove owner-pressure routing is enabled, and a one-request relaxation can close again as soon as the waiter clears. The safe design needs an explicit routing-policy opt-in plus a bounded one-in-flight seeding lease/cooldown.

This PR should remain draft. Required next gate: couple distribution-aware admission to owner-pressure routing, then rerun the corrected enforced canary before code-owner review and activation.