Skip to content

feat: add engine-feedback admission and tenant fair sharing - #40

Merged
DavidBellamy merged 8 commits into
prodfrom
feat/engine-feedback-admission
Aug 5, 2026
Merged

feat: add engine-feedback admission and tenant fair sharing#40
DavidBellamy merged 8 commits into
prodfrom
feat/engine-feedback-admission

Conversation

@DavidBellamy

Copy link
Copy Markdown

Description

Problem

Static per-partition concurrency ceilings do not reflect heterogeneous worker limits or the live throughput knee, while priority classes alone cannot allocate actual output-token service fairly among users. The earlier prediction-based draft in #31 also couples admission to request-length estimation before engine behavior is sufficiently calibrated.

Solution

Add engine-feedback admission that learns useful running capacity from fresh worker telemetry, probes above the observed knee, backs off on fleet waiting and token or KV pressure, and fails open when coverage is insufficient. Worker-reported ceilings scale partition capacity without changing the global work-conserving clamp.

Add optional weighted fair sharing based on settled output tokens. Flat configuration remains process-global across partitions; optional model profiles use hierarchical named-user and aggregate-other buckets while retaining each real tenant identity. Selection remains work-conserving within each eligible partition, batch and interactive traffic share accounting, idle users do not bank unlimited credit, and missing terminal usage retains the provisional charge.

Trusted tenant, model, and output-estimate headers remain independently disabled by default.

Refs #30. Supersedes #31.

Changes

  • learn per-partition admission capacity from engine feedback and representative worker ceilings
  • expose the strategy through Rust and Python configuration
  • add process-global actual-output accounting and model-scoped hierarchical WFQ
  • settle completed JSON and SSE usage with bounded fallback handling
  • add trusted tenant/model identity resolution and spoof-resistant configuration seams
  • export admission, capacity, tenant service, queue wait, fallback, and coverage metrics
  • document work-conserving and process-local limitations

Test Plan

  • cargo test --offline -p smg --lib --quiet -- --test-threads=1: 1,458 passed, 5 ignored
  • focused scheduler suite: 169 passed
  • strict all-target clippy passed with the documented pre-existing baseline allowances
  • cargo +nightly fmt --all -- --check passed
  • cargo check --offline --manifest-path bindings/python/Cargo.toml passed
  • Python parser tests, DCO, and diff checks passed

Signed-off-by: David <12414531+DavidBellamy@users.noreply.github.com>
Signed-off-by: David <12414531+DavidBellamy@users.noreply.github.com>
Signed-off-by: David <12414531+DavidBellamy@users.noreply.github.com>
Signed-off-by: David <12414531+DavidBellamy@users.noreply.github.com>
Signed-off-by: David <12414531+DavidBellamy@users.noreply.github.com>
Signed-off-by: David <12414531+DavidBellamy@users.noreply.github.com>
Isolate scheduling debt by canonical model while retaining process-global output accounting. Use hierarchical named and other buckets without idling non-fungible pools.

Signed-off-by: David <12414531+DavidBellamy@users.noreply.github.com>
Signed-off-by: David <12414531+DavidBellamy@users.noreply.github.com>
@DavidBellamy DavidBellamy changed the title Add engine-feedback admission and tenant fair sharing feat: add engine-feedback admission and tenant fair sharing Aug 5, 2026
@DavidBellamy
DavidBellamy marked this pull request as ready for review August 5, 2026 08:06
@DavidBellamy
DavidBellamy requested a review from a team August 5, 2026 08:06
@DavidBellamy
DavidBellamy merged commit 7650624 into prod Aug 5, 2026
10 of 12 checks passed
@DavidBellamy
DavidBellamy deleted the feat/engine-feedback-admission branch August 5, 2026 08:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant