feat: add engine-feedback admission and tenant fair sharing - #40
Merged
Conversation
Signed-off-by: David <12414531+DavidBellamy@users.noreply.github.com>
Signed-off-by: David <12414531+DavidBellamy@users.noreply.github.com>
Signed-off-by: David <12414531+DavidBellamy@users.noreply.github.com>
Signed-off-by: David <12414531+DavidBellamy@users.noreply.github.com>
Signed-off-by: David <12414531+DavidBellamy@users.noreply.github.com>
Signed-off-by: David <12414531+DavidBellamy@users.noreply.github.com>
Isolate scheduling debt by canonical model while retaining process-global output accounting. Use hierarchical named and other buckets without idling non-fungible pools. Signed-off-by: David <12414531+DavidBellamy@users.noreply.github.com>
Signed-off-by: David <12414531+DavidBellamy@users.noreply.github.com>
DavidBellamy
marked this pull request as ready for review
August 5, 2026 08:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Problem
Static per-partition concurrency ceilings do not reflect heterogeneous worker limits or the live throughput knee, while priority classes alone cannot allocate actual output-token service fairly among users. The earlier prediction-based draft in #31 also couples admission to request-length estimation before engine behavior is sufficiently calibrated.
Solution
Add engine-feedback admission that learns useful running capacity from fresh worker telemetry, probes above the observed knee, backs off on fleet waiting and token or KV pressure, and fails open when coverage is insufficient. Worker-reported ceilings scale partition capacity without changing the global work-conserving clamp.
Add optional weighted fair sharing based on settled output tokens. Flat configuration remains process-global across partitions; optional model profiles use hierarchical named-user and aggregate-other buckets while retaining each real tenant identity. Selection remains work-conserving within each eligible partition, batch and interactive traffic share accounting, idle users do not bank unlimited credit, and missing terminal usage retains the provisional charge.
Trusted tenant, model, and output-estimate headers remain independently disabled by default.
Refs #30. Supersedes #31.
Changes
Test Plan
cargo test --offline -p smg --lib --quiet -- --test-threads=1: 1,458 passed, 5 ignoredcargo +nightly fmt --all -- --checkpassedcargo check --offline --manifest-path bindings/python/Cargo.tomlpassed