Repository navigation
Conversation
vLLM 0.32 ships a first-class structured-decisions API (/v1/systemone) that reads a labelled question as the reply's first token. Two gaps stop it serving the Granite Switch decision adapter out of the box: GraniteSwitchForCausalLM is not in its READ_STRATEGIES allowlist, and its native read never passes adapter_name to apply_chat_template, so it would read the base model's guess rather than activating the trained adapter via the <|decision|> control token. Add GraniteSwitchDecisionStrategy (a NextTokenStrategy subclass) that overrides only the render-time behaviour: it forces adapter_name into chat_template_kwargs, builds the two-turn "### DECISION" marker layout the adapter was trained on (with U+2060 escaping of all caller text), and uses the adapter's letter-alias label pool instead of the native digit/yes-no defaults. A startup activation probe fails loudly if the marker does not emit exactly one control token inside the question turn. register_structured_decisions is wired as a vllm.general_plugins entry point and is a no-op on vLLM lines without the API (< 0.32). Render helpers are copied verbatim from the systemone endpoint plugin's renderer to keep one source of truth for the trained layout. Bump the vllm extra to <0.33 and add the vllm32 / dev-vllm32 groups (dev-latest now points at 0.32). Unit tests cover the render layout, label pool, escaping, activation probe, and registration; they run on CPU without vLLM installed.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
|
||
| slots, engine_inputs, params = [], [], [] | ||
| for q in questions: | ||
| descriptions = [o.description for o in q.options] |
There was a problem hiding this comment.
It passes only o.description to render, dropping o.name. When a description is missing, render falls back to showing the label letter, so a noul question with no criteria reaches the model as A: A / B: B and it's never told which letter means yes. Description-less choice options hit the same path, and score is worse: its level text lives in name with description always null, so every score level renders as A: A, B: B, … and the scale never reaches the prompt at all.
vLLM's own reply_line renders name and appends the description when present, which is the shape that survives both cases. Worth passing o.name or o.description here (or falling back to the name inside render) before the prompt is built.
| except ValueError as e: | ||
| # build_label_pool raises with "could only build N single-token labels" | ||
| built = int(str(e).split("could only build ", 1)[1].split(" ", 1)[0]) | ||
| return DecisionLimits(max_questions=64, max_options=built) |
There was a problem hiding this comment.
limits() advertises the two-letter alias pool as max_options, but build_question caps at min(max_options, len(qtype.label_set)) and ChoiceQuestion.label_set is 26 letters, so anything past 26 options is still rejected. From 0.33, DecisionLimits has a labeled flag that switches the cap to max_options alone, so labeled=False looks like the intended fix.
| no structured-decisions API (< 0.32), so the plugin is safe to load on older lines. | ||
| """ | ||
| try: | ||
| from vllm.entrypoints.generate.structured_decisions.strategies import ( |
There was a problem hiding this comment.
In 0.33 the strategies module moves to vllm.entrypoints.openai.decisions, so this import will need the new path. As written, the except ImportError quietly skips registration rather than failing.
vLLM 0.32 ships a first-class structured-decisions API (/v1/systemone) that reads a labelled question as the reply's first token. Two gaps stop it serving the Granite Switch decision adapter out of the box: GraniteSwitchForCausalLM is not in its READ_STRATEGIES allowlist, and its native read never passes adapter_name to apply_chat_template, so it would read the base model's guess rather than activating the trained adapter via the <|decision|> control token.
Add GraniteSwitchDecisionStrategy (a NextTokenStrategy subclass) that overrides only the render-time behaviour: it forces adapter_name into chat_template_kwargs, builds the two-turn "### DECISION" marker layout the adapter was trained on (with U+2060 escaping of all caller text), and uses the adapter's letter-alias label pool instead of the native digit/yes-no defaults. A startup activation probe fails loudly if the marker does not emit exactly one control token inside the question turn. register_structured_decisions is wired as a vllm.general_plugins entry point and is a no-op on vLLM lines without the API (< 0.32). Render helpers are copied verbatim from the systemone endpoint plugin's renderer to keep one source of truth for the trained layout.
Bump the vllm extra to <0.33 and add the vllm32 / dev-vllm32 groups (dev-latest now points at 0.32). Unit tests cover the render layout, label pool, escaping, activation probe, and registration; they run on CPU without vLLM installed.