Skip to content

feat(vllm): native structured-decisions support for the decision adapter - #162

Draft
antonpibm wants to merge 1 commit into
mainfrom
feature/native-structured-decisions
Draft

antonpibm wants to merge 1 commit into
mainfrom
feature/native-structured-decisions

Conversation

@antonpibm

Copy link
Copy Markdown
Collaborator

vLLM 0.32 ships a first-class structured-decisions API (/v1/systemone) that reads a labelled question as the reply's first token. Two gaps stop it serving the Granite Switch decision adapter out of the box: GraniteSwitchForCausalLM is not in its READ_STRATEGIES allowlist, and its native read never passes adapter_name to apply_chat_template, so it would read the base model's guess rather than activating the trained adapter via the <|decision|> control token.

Add GraniteSwitchDecisionStrategy (a NextTokenStrategy subclass) that overrides only the render-time behaviour: it forces adapter_name into chat_template_kwargs, builds the two-turn "### DECISION" marker layout the adapter was trained on (with U+2060 escaping of all caller text), and uses the adapter's letter-alias label pool instead of the native digit/yes-no defaults. A startup activation probe fails loudly if the marker does not emit exactly one control token inside the question turn. register_structured_decisions is wired as a vllm.general_plugins entry point and is a no-op on vLLM lines without the API (< 0.32). Render helpers are copied verbatim from the systemone endpoint plugin's renderer to keep one source of truth for the trained layout.

Bump the vllm extra to <0.33 and add the vllm32 / dev-vllm32 groups (dev-latest now points at 0.32). Unit tests cover the render layout, label pool, escaping, activation probe, and registration; they run on CPU without vLLM installed.

vLLM 0.32 ships a first-class structured-decisions API (/v1/systemone) that reads a
labelled question as the reply's first token. Two gaps stop it serving the Granite
Switch decision adapter out of the box: GraniteSwitchForCausalLM is not in its
READ_STRATEGIES allowlist, and its native read never passes adapter_name to
apply_chat_template, so it would read the base model's guess rather than activating the
trained adapter via the <|decision|> control token.

Add GraniteSwitchDecisionStrategy (a NextTokenStrategy subclass) that overrides only the
render-time behaviour: it forces adapter_name into chat_template_kwargs, builds the
two-turn "### DECISION" marker layout the adapter was trained on (with U+2060 escaping of
all caller text), and uses the adapter's letter-alias label pool instead of the native
digit/yes-no defaults. A startup activation probe fails loudly if the marker does not
emit exactly one control token inside the question turn. register_structured_decisions
is wired as a vllm.general_plugins entry point and is a no-op on vLLM lines without the
API (< 0.32). Render helpers are copied verbatim from the systemone endpoint plugin's
renderer to keep one source of truth for the trained layout.

Bump the vllm extra to <0.33 and add the vllm32 / dev-vllm32 groups (dev-latest now
points at 0.32). Unit tests cover the render layout, label pool, escaping, activation
probe, and registration; they run on CPU without vLLM installed.
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!


slots, engine_inputs, params = [], [], []
for q in questions:
descriptions = [o.description for o in q.options]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It passes only o.description to render, dropping o.name. When a description is missing, render falls back to showing the label letter, so a noul question with no criteria reaches the model as A: A / B: B and it's never told which letter means yes. Description-less choice options hit the same path, and score is worse: its level text lives in name with description always null, so every score level renders as A: A, B: B, … and the scale never reaches the prompt at all.

vLLM's own reply_line renders name and appends the description when present, which is the shape that survives both cases. Worth passing o.name or o.description here (or falling back to the name inside render) before the prompt is built.

except ValueError as e:
# build_label_pool raises with "could only build N single-token labels"
built = int(str(e).split("could only build ", 1)[1].split(" ", 1)[0])
return DecisionLimits(max_questions=64, max_options=built)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

limits() advertises the two-letter alias pool as max_options, but build_question caps at min(max_options, len(qtype.label_set)) and ChoiceQuestion.label_set is 26 letters, so anything past 26 options is still rejected. From 0.33, DecisionLimits has a labeled flag that switches the cap to max_options alone, so labeled=False looks like the intended fix.

no structured-decisions API (< 0.32), so the plugin is safe to load on older lines.
"""
try:
from vllm.entrypoints.generate.structured_decisions.strategies import (

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In 0.33 the strategies module moves to vllm.entrypoints.openai.decisions, so this import will need the new path. As written, the except ImportError quietly skips registration rather than failing.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants