This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
granite-switch implements Granite Switch, a system for building and deploying Granite models with embedded LoRA adapters. The system is a single unified Python package (granite_switch) with optional extras for different backends.
- Building models with embedded adapters - Combine a base Granite model with multiple LoRA adapters into a single checkpoint
- Automatic adapter control - Activate adapters via special control tokens or chat templates
- Fast inference - Deploy with vLLM for speedup over standard HuggingFace inference
- Multi-transition switching - MultiSwitch picks the adapter per token from the control tokens seen so far (latest wins), so one request can switch adapters any number of times
granite-switch/
├── pyproject.toml # Single package definition with optional extras
├── src/
│ └── granite_switch/ # Unified package
│ ├── __init__.py # Core exports (GraniteSwitchConfig, __version__)
│ ├── config.py # Unified GraniteSwitchConfig
│ ├── conversation.py # Conversation: multi-turn, per-turn adapter, KV-history policy
│ ├── token_exchange.py # Control-token -> substitute-id LUT (shared by both backends)
│ │
│ ├── composer/ # Compose system (requires [compose] extra)
│ │ ├── __init__.py
│ │ ├── adapter_discovery.py # Adapter discovery and resolution
│ │ ├── adapter_loader.py # Adapter weight loading
│ │ ├── arch.py # Architecture definitions
│ │ ├── compose_granite_switch.py # Main compose script (CLI entry point)
│ │ ├── compose_utils.py # GraniteSwitchComposer class
│ │ ├── tokenizer_setup.py # Tokenizer configuration for control tokens
│ │ ├── validator.py # Compose validation checks
│ │ ├── weight_remapper.py # Adapter name remapping (AdapterRemapper)
│ │ ├── weight_transfer.py # Base model weight transfer
│ │ └── reporting/ # Compose reporting utilities
│ │ ├── __init__.py
│ │ ├── adapter_analysis.py
│ │ ├── compose_report.py
│ │ ├── model_card.py
│ │ └── population_table.py
│ │
│ ├── hf/ # HuggingFace backend (requires [hf] extra)
│ │ ├── __init__.py # Registers with transformers AutoConfig/AutoModel
│ │ ├── modeling_granite_switch.py
│ │ ├── core/
│ │ │ ├── __init__.py
│ │ │ └── lora.py # SwitchedLoRALinear, MergedSwitchedLoRALinear
│ │ └── switch/
│ │ ├── __init__.py # create_switch() -> MultiSwitch
│ │ ├── multi.py # MultiSwitch (HF attention backends)
│ │ └── codes/ # Kerdock/DG codebook generation for the memory head
│ │
│ ├── kernels/ # Backend-agnostic Triton kernels
│ │ ├── __init__.py
│ │ └── switch_lora_kernel.py # SWITCH fused-LoRA kernel: expand + swiglu + W_cross shrink
│ │
│ └── vllm/ # vLLM backend (requires [vllm] extra)
│ ├── __init__.py # register() for vLLM plugin system
│ ├── granite_switch_model.py # GraniteSwitch{Model,ForCausalLM}: LoRA + SR, TP/PP
│ ├── core/
│ │ ├── __init__.py
│ │ ├── lora.py # SwitchedLoRALinear (SWITCH kernel backend)
│ │ ├── lora_kernel_meta.py # FusedLoRAKernelMeta: per-module remap + tile bitmasks
│ │ └── lora_ops.py # torch custom-op wrappers for the SWITCH launchers
│ ├── decoder/ # per-adaptation decoder tier (DecoderInterface)
│ │ ├── __init__.py
│ │ ├── interface.py
│ │ ├── lora/decoder.py # GraniteLoRAEmbeddedAttention (LoRA / aLoRA)
│ │ └── shadow_residual/ # SR dual-stream decoder
│ │ ├── decoder.py # ShadowResidualAttention (doubled-Q, TP-aware)
│ │ ├── wcross_shunt.py # WCrossShunt (base->adapter cross-stream)
│ │ ├── kernel_meta.py # SRFusedLoRAKernelMeta
│ │ └── _sr_ops.py # Q-head interleave / deinterleave
│ ├── audio/ # Audio input (alpha): ASR cascade, chunking, processor
│ └── switch/
│ ├── __init__.py # create_switch() -> MultiSwitch
│ └── multi.py # MultiSwitch (vLLM Attention)
│
├── tests/ # All tests
│ ├── unit/ # Unit tests (fastest, CPU)
│ ├── hf/ # HuggingFace-specific tests
│ ├── vllm/ # vLLM-specific tests
│ ├── composer/ # Compose system tests
│ ├── integration/ # Cross-backend integration tests
│ ├── audio/ # Audio test fixtures (sample .wav)
│ └── shared/ # Shared test utilities and parametrized cases
│
├── .pre-commit/ # Pre-commit hook scripts (validate_links.py)
├── .pre-commit-config.yaml # Pre-commit hook configuration
├── scratch/ # Throwaway debug/diagnostic scripts (gitignored)
├── docs/ # Documentation
├── tutorials/ # Tutorials and how-to guides
├── CLAUDE.md # This file
└── README.md
This project uses uv.
# Core package only (config)
uv sync
# With HuggingFace backend
uv sync --extra hf
# With vLLM backend
uv sync --extra vllm
# With compose tools
uv sync --extra compose
# Development (hf + compose + audio extras, pytest, vLLM 0.26) -- `dev` is a
# dependency group, not an extra. `--group dev-latest` swaps in the newest vLLM line;
# add `--extra tutorials` to run the tutorials.
uv sync --group dev# Config (shared by all backends)
from granite_switch import GraniteSwitchConfig
from granite_switch.config import GraniteSwitchConfig # equivalent
# HuggingFace backend
from granite_switch.hf import GraniteSwitchForCausalLM
from granite_switch.hf.core.lora import SwitchedLoRALinear
from granite_switch.hf.switch import MultiSwitch, create_switch
# vLLM backend (auto-registered via plugin entry point)
from granite_switch.vllm import register
# Compose system
from granite_switch.composer import GraniteSwitchComposerIMPORTANT: Keep the repository organized by placing files in their designated directories.
All .md documentation files MUST go in a docs/ directory:
- Root-level docs (
docs/): Cross-implementation documentation, guides, and architecture docs - Exceptions: Only
CLAUDE.mdandREADME.mdmay be at the repository root
All test_*.py test files MUST go in a tests/ directory:
tests/unit/: Unit tests (fastest, CPU-only)tests/hf/: HuggingFace implementation teststests/vllm/: vLLM implementation teststests/composer/: Compose system teststests/integration/: Cross-implementation and end-to-end integration teststests/shared/: Shared test utilities and parametrized cases
IMPORTANT: tests/ is for official regression tests ONLY. Do NOT place throwaway diagnostic,
debugging, or exploratory scripts in tests/. Use scratch/ instead (it is gitignored). Running
pytest tests/ should only execute curated, maintained tests — never one-off investigations.
- Test files:
test_*.py - Documentation:
UPPER_CASE.md - Scripts:
snake_case.py
# Compose with HuggingFace adapters
python -m granite_switch.composer.compose_granite_switch \
--adapters ibm-granite/granitelib-rag-r1.0
# Multiple adapters
python -m granite_switch.composer.compose_granite_switch \
--adapters ibm-granite/granitelib-rag-r1.0 your-org/extra-adapter
# Custom output directory
python -m granite_switch.composer.compose_granite_switch \
--adapters ibm-granite/granitelib-rag-r1.0 --output ./my-custom-modelAlways use -v -s --tb=short when running tests. -v (verbose) prints each test name as
it starts, giving real-time progress visibility. -s disables output capture so print()
statements inside tests appear immediately instead of being swallowed. Without these, long-running
test files produce no output until they finish. -x (fail fast) stops on the first failure —
no point running 200 more tests after something breaks.
Check GPU availability first — the underlying hardware can change between sessions:
python -c "import torch; print('GPU' if torch.cuda.is_available() else 'CPU only')"This determines which tests can run. vLLM and integration tests require a GPU; unit and HF tests run on CPU.
Run tests incrementally by directory, in order of speed — don't run the full suite as a single command:
# 1. Unit tests first (fastest, CPU)
pytest tests/unit/ -v -s --tb=short -x
# 2. HF tests by file (CPU)
pytest tests/hf/test_multi_switch.py -v -s --tb=short -x
pytest tests/hf/test_model_forward.py -v -s --tb=short -x
# 3. vLLM tests by file (GPU required)
pytest tests/vllm/test_multi_switch.py -v -s --tb=short -x
pytest tests/vllm/test_model_forward.py -v -s --tb=short -x
# 4. Integration tests last (slowest, GPU required)
pytest tests/integration/ -v -s --tb=short -x
# Run a specific test pattern when debugging
pytest tests/ -k "pattern" -v -s --tb=short -x# Verify plugin registration
python -c "from vllm.plugins import load_general_plugins; \
from vllm import ModelRegistry; \
load_general_plugins(); \
print('OK' if 'GraniteSwitchForCausalLM' in ModelRegistry.get_supported_archs() else 'FAIL')"
# Start API server
python -m vllm.entrypoints.openai.api_server \
--model ./granite-with-all-aloras \
--port 8000The Granite Switch extends the base Granite model with:
-
Embedded LoRA Adapters (frozen during inference)
- Multiple task/domain-specific adapters embedded in the same checkpoint
- Each adapter has LoRA weights (lora_A, lora_B) stacked in tensors
- Controlled via control tokens; MultiSwitch turns them into a per-token adapter index
-
Control Tokens
- Each adapter has a control token
<|adapter|>that fires the switch - The switch reads the original ids, then rewrites each control token to its substitute id
(
adapter_substitute_token_ids,token_exchange.py) so the decoder never embeds it
- Each adapter has a control token
-
Chat Template Integration
- Maps adapter names to control tokens
- Automatic token placement based on adapter type (ALORA vs LORA)
-
MultiSwitch (the only switch engine; SingleSwitch was removed)
- Two small attention heads compute the adapter index per token: a counting head turns "how many control tokens so far" into an address, and a Kerdock/DG coded-memory head reads back the adapter written there
- Routing is latest-wins and supports any number of transitions per request, including a
return to base via
<|base_reset|>(only when composed with--base-reset-token) - The codebook is constructed, not trained; config knobs are the
ms_*fields - See docs/MULTISWITCH_EXPLAINED.md
Purpose: Model building, prototyping, and reference inference
- Full
transformersintegration (PreTrainedModel,GenerationMixin) - Standard PyTorch operations
Purpose: Fast production inference (10-20x speedup)
- SWITCH fused-LoRA Triton kernel for optimized adapter computation (replaces Punica)
- PagedAttention for efficient KV cache
- Continuous batching, tensor/pipeline parallelism
- OpenAI-compatible API server
SR runs a frozen base stream and a LoRA adapter stream through the same layer, connected by a
cross_stream low-rank injection. The two streams share every parameter (the base stream is
the same modules called with adapter_indices=None), and K/V always come from the base
stream. SR uses the same fused projections as the rest of the HF backend.
One GraniteSwitchModel / GraniteSwitchForCausalLM pair serves both modes and picks its
decoder layer class in __init__ from config.dual_stream: SRSwitchDecoderLayer (a
subclass) when True, GraniteSwitchAttentionDecoderLayer otherwise. A checkpoint holds
either SR adapters or standard LoRA/aLoRA adapters — the composer rejects a mixed
set. See docs/SR_ARCHITECTURE.md for full details.
Both backends share the same weight format:
# Built with HuggingFace
model_hf.save_pretrained("./checkpoint")
# Loaded directly with vLLM
llm = LLM(model="./checkpoint")attention_multiplier: Attention score scaling (instead of1/sqrt(head_dim))logits_scaling: Applied to final logits (main architectural difference with Llama)residual_multiplier: Applied to residual connectionsembedding_multiplier: Applied to input embeddings
Always use config values - never hardcode these parameters.
{
"model_type": "granite_switch",
"architectures": ["GraniteSwitchForCausalLM"],
"num_adapters": 4,
"adapter_token_ids": [100, 101, 102, 103],
"adapter_names": ["adapter_0", "adapter_1", "adapter_2", "adapter_3"],
"adapter_substitute_token_ids": [10, 11, 12, 13],
"max_lora_rank": 8,
"adapter_ranks": [8, 8, 8, 8],
"switch_head_dim": 32,
"ms_code_type": "kerdock",
"ms_code_m": 6
}Control tokens: 0 = no adapter, 1+ = adapter indices
SWITCH kernel (vLLM): per-module kernel-local indices — 0 = base (no adapter, or not applicable to this module), 1.. = that module's applicable adapters in ascending rank order. Global adapter ids are remapped per module centrally (see vllm/core/lora_kernel_meta.py).
All control tokens are freely generatable — there is no runtime suppression. The model can produce any control token during generation.
- ALORA adapters: Token placed either in user message by matching invocation sequence or right before generation prompt
- LORA adapters: Token placed at sequence beginning
Two Granite chat-template families are supported, auto-detected by
detect_template_format from the base tokenizer's template: the
<|start_of_role|>…<|end_of_role|> role-marker format (4.0/4.1) and the ChatML
<|im_start|>…<|im_end|> format (4.2). All adapter control-token injection is
driven by the detected TemplateFormat rather than hardcoded per format.
- Granite uses
logits_scaling(typically 8.0) - Custom attention scaling via
attention_multiplier - Different residual and embedding multipliers
Always load from config, never hardcode.
No test should manually assemble GraniteSwitchConfig or call transfer_base_weights
directly. All model construction must go through GraniteSwitchComposer so that the
compose pipeline itself is what's being tested. If the composer can't handle a use case
(e.g., zero-adapter skinning), extend the composer — don't work around it in tests.
The eager backend does NOT handle attention_mask=None as causal — it treats None as no mask
(full attention). SDPA and FlashAttention handle attention_mask=None correctly via is_causal
attribute on the module.
The HF MultiSwitch tests (tests/hf/test_multi_switch.py) probe each non-eager attention backend once
and parametrize over the ones that work on the current platform. Unavailable backends are skipped.
SwitchedLoRALinear's row-parallel bypass path passes bias to all TP ranks instead of
suppressing it for rank > 0. After all-reduce this doubles the bias. Not affected: all Granite
architectures (4.0, 4.1) use attention_bias=False and mlp_bias=False.
The GraniteSwitch HF backend uses fused QKV and gate-up projections, symmetric with the vLLM
backend architecture. Upstream HuggingFace GraniteMoeHybridForCausalLM uses separate projections.
Fused projections change the floating-point reduction order, so bit-exact skinning equivalence
with the upstream HF model is not achievable. The HF skinning tests in
tests/composer/test_skinning_equivalence.py are skipped for this reason.
The vLLM check (tests/vllm/test_generation_equivalence.py) is the authoritative equivalence
test, but it is not token-exact either. The skinned model runs its projections through the
fused SWITCH Triton kernel (kernels/switch_lora_kernel.py), whose float-reduction order differs
from vLLM's native linear even with a zero adapter, so the two paths are numerically close but not
bit-identical. That sub-ULP difference can flip a near-tie greedy argmax — observed under vLLM 0.20
but not 0.19, because each version's numerics land the tie on a different side. The test therefore
gates distribution equivalence (per-position top-k JSD + Jaccard), not token-for-token match:
robust to benign ties while still catching a real logit/weight regression. See
tests/vllm/_generation_equivalence_worker.py for the metric and thresholds.
The coded engine recovers a control token's write address from a 1/(1+n) attention signal.
The HF backend forces that signal to fp32 and is exact past 4095; the vLLM backend's heads are
paged-KV vllm.Attention modules, so the signal takes the KV-cache dtype and bf16 inverts
exactly only up to n = 188 (189 aliases). This is a different limit from the codebook's
capacity == 2048, which bounds the memory head, not the counting head. Handled only in
Conversation, and by fallback rather than refusal: above MAX_RETAINED_CONTROL_TOKENS = 188 it
re-prefills the turn, resetting the count to one and losing history's adapter attribution (logged;
counted in Conversation.reprefills). A raw /v1/completions request still bypasses this entirely.
--kv-cache-dtype fp8 is also unguarded and would saturate _NEG_INF and code(n) * 28.
See section 8 of docs/MULTISWITCH_EXPLAINED.md.
_debug_write_addresses and _debug_counting_signal are written behind
if not torch.compiler.is_compiling(): (src/granite_switch/vllm/switch/multi.py), and the switch
runs inside the @support_torch_compile region (src/granite_switch/vllm/granite_switch_model.py).
is_compiling() is True during tracing, so the body is absent from the compiled graph — a
default-configured server never sets them. Any test reading them must pass enforce_eager=True and
must assert they are present, not skip on None. The same reason is why a host-side validation check
cannot live in MultiSwitch.forward.
SR used to build unfused Q/K/V and gate/up/down projections (unfused_qkv=True); it now uses
the same fused projections as everything else. Since transformers silently keeps unknown
config keys, an old unfused checkpoint would otherwise load into a fused model and quietly
mismatch keys, so GraniteSwitchConfig rejects unfused_qkv=True with a re-compose error.
See docs/SR_ARCHITECTURE.md for details.
A pure sparse-MoE base (granitemoe) has no dense shared MLP; upstream encodes this as
shared_intermediate_size == 0. GraniteMoeSharedMLP.__init__ sets self.hidden_size = config.shared_intermediate_size, so constructing it at 0 would register [0, H]/[H, 0] weights
that no checkpoint ships and then add their output into the MoE result. All three decoders (hf,
vllm/decoder/lora, vllm/decoder/shadow_residual) therefore gate on shared_intermediate_size > 0, set shared_mlp = None otherwise, and config.py correspondingly drops the
shared_input_linear/shared_output_linear LoRA targets so no zero-width LoRA is built either.
A layer with neither experts nor a shared MLP is invalid: the three decoders raise ValueError,
but the config itself does not — the guard is decoder-level. This is why the vLLM-LoRA forward's
not has_experts → shared_mlp(x) branch is safe only because that ValueError upstream rules out
the no-MLP case; protect it if you refactor the gate. Config-level coverage:
tests/unit/test_config_edge_cases.py.
See docs/CICD.md for the full CI/CD setup — pre-commit hook list, setup steps, and what runs on every commit vs. in CI.
This repo uses pre-commit (ruff, hygiene hooks, SPDX-header and DCO-signoff checks, and a local link validator) to enforce quality before a commit lands. Hooks run automatically on git commit and most are auto-fixing, so re-stage and commit again if a hook modifies files. Do NOT use --no-verify to bypass — fix the underlying issue instead.
CONTRIBUTING.md- Contribution guidelines, uv/dev setup, and DCO sign-offdocs/GIT_WORKFLOW.md- Git branching strategy and commit guidelinesdocs/SUPPORTED_MODELS.md- Model compatibilitydocs/SR_ARCHITECTURE.md- Shadow Residual dual-stream architecturedocs/MULTISWITCH_EXPLAINED.md- MultiSwitch coded-memory routing engine and its limitsdocs/AUDIO.md- Audio input (ASR cascade)docs/CICD.md- Pre-commit hooks and CI setup
See docs/GIT_WORKFLOW.md for complete git workflow guidelines.
Quick reference:
- Branch naming:
feature/ticket-ID-descriptionorbugfix/ticket-ID-description - Workflow: Branch from
main→ develop → rebase → PR → merge → delete branch - Critical: Always verify comments match code before committing (see GIT_WORKFLOW.md)
- Commit format: Clear summary + explanation of WHAT changed and WHY
When committing, never sign as Claude (per project instructions)
Apache-2.0 (as indicated by SPDX headers in source files)