Skip to content

Pin and name the ASR in the config, add --bundle-asr, and add a release test tier - #163

Closed
aviv1ron1 wants to merge 3 commits into
mainfrom
feature/release-test-tier
Closed

aviv1ron1 wants to merge 3 commits into
mainfrom
feature/release-test-tier

Conversation

@aviv1ron1

Copy link
Copy Markdown
Collaborator

Fixes #155
Fixes #158

Summary

Three related changes:

  1. audio: unpinned default ASR model changed on the Hub, breaking test_asr_ctc_default_gpu.py #155: pin the default ASR and name it in the config. The default ASR loaded by name with no revision, so the Hub repo's 2026-10-21 multilingual release silently changed every transcript (now capitalized and punctuated) and broke test_asr_ctc_default_gpu.py. The default also existed only inside the library, so a checkpoint's config didn't show which ASR it ran.
  2. audio: include local weights for ASR #158: --bundle-asr for airgapped deployments. Builds the ASR weights into the checkpoint so it serves audio with no Hub access.
  3. A release test tier. Real-model checks and the non-representative cases of the expensive GPU matrices move out of PR runs and run only before releases. On the current CI legs this is expected to move about 1h30m out of every full PR run down from 4h roughly to 2.5h.

1. ASR pinned and named in the config (#155)

  • With --enable-audio, compose always writes asr_model_id and a new asr_model_revision. The default model ibm-granite/granite-speech-5.0-470m-turboctc is pinned to its English-only revision 947f59a.
  • New --asr-revision flag. A custom --asr-model without it composes with a warning and loads unpinned.
  • Breaking: serving no longer falls back to a built-in ASR. An audio checkpoint without asr_model_id fails at server startup, with the two config.json lines that fix it.
  • DEFAULT_ASR_MODEL_ID moved to granite_switch.config. ASRTranscriber and get_transcriber now require model_id and accept a revision.
  • BUILD.md now lists the ASR model, its revision and whether it is bundled.
  • Version bumped to 0.2.0. CHANGELOG.md has a Breaking section. docs/AUDIO.md covers the new fields, migrating older checkpoints, and the fact that an older library ignores the new fields: it loads the ASR unpinned and still goes online for a bundled checkpoint.

2. --bundle-asr (#158)

  • It's opt-in and turns on --enable-audio. It copies the ASR into <output>/asr/ (config, safetensors weights, tokenizer, feature extractor, README and license) and sets asr_bundled_path.
  • --asr-model can be a local directory, so the bundle can be built inside the airgapped network.
  • Compose fails unless the bundle's config, feature extractor and tokenizer load with local_files_only=True.
  • At serve time a bundled checkpoint loads its ASR from disk. If it's served by Hub repo id, only asr/ is fetched, which also works offline from the HF cache.
  • vLLM's loader only reads top-level weight files, so it doesn't load asr/*.safetensors as LLM weights.
  • docs/AUDIO.md has a new "Airgapped deployment" section.

3. Release test tier

The new release marker is excluded by default (addopts: not deep and not release) and runs with pytest -m "release and not deep", or /gpu-test-release once the runner change below lands.

The real ASR is downloaded only in the release tier. test_asr_ctc_default_gpu.py is release-only and no longer asserts transcript text, which is model quality. Every other audio test uses a 30 KB random CTC model built locally (tests/shared/tiny_asr.py).

Release only:

Test Why
tests/integration/test_asr_ctc_default_gpu.py the real default ASR at its pinned revision
tests/vllm/test_granite4_fullsize.py (GPU class) test_granite4_mini.py covers the same equivalence at small size on every PR
tests/composer/test_upstream_files.py checks the published base repo's file set
tests/composer/test_selective_download.py::TestRealHubMetadata checks the published adapter repos' layout
tests/hf/test_quantization.py checks the quantization libraries on a real checkpoint

Split: one case on PRs, the rest in the release tier (pytest.param(..., marks=pytest.mark.release)):

Test PR runs Release adds
tests/vllm/test_granite4_mini.py (3 vLLM classes) 4.0-micro 4.0-1b, 4.0-350m
tests/vllm/test_multi_adapter_equivalence.py granite-4.1-3b (lora + sr) granite-4.2-3b (lora + sr)
tests/integration/test_audio_serving_smoke.py granite-4.0-micro granite-4.1-3b

Other test changes:

  • test_quantization.py never ran in CI: all 16 tests were skipped on vllm31 and vllm32, because it's gated on GRANITE_SWITCH_E2E_MODELS=1, which no GPU pod sets. tests/conftest.py now sets the variable when -m selects release.
  • test_adapter_routing_audio_enabled.py stays on PRs but is now cheap. It composes 13 empty built-in slots on granite-4.0-micro instead of three real adapter libraries on granite-4.1-3b. Routing reads only the control-token table.
  • test_audio_serving_smoke.py and test_audio_uncached_processor.py compose with --bundle-asr and the tiny ASR, so they also cover serving a bundled ASR.
  • docs/CICD.md: a new "Release test tier" section, plus a release-checklist step to run it.

The default ASR model was loaded by name with no revision, so the Hub
repo's 2026-10-21 multilingual release silently changed every
transcript (now capitalized and punctuated) and broke the GPU tests
(#155). The default also lived only in the library, so a checkpoint's
config did not show which ASR it ran.

Compose with --enable-audio now always writes asr_model_id and a new
asr_model_revision; the default model is pinned to its English-only
revision 947f59a, and --asr-revision sets it for a custom model (left
unpinned with a warning otherwise). Serving has no fallback: an audio
checkpoint without asr_model_id fails at startup with the config fix.
This is breaking for checkpoints composed before 0.2.0, hence the
version bump.

For airgapped customers (#158), --bundle-asr copies the ASR into the
checkpoint's asr/ folder, sets asr_bundled_path, and checks at compose
time that it loads with no network. Serving loads a bundled ASR from
disk, or fetches only asr/ when served by Hub repo id. --asr-model may
be a local directory so the bundle can be built offline.

Tests: transcript text is no longer asserted (model quality, not ours
to own). The real default ASR is downloaded only by the new `release`
tier (pytest -m release, /gpu-test-release), run before releases;
every other audio test uses a 30 KB local CTC model
(tests/shared/tiny_asr.py) and downloads no ASR.

Fixes #155
Fixes #158

Signed-off-by: aviv ron <rona@il.ibm.com>
The release tier introduced for the default ASR now also takes the
tests whose job is compatibility with real external artifacts, and the
non-representative cases of the expensive GPU matrices. PR runs keep
one case per matrix; `pytest -m release` (/gpu-test-release) runs the
rest. In the last full vllm26 run these matrices took roughly 1h40m of
2h36m.

Release only: TestRealHubMetadata, test_upstream_files.py,
test_quantization.py, and test_granite4_fullsize.py's GPU class.
Split by pytest.param marks: test_granite4_mini.py (4.0-micro on PRs),
test_multi_adapter_equivalence.py (4.1-3b on PRs, 4.2-3b in release),
test_audio_serving_smoke.py (4.0-micro on PRs, 4.1-3b in release).

test_quantization.py is gated on GRANITE_SWITCH_E2E_MODELS=1, which no
GPU pod sets, so it never ran in CI. tests/conftest.py now sets it when
-m selects release.

test_adapter_routing_audio_enabled.py stays on PRs but composes 13
empty built-in slots on 4.0-micro instead of three real adapter
libraries on 4.1-3b: routing reads only the control-token table.

Signed-off-by: aviv ron <rona@il.ibm.com>
pytest's -m replaces the addopts filter, so a bare -m "release" would
also run any test marked both deep and release.

Signed-off-by: aviv ron <rona@il.ibm.com>
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@aviv1ron1

Copy link
Copy Markdown
Collaborator Author

splitting into two seperate prs

@aviv1ron1 aviv1ron1 closed this Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

audio: include local weights for ASR audio: unpinned default ASR model changed on the Hub, breaking test_asr_ctc_default_gpu.py

2 participants