Repository navigation
Conversation
The default ASR model was loaded by name with no revision, so the Hub repo's 2026-10-21 multilingual release silently changed every transcript (now capitalized and punctuated) and broke the GPU tests (#155). The default also lived only in the library, so a checkpoint's config did not show which ASR it ran. Compose with --enable-audio now always writes asr_model_id and a new asr_model_revision; the default model is pinned to its English-only revision 947f59a, and --asr-revision sets it for a custom model (left unpinned with a warning otherwise). Serving has no fallback: an audio checkpoint without asr_model_id fails at startup with the config fix. This is breaking for checkpoints composed before 0.2.0, hence the version bump. For airgapped customers (#158), --bundle-asr copies the ASR into the checkpoint's asr/ folder, sets asr_bundled_path, and checks at compose time that it loads with no network. Serving loads a bundled ASR from disk, or fetches only asr/ when served by Hub repo id. --asr-model may be a local directory so the bundle can be built offline. Tests: transcript text is no longer asserted (model quality, not ours to own). The real default ASR is downloaded only by the new `release` tier (pytest -m release, /gpu-test-release), run before releases; every other audio test uses a 30 KB local CTC model (tests/shared/tiny_asr.py) and downloads no ASR. Fixes #155 Fixes #158 Signed-off-by: aviv ron <rona@il.ibm.com>
The release tier introduced for the default ASR now also takes the tests whose job is compatibility with real external artifacts, and the non-representative cases of the expensive GPU matrices. PR runs keep one case per matrix; `pytest -m release` (/gpu-test-release) runs the rest. In the last full vllm26 run these matrices took roughly 1h40m of 2h36m. Release only: TestRealHubMetadata, test_upstream_files.py, test_quantization.py, and test_granite4_fullsize.py's GPU class. Split by pytest.param marks: test_granite4_mini.py (4.0-micro on PRs), test_multi_adapter_equivalence.py (4.1-3b on PRs, 4.2-3b in release), test_audio_serving_smoke.py (4.0-micro on PRs, 4.1-3b in release). test_quantization.py is gated on GRANITE_SWITCH_E2E_MODELS=1, which no GPU pod sets, so it never ran in CI. tests/conftest.py now sets it when -m selects release. test_adapter_routing_audio_enabled.py stays on PRs but composes 13 empty built-in slots on 4.0-micro instead of three real adapter libraries on 4.1-3b: routing reads only the control-token table. Signed-off-by: aviv ron <rona@il.ibm.com>
pytest's -m replaces the addopts filter, so a bare -m "release" would also run any test marked both deep and release. Signed-off-by: aviv ron <rona@il.ibm.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Collaborator
Author
|
splitting into two seperate prs |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #155
Fixes #158
Summary
Three related changes:
test_asr_ctc_default_gpu.py. The default also existed only inside the library, so a checkpoint's config didn't show which ASR it ran.--bundle-asrfor airgapped deployments. Builds the ASR weights into the checkpoint so it serves audio with no Hub access.releasetest tier. Real-model checks and the non-representative cases of the expensive GPU matrices move out of PR runs and run only before releases. On the current CI legs this is expected to move about 1h30m out of every full PR run down from 4h roughly to 2.5h.1. ASR pinned and named in the config (#155)
--enable-audio, compose always writesasr_model_idand a newasr_model_revision. The default modelibm-granite/granite-speech-5.0-470m-turboctcis pinned to its English-only revision947f59a.--asr-revisionflag. A custom--asr-modelwithout it composes with a warning and loads unpinned.asr_model_idfails at server startup, with the twoconfig.jsonlines that fix it.DEFAULT_ASR_MODEL_IDmoved togranite_switch.config.ASRTranscriberandget_transcribernow requiremodel_idand accept arevision.CHANGELOG.mdhas a Breaking section.docs/AUDIO.mdcovers the new fields, migrating older checkpoints, and the fact that an older library ignores the new fields: it loads the ASR unpinned and still goes online for a bundled checkpoint.2.
--bundle-asr(#158)--enable-audio. It copies the ASR into<output>/asr/(config, safetensors weights, tokenizer, feature extractor, README and license) and setsasr_bundled_path.--asr-modelcan be a local directory, so the bundle can be built inside the airgapped network.local_files_only=True.asr/is fetched, which also works offline from the HF cache.asr/*.safetensorsas LLM weights.docs/AUDIO.mdhas a new "Airgapped deployment" section.3. Release test tier
The new
releasemarker is excluded by default (addopts:not deep and not release) and runs withpytest -m "release and not deep", or/gpu-test-releaseonce the runner change below lands.The real ASR is downloaded only in the release tier.
test_asr_ctc_default_gpu.pyis release-only and no longer asserts transcript text, which is model quality. Every other audio test uses a 30 KB random CTC model built locally (tests/shared/tiny_asr.py).Release only:
tests/integration/test_asr_ctc_default_gpu.pytests/vllm/test_granite4_fullsize.py(GPU class)test_granite4_mini.pycovers the same equivalence at small size on every PRtests/composer/test_upstream_files.pytests/composer/test_selective_download.py::TestRealHubMetadatatests/hf/test_quantization.pySplit: one case on PRs, the rest in the release tier (
pytest.param(..., marks=pytest.mark.release)):tests/vllm/test_granite4_mini.py(3 vLLM classes)4.0-micro4.0-1b,4.0-350mtests/vllm/test_multi_adapter_equivalence.pygranite-4.1-3b(lora + sr)granite-4.2-3b(lora + sr)tests/integration/test_audio_serving_smoke.pygranite-4.0-microgranite-4.1-3bOther test changes:
test_quantization.pynever ran in CI: all 16 tests were skipped on vllm31 and vllm32, because it's gated onGRANITE_SWITCH_E2E_MODELS=1, which no GPU pod sets.tests/conftest.pynow sets the variable when-mselectsrelease.test_adapter_routing_audio_enabled.pystays on PRs but is now cheap. It composes 13 empty built-in slots ongranite-4.0-microinstead of three real adapter libraries ongranite-4.1-3b. Routing reads only the control-token table.test_audio_serving_smoke.pyandtest_audio_uncached_processor.pycompose with--bundle-asrand the tiny ASR, so they also cover serving a bundled ASR.docs/CICD.md: a new "Release test tier" section, plus a release-checklist step to run it.