[ohxiaoguang]

Problem

The type-2 residue decoder clamps the interleaved vector to n*2, even when more than two channels participate. This truncates residue decoding and desynchronizes subsequent codebook reads, producing severe distortion in 5.1 streams.

Vorbis I section 8.5 defines the type-2 vector as n*ch, where ch is the number of channels in the current submap: https://www.xiph.org/vorbis/doc/Vorbis_I_spec.html

Fix

• Use the current submap's channel count in decode_residue().
• Account for all stream channels when estimating the maximum type-2 classification scratch space in start_decoder(). The stream channel count is a conservative upper bound for any submap. This is necessary for caller-owned allocation buffers: changing only the decoding length causes the new synthetic six-channel test to fail with insufficient scratch memory.
• Add a standalone synthetic regression test and make -C tests test_vorbis_multichannel target. No external encoder or audio fixtures are required.

The scratch-space change is related to #1487 and the closed, unmerged #1889, but accounts for arbitrary supported channel counts rather than assuming stereo. Types 0 and 1 retain their existing bounds. Muted channels retain their positions in type-2 interleaving.

Reproduction and validation

Both six-channel inputs (test1.ogg, test2.ogg) attached to #1594 reproduce the failure on unmodified master, and decode correctly with this patch. Each input has 2,660,352 frames at 48 kHz. Full-stream float32 comparisons against FFmpeg 9.0.2 use the appropriate Vorbis-to-FFmpeg channel permutation, with zero sample offset and no head/tail trimming, gain adjustment, or filtering. Both outputs contain the same frame count as the reference.

Input	Original maximum PCM16 error	Fixed maximum PCM16 error	Fixed worst-channel PCM16 RMS error
test1.ogg	64587	1	0.020628
test2.ogg	65107	1	0.021150

Float RMS error is at most 2.19e-8 per channel after the fix. PCM16 figures apply the same rounding and saturation to both float outputs. The two rear channels in these public files are silent; the synthetic test separately checks every channel's residue coefficients.

Tests run on Apple Silicon with Apple Clang 21:

• 24 synthetic configurations covering mono/stereo residue types 0/1/2; 3/6/8-channel type 2; block sizes 64, 256, 1024, 4096, and additional six-channel 8192 cases; and two-submap cases.
• Heap and exactly-sized caller-owned arenas, with both pull and push APIs. Default and STB_VORBIS_DIVIDES_IN_RESIDUE builds pass ASan/UBSan: 192 configuration/API/allocation combinations in total. Partial/all-channel skip paths are checked directly.
• Negative controls: unmodified master fails the three-channel coefficient assertion; the decoding-length fix without the scratch-budget correction fails in the six-channel arena path under ASan/UBSan.
• Both complete public inputs pass ASan/UBSan with exactly-sized arenas; their output is byte-identical to the ordinary heap decode.
• The Makefile regression target passes. Library compilation passes as C++11 and as C89 with the existing GNU inline macro adapted via -Dinline=__inline__, including -Werror=declaration-after-statement.

Earlier, separate nxnative testing reproduced this problem in Link's Awakening's 5.1 Vorbis movie audio. The patched macOS build was verified by listening; its three tested movie tracks were within one PCM16 unit of FFmpeg. That earlier game verification is distinct from the new upstream/public-fixture tests above. iOS device audio has not been verified. No commercial assets or public audio attachments are included in this PR.

Fixes #1594.

Related to #879: no public input was available for independent reproduction, so this PR does not claim to establish the same root cause or close that issue.

Disclosure: AI tools assisted with the implementation, synthetic regression test, numerical analysis, and this PR description. This disclosure does not imply an exception to the repository's contribution policy.

[nothings]

This disclosure does not imply an exception to the repository's contribution policy.

How do you figure that?

[ohxiaoguang]

This disclosure does not imply an exception to the repository's contribution policy.

How do you figure that?

I felt this bug was important enough to report and propose a fix. Although I used AI assistance, the resulting changes are straightforward and easy to understand and review. That's why I decided to submit the PR.

I understand and respect the project's policy and your decision.

[nothings]

AI