stb_vorbis: integer overflow in codebook multiplicands allocation

stb_vorbis.c:3880:
setup_malloc(f, sizeof(float) * c->entries * c->dimensions)

entries and dimensions come from the vorbis stream. entries=1048577, dimensions=4096: product overflows int32, setup_malloc gets 16KB, loop writes 16GB.

Fix: check for overflow before allocation, cast to size_t.

Fix: check for overflow before allocation, cast to size_t.

sizeof(float) is already size_t.

sizeof(float) is size_t, but entries * dimensions overflows as int * int before the promotion happens. The widening to size_t comes after the damage is done.

but entries * dimensions overflows as int * int before the promotion happens.

What tool are you using to come to these conclusions?

a * b * c is evaluated as (a * b) * c: https://c.godbolt.org/z/jfzz6q51n

Oh, okey, you are right. Sorry for my mistake.

It might still be worth sanitizing stuff like this rather than leaving it subtly correct.

It might still be worth sanitizing stuff like this rather than leaving it subtly correct.

I haven't checked if entries and dimensions are sanitized or not. If not, then even in size_t they can still overflow/wraparound, e.g on 32 bit systems, and allocate wrong amount.

Oh, okey, you are right. Sorry for my mistake.

Sorry for not explaining clearly, but my problem was with the suggested fix. The overflow bug still maybe valid as stated above. You can reopen the issue.

Okey. Thanks.

This issue was closed on incorrect grounds. While it's true that sizeof(float) * entries * dimensions evaluates correctly as size_t (left-to-right promotion), the product is then truncated to int when passed to setup_malloc(vorb *f, int sz). This truncation is the root cause of the heap overflow.

See #1947 for a complete analysis and a 205-byte PoC OGG file that triggers an ASan-detected heap-buffer-overflow in whisper.cpp via this exact path.

We partially patched this in libxmp by prefacing setup_malloc with the check

if (sz <= 0 || INT_MAX - 7 < sz) return NULL;

but we are affected by the same setup_malloc prototype bug.

From what I remember looking at the spec, Vorbis does not help bound these types of cases to 32-bit values. Properly fixing this requires a 64-bit integer type. (Also see #1168 (comment), where the c->lookup_values multiplication and thus resulting setup_temp_malloc has the same problem.)

static void *setup_malloc(vorb *f, uint64 sz)

(uint64)sizeof(float) * entries * dimensions

stb_vorbis: Increase setup_malloc() size parameter to uint64

stb_vorbis: Increase setup_malloc() size parameter to uint64 (#999)

stb_vorbis: Increase setup_malloc() size parameter to uint64

Ok this one might mess with everyone's head but the fix is not to do any of the code changes you all made, and I sort of pointed out in the other issue with this very same "bug"

Do validation of c->entries and c->dimensions, they both have limits. You check them when you set the value on the struct and if someone is playing with binary files, that when you know and signal "Bad File".
Just in case some stray cosmic ray flips a bit on entires and/or dimensions in setup_malloc() define a const that says "Th'all Shall No Alloc Over 1Gb" or some sensible number.
Ow yes I would also use size_t siz as the arg to setup_malloc() (pointed out in the other issue).

Do validation of c->entries and c->dimensions, they both have limits. You check them when you set the value on the struct and if someone is playing with binary files, that when you know and signal "Bad File".

Yes, c->entries and c->dimensions have limits specified by the Vorbis I specification, specifically (2^24-1) and (2^16-1). This results in an upper bound for lookup_values in the 40-bit range.

https://www.xiph.org/vorbis/doc/Vorbis_I_spec.html#x1-510003.2.1

16 bit [codebook_dimensions] and 24 bit [codebook_entries] fields:


1
2byte 3: [ X X X X X X X X ]
3byte 4: [ X X X X X X X X ] [codebook_dimensions] (16 bit unsigned)
4
5byte 5: [ X X X X X X X X ]
6byte 6: [ X X X X X X X X ]
7byte 7: [ X X X X X X X X ] [codebook_entries] (24 bit unsigned)

If there are lower bounds in the specification, please link the exact location of them.

Just in case some stray cosmic ray flips a bit on entires and/or dimensions in setup_malloc() define a const that says "Th'all Shall No Alloc Over 1Gb" or some sensible number.

My patch does exactly this (INT_MAX - 7), because there are user-facing fields for memory buffers that still use unsigned int and internal fields that use int, and changing these could adversely affect user code.

Ow yes I would also use size_t siz as the arg to setup_malloc() (pointed out in the other issue).

This doesn't fix the bug, which is that lookup_values has a >32-bit bound, even on 32-bit hardware, and size_t is 32-bit on most 32-bit hardware. Unless you can point to a source in the Vorbis I specification for a lower bound, the correct solution is to use uint64 math so the result can be correctly bounded after.

In this particular case I don't actually care too much what absolute limits the specification implies; I'm completely comfortable with stb_vorbis enforcing much tighter limits than the Vorbis spec does, since the Xiph encoder does not actually emit enormous codebooks, nor ever did so. Decoding every possible technically legal Vorbis file is a non-goal for stb_vorbis.

A quick survey of all the codebooks in libvorbis indicates that the largest lookup size you can get in a Vorbis file from a current encoder is 52488 lookup_values (attained for _16c0_s_p1_0 in coupled/res_books_stereo.h, which is 8*6561). None of the other codebook files have anything that hits even 16k lookup_values.

With some generous safety factor, I think that a 262144 (2^18) limit ought to be plenty under these circumstances, and that limits lookups to sizes that not just comfortably fit within 32 bits, but are also unlikely to lead to resource exhaustion attacks or similar. I'm loathe to introduce hard limits of that type in stb_image since some people genuinely deal with large images on a regular basis (I deal with 16384x16384 and more pretty commonly at work) but in this case, this is a limit on VQ codebook size not on the number of samples encoded in the Vorbis file or similar, and with the Xiph encoder never actually producing those, I can't think of a good reason to actually support them in the first place.

And a limit on the codebook size here boils down to something like
if (c->entries <= 0 || CODEBOOK_LIMIT / entries > c->dimensions) return error(f, VORBIS_invalid_setup);
which is dead simple. I'll be moving ahead with that if that's OK with everyone?

codebook size here boils down to something like
if (c->entries <= 0 || CODEBOOK_LIMIT / entries > c->dimensions) return error(f, VORBIS_invalid_setup);
which is dead simple. I'll be moving ahead with that if that's OK with everyone?

This should be OK—would you like me to apply it to #1960 (dropping uint64 altogether)?

edit: went ahead and pushed this update. Note that c->entries * c->dimensions > CODEBOOK_LIMIT → c->dimensions > CODEBOOK_LIMIT / c->entries is actually the error case (the inequality is wrong in the quoted line).

Thanks, much appreciated, LGTM! Will merge.

stb_vorbis: Overflow checks for various sizes.

Fix merged into dev branch, will be in the next release.

Navigation Menu

Repository navigation

stb_vorbis: integer overflow in codebook multiplicands allocation #1933

3 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

stb_vorbis: integer overflow in codebook multiplicands allocation #1933

Description

Activity

N-R-K commented on Apr 10, 2026

Medoedus commented on Apr 10, 2026

N-R-K commented on Apr 10, 2026

Medoedus commented on Apr 10, 2026

nothings commented on Apr 10, 2026

N-R-K commented on Apr 11, 2026

Medoedus commented on Apr 11, 2026

Tomer-PL commented on May 10, 2026

AliceLR commented on Jun 3, 2026

3 remaining items

madwax commented on Jun 12, 2026

AliceLR commented on Jun 12, 2026

rygorous commented on Oct 25, 2026

AliceLR commented on Oct 26, 2026

rygorous commented on Oct 26, 2026

rygorous commented on Oct 26, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions