From cabdb72ac95add9c3a7a31f5ddc338598cd8cbd7 Mon Sep 17 00:00:00 2001 From: Davide Angelocola Date: Wed, 5 Aug 2026 21:21:26 +0200 Subject: [PATCH 1/2] fix(reader): harden VarBin/Dict/Bitpacked/ALP/Sparse/Chunked/Struct against malformed input MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Part of TODO.md's "Per-encoding adversarial tests" security item (CLAUDE.md §Security contract): a malformed file must always throw VortexException, never a raw JDK exception. - DecodeContext.buffer(i)/decodeChild(i): centralize bounds checks on both the position and the segment/child index it holds. - VarBinArray (OffsetMode/DictMode/ViewMode): checkedLength() guards NegativeArraySizeException/IndexOutOfBoundsException from corrupted offsets; forEachByteLength paths get boundary catch-and-wrap without per-element branches. - DictEncodingDecoder: expand() guards the values-pool index and empty-child division by zero; legacy-dict path routed through the hardened buffer accessor instead of raw array indexing. - BitpackedEncodingDecoder: bit_width/offset range-checked upfront; packed-buffer-too-short wrapped at the unpack call site. - AlpEncodingDecoder: exponent table indices and patch indices range-checked; empty encoded/patch children guarded. - SparseEncodingDecoder: patch count bounded by row count; empty patch children rejected; patch-value casts checked instead of raw; missing patches metadata rejected; varbin merge loop boundary-wrapped. - SparseArrays.walkPatches: rejects non-ascending patch indices, which previously desynced the fill/patch cursor and over-emitted callbacks past the array's own length. - ChunkedEncodingDecoder: chunk offsets validated non-negative, non-decreasing, and spanning exactly rowCount; zero chunks with nonzero rowCount rejected uniformly across dtypes. - StructEncodingDecoder: added test-only coverage (guards already existed at the DType/PostscriptParser layer). - PrimitiveEncodingDecoder: reject a non-primitive requested dtype instead of an unchecked cast (a child can be decoded under a dtype not its own). TODO.md: mark these encodings done, seven of eleven per-encoding gotchas now closed. --- TODO.md | 19 +- .../vortex/reader/array/SparseArrays.java | 12 + .../dfa1/vortex/reader/array/VarBinArray.java | 219 ++++++++++++++-- .../reader/decode/AlpEncodingDecoder.java | 62 +++++ .../decode/BitpackedEncodingDecoder.java | 31 ++- .../reader/decode/ChunkedEncodingDecoder.java | 45 +++- .../vortex/reader/decode/DecodeContext.java | 68 ++++- .../reader/decode/DictEncodingDecoder.java | 53 +++- .../decode/PrimitiveEncodingDecoder.java | 9 +- .../reader/decode/SparseEncodingDecoder.java | 146 ++++++++--- .../reader/array/LazySparseArrayTest.java | 39 +++ .../reader/decode/AlpEncodingDecoderTest.java | 134 +++++++++- .../decode/BitpackedEncodingDecoderTest.java | 90 +++++++ .../decode/ChunkedEncodingDecoderTest.java | 148 +++++++++++ .../reader/decode/DecodeContextTest.java | 147 +++++++++++ .../decode/DictEncodingDecoderTest.java | 247 +++++++++++++++++- .../decode/SparseEncodingDecoderTest.java | 171 ++++++++++++ .../decode/StructEncodingDecoderTest.java | 168 ++++++++++++ .../decode/VarBinEncodingDecoderTest.java | 107 ++++++++ .../decode/VarBinViewEncodingDecoderTest.java | 118 +++++++++ 20 files changed, 1929 insertions(+), 104 deletions(-) create mode 100644 reader/src/test/java/io/github/dfa1/vortex/reader/decode/ChunkedEncodingDecoderTest.java create mode 100644 reader/src/test/java/io/github/dfa1/vortex/reader/decode/DecodeContextTest.java create mode 100644 reader/src/test/java/io/github/dfa1/vortex/reader/decode/StructEncodingDecoderTest.java diff --git a/TODO.md b/TODO.md index 2439be914..92f59f8a1 100644 --- a/TODO.md +++ b/TODO.md @@ -35,20 +35,11 @@ known gap, a contract audit, or supporting infra. ### Per-encoding adversarial tests -Each encoding's `decode(DecodeContext)` should be exercised against: -- `bufferIndices[i] >= ctx.bufferCount()` → centralize check in `DecodeContext.buffer(i)`. -- Crafted metadata that decodes but disagrees with the buffer payload. - -Per-encoding gotchas: -- [ ] **VarBin**: offsets non-monotonic, negative, past data-buffer length. -- [ ] **Dict**: `codes[i] >= values.length`; `codes` ptype declared u8 but values count > 256. -- [ ] **Bitpacked**: `bit_width < 0 || > 64`; `packed_len < n * bit_width / 8`. -- [ ] **ALP**: `dim < 0`, `f_or_d` byte out of enum range; `exceptions_count > n`. -- [ ] **Sparse**: indices non-sorted or `indices[i] >= length`; values count - mismatches indices count. -- [ ] **Chunked**: zero children with non-zero `row_count`; child layout self-referencing - (already protected by depth limit, but add explicit test). -- [ ] **Struct**: `fieldNames.size() != children.size()`; field name UTF-8 invalid. +Each encoding's `decode(DecodeContext)` should be exercised against crafted metadata that +decodes but disagrees with the buffer payload. `bufferIndices[i] >= ctx.bufferCount()` (and the +equivalent child-index check) is centralized in `DecodeContext.buffer(i)`/`decodeChild(i)`. +VarBin, Dict, Bitpacked, ALP, Sparse, Chunked, and Struct are done — remaining gotchas: + - [ ] **RLE / RunEnd**: `run_ends` non-monotonic; last `run_end` ≠ `row_count`. - [ ] **Constant**: protobuf scalar value missing or type-mismatched against declared `DType`. - [ ] **Zoned**: zone-map min > max; zone count ≠ child chunk count. diff --git a/reader/src/main/java/io/github/dfa1/vortex/reader/array/SparseArrays.java b/reader/src/main/java/io/github/dfa1/vortex/reader/array/SparseArrays.java index a9d591a71..7c8ffc809 100644 --- a/reader/src/main/java/io/github/dfa1/vortex/reader/array/SparseArrays.java +++ b/reader/src/main/java/io/github/dfa1/vortex/reader/array/SparseArrays.java @@ -89,6 +89,18 @@ static void walkPatches(Array patchIndices, long numPatches, long absStart, long if (patchAbs >= absEnd) { break; } + // patchIndices is a wire-supplied, untrusted array that the format requires to be + // sorted ascending: findFirstAtOrAfter's binary search assumes it, and this walk + // advances `pos` to `patchAbs + 1` on each patch. A patch that goes backwards moves + // `pos` backwards too, so a later fill/patch run re-covers already-emitted + // positions — the walk then emits more callbacks than `absEnd - absStart`, which + // overflows a caller's fixed-size output buffer as a raw IndexOutOfBoundsException + // rather than a VortexException (ADR 0003). Checked once per patch, not per row. + if (patchAbs < pos) { + throw new VortexException( + "Sparse patch indices not sorted: index " + patchAbs + " at patch " + p + + " precedes position " + pos); + } for (long r = pos; r < patchAbs; r++) { fillSlot.run(); } diff --git a/reader/src/main/java/io/github/dfa1/vortex/reader/array/VarBinArray.java b/reader/src/main/java/io/github/dfa1/vortex/reader/array/VarBinArray.java index bcae7dcef..95fc822a9 100644 --- a/reader/src/main/java/io/github/dfa1/vortex/reader/array/VarBinArray.java +++ b/reader/src/main/java/io/github/dfa1/vortex/reader/array/VarBinArray.java @@ -169,6 +169,29 @@ static OffsetMode toOffsetMode(VarBinArray src, SegmentAllocator arena) { return new OffsetMode(src.dtype(), n, outBytes.asReadOnly(), outOffsets, PType.I64); } + /// Validates that element bytes `[start, end)` lie inside `bytes` and returns the + /// element length. + /// + /// Offsets arrive from an untrusted file and are deliberately not scanned at decode + /// time (VarBin decode stays zero-copy and lazy), so a non-monotonic, negative or + /// past-the-end pair has to be rejected here — as a [VortexException], never as a raw + /// `NegativeArraySizeException` from `new byte[end - start]` or an + /// `IndexOutOfBoundsException` from [MemorySegment#copy(MemorySegment, long, MemorySegment, long, long)] + /// (ADR 0003). + /// + /// @param bytes data buffer the offsets index into + /// @param start start offset of the element + /// @param end end offset of the element, exclusive + /// @return the element length in bytes + private static int checkedLength(MemorySegment bytes, long start, long end) { + long len = end - start; + if (start < 0 || len < 0 || end > bytes.byteSize() || len > Integer.MAX_VALUE) { + throw new VortexException("varbin element bytes [" + start + ", " + end + + ") out of range for a data buffer of " + bytes.byteSize() + " bytes"); + } + return (int) len; + } + /// Creates a dict-mode `VarBinArray`. Lengths and bytes are resolved via the /// dictionary on each access; no string materialization occurs at construction time. /// @@ -206,8 +229,9 @@ record OffsetMode(DType dtype, long length, MemorySegment bytesSegment, public byte[] getBytes(long i) { long start = readOffset(i); long end = readOffset(i + 1); - byte[] out = new byte[(int) (end - start)]; - MemorySegment.copy(bytesSegment, start, MemorySegment.ofArray(out), 0, end - start); + int len = checkedLength(bytesSegment, start, end); + byte[] out = new byte[len]; + MemorySegment.copy(bytesSegment, start, MemorySegment.ofArray(out), 0, len); return out; } @@ -218,12 +242,18 @@ public String getString(long i) { @Override public int getByteLength(long i) { - return (int) (readOffset(i + 1) - readOffset(i)); + return checkedLength(bytesSegment, readOffset(i), readOffset(i + 1)); } @Override public void forEachByteLength(IntConsumer c) { long n = length; + // The loop reads offsets[0..n] at a fixed stride and must stay uniform to + // vectorize (CLAUDE.md hot-loop rule), so the untrusted offsets segment is + // sized once here rather than bounds-checked per row. A non-monotonic pair + // still yields a negative length to the consumer; the typed accessors + // ([#getBytes(long)], [#getByteLength(long)]) reject it when the row is read. + checkOffsetsExtent(n); if (offsetsPtype == PType.I32 || offsetsPtype == PType.U32) { for (long i = 0; i < n; i++) { c.accept(offsetsSegment.getAtIndex(VortexFormat.LE_INT, i + 1) @@ -242,19 +272,52 @@ public VarBinArray limited(long rows) { if (rows >= length) { return this; } + checkOffsetsExtent(rows); long byteEnd = readOffset(rows); - int offBytes = (offsetsPtype == PType.I32 || offsetsPtype == PType.U32) - ? Integer.BYTES : Long.BYTES; + if (byteEnd < 0 || byteEnd > bytesSegment.byteSize()) { + throw new VortexException("varbin offset " + byteEnd + " at row " + rows + + " out of range for a data buffer of " + bytesSegment.byteSize() + " bytes"); + } + int offBytes = offsetWidth(); MemorySegment newOffsetsSeg = offsetsSegment.asSlice(0, (rows + 1) * offBytes); return new OffsetMode(dtype, rows, - bytesSegment.asSlice(0, byteEnd > 0 ? byteEnd : 0), newOffsetsSeg, offsetsPtype); + bytesSegment.asSlice(0, byteEnd), newOffsetsSeg, offsetsPtype); + } + + /// Verifies the offsets segment holds the `rows + 1` offsets the array claims. + /// + /// @param rows number of rows whose offsets are about to be read + private void checkOffsetsExtent(long rows) { + int width = offsetWidth(); + if (rows + 1 > offsetsSegment.byteSize() / width) { + throw new VortexException("varbin offsets segment of " + offsetsSegment.byteSize() + + " bytes holds fewer than " + (rows + 1) + " " + offsetsPtype + " offsets"); + } } + private int offsetWidth() { + return (offsetsPtype == PType.I32 || offsetsPtype == PType.U32) ? Integer.BYTES : Long.BYTES; + } + + /// Reads offset `i` at the width of [#offsetsPtype]. + /// + /// The offsets segment comes straight from an untrusted file, so an index past its + /// end must surface as a [VortexException] rather than a raw + /// `IndexOutOfBoundsException` (ADR 0003). + /// + /// @param i zero-based offset index, in `[0, length]` + /// @return the offset value widened to a signed long private long readOffset(long i) { - if (offsetsPtype == PType.I32 || offsetsPtype == PType.U32) { - return offsetsSegment.getAtIndex(VortexFormat.LE_INT, i); + try { + if (offsetsPtype == PType.I32 || offsetsPtype == PType.U32) { + return offsetsSegment.getAtIndex(VortexFormat.LE_INT, i); + } + return offsetsSegment.getAtIndex(VortexFormat.LE_LONG, i); + } catch (IndexOutOfBoundsException e) { + throw new VortexException("varbin offset index " + i + " (" + offsetsPtype + + ") out of range for an offsets segment of " + + offsetsSegment.byteSize() + " bytes", e); } - return offsetsSegment.getAtIndex(VortexFormat.LE_LONG, i); } } @@ -281,8 +344,9 @@ public byte[] getBytes(long i) { long code = dictReadCode(i); long start = dictReadOff(code); long end = dictReadOff(code + 1); - byte[] out = new byte[(int) (end - start)]; - MemorySegment.copy(bytesSegment, start, MemorySegment.ofArray(out), 0, end - start); + int len = checkedLength(bytesSegment, start, end); + byte[] out = new byte[len]; + MemorySegment.copy(bytesSegment, start, MemorySegment.ofArray(out), 0, len); return out; } @@ -294,7 +358,7 @@ public String getString(long i) { @Override public int getByteLength(long i) { long code = dictReadCode(i); - return (int) (dictReadOff(code + 1) - dictReadOff(code)); + return checkedLength(bytesSegment, dictReadOff(code), dictReadOff(code + 1)); } @Override @@ -307,16 +371,54 @@ public void forEachByteLength(IntConsumer c) { // most common (FSST + most dict encodings emit 32-bit offsets), so the fast // path reads offsets at a constant 4-byte stride and branch-splits the code // read once; wider offset ptypes take the general per-row path. - if (dictValOffPType == PType.I32) { - forEachI32OffsetByteLength(c); - } else { - for (long i = 0; i < length; i++) { - long code = dictReadCode(i); - c.accept((int) (dictReadOff(code + 1) - dictReadOff(code))); + // + // The per-row body must stay branch-free, so the untrusted-code bounds check is + // a boundary catch-and-wrap around the whole loop rather than a test per row: + // an out-of-pool code (or a truncated codes buffer) trips the segment access and + // must surface as a VortexException, never a raw IndexOutOfBoundsException. The + // handler re-validates on the cold path so the blame — and the exception type — + // land on whichever input actually failed, including the caller's own consumer. + try { + if (dictValOffPType == PType.I32) { + forEachI32OffsetByteLength(c); + } else { + for (long i = 0; i < length; i++) { + long code = dictReadCode(i); + c.accept((int) (dictReadOff(code + 1) - dictReadOff(code))); + } } + } catch (IndexOutOfBoundsException e) { + throw attribute(e); } } + /// Cold path: works out which untrusted input made a bulk length walk run off the + /// end, so the message names the segment that actually failed. + /// + /// Re-checks the codes extent, then every code against the value-offsets extent — + /// affordable here because it only runs after something has already thrown. When + /// both check out the failure came from the caller's [IntConsumer], and that + /// exception is returned unchanged rather than relabeled as malformed input. + /// + /// @param e the out-of-bounds failure raised by the walk + /// @return a [VortexException] describing the malformed input, or `e` itself + private RuntimeException attribute(IndexOutOfBoundsException e) { + int codeWidth = dictCodesPType.byteSize(); + if (length > dictCodesSegs.byteSize() / codeWidth) { + return new VortexException("dict codes segment of " + dictCodesSegs.byteSize() + + " bytes holds fewer than " + length + " " + dictCodesPType + " codes"); + } + long offsetCount = dictValOffsets.byteSize() / dictValOffPType.byteSize(); + for (long i = 0; i < length; i++) { + long code = readCodeAt(i); + if (code < 0 || code + 1 >= offsetCount) { + return new VortexException("dict code " + code + " at row " + i + + " out of range for " + offsetCount + " value offsets"); + } + } + return e; + } + /// Fast path of [#forEachByteLength(IntConsumer)] for I32 dict-value offsets: the /// code-ptype switch is hoisted out of the loop so each specialized loop body reads /// codes at a single fixed stride and computes lengths from a constant 4-byte offset @@ -376,11 +478,33 @@ public VarBinArray limited(long rows) { return this; } int codeBytes = dictCodesPType.byteSize(); + if (rows > dictCodesSegs.byteSize() / codeBytes) { + throw new VortexException("dict codes segment of " + dictCodesSegs.byteSize() + + " bytes holds fewer than " + rows + " " + dictCodesPType + " codes"); + } return VarBinArray.ofDict(dtype, rows, bytesSegment, dictValOffsets, dictValOffPType, dictCodesSegs.asSlice(0, rows * codeBytes), dictCodesPType); } + /// Reads the dictionary code for row `i` at the width of [#dictCodesPType]. + /// + /// The codes buffer is untrusted and may be shorter than [#length()], so an + /// overrun is reported as a [VortexException] instead of a raw + /// `IndexOutOfBoundsException` (ADR 0003). + /// + /// @param i zero-based row index + /// @return the dictionary code, widened to a signed long private long dictReadCode(long i) { + try { + return readCodeAt(i); + } catch (IndexOutOfBoundsException e) { + throw new VortexException("dict code index " + i + " (" + dictCodesPType + + ") out of range for a codes segment of " + + dictCodesSegs.byteSize() + " bytes", e); + } + } + + private long readCodeAt(long i) { return switch (dictCodesPType) { case U8 -> Byte.toUnsignedLong(dictCodesSegs.get(ValueLayout.JAVA_BYTE, i)); case U16 -> Short.toUnsignedLong(dictCodesSegs.getAtIndex(VortexFormat.LE_SHORT, i)); @@ -629,20 +753,31 @@ public Optional segmentIfPresent() { @Override public int getByteLength(long i) { - return views.get(VortexFormat.LE_INT, i * VIEW_SIZE); + return checkedSize(views.get(VortexFormat.LE_INT, viewOffset(i))); } @Override public byte[] getBytes(long i) { - long viewOff = i * VIEW_SIZE; - int size = views.get(VortexFormat.LE_INT, viewOff); + long viewOff = viewOffset(i); + int size = checkedSize(views.get(VortexFormat.LE_INT, viewOff)); byte[] out = new byte[size]; if (size <= MAX_INLINED_SIZE) { + // Inlined data always fits the remaining 12 bytes of the view itself. MemorySegment.copy(views, viewOff + 4, MemorySegment.ofArray(out), 0, size); } else { int bufferIndex = views.get(VortexFormat.LE_INT, viewOff + 8); long srcOffset = Integer.toUnsignedLong(views.get(VortexFormat.LE_INT, viewOff + 12)); - MemorySegment.copy(dataBufs[bufferIndex], srcOffset, MemorySegment.ofArray(out), 0, size); + if (bufferIndex < 0 || bufferIndex >= dataBufs.length) { + throw new VortexException("varbin view at row " + i + " references data buffer " + + bufferIndex + " of " + dataBufs.length); + } + MemorySegment buf = dataBufs[bufferIndex]; + if (srcOffset + size > buf.byteSize()) { + throw new VortexException("varbin view bytes [" + srcOffset + ", " + + (srcOffset + size) + ") out of range for data buffer " + bufferIndex + + " of " + buf.byteSize() + " bytes"); + } + MemorySegment.copy(buf, srcOffset, MemorySegment.ofArray(out), 0, size); } return out; } @@ -655,6 +790,10 @@ public String getString(long i) { @Override public void forEachByteLength(IntConsumer c) { long n = length; + // Sized once, outside the loop, so the per-row body stays uniform — same + // trade-off as OffsetMode: a negative size on the wire still reaches the + // consumer, and [#getBytes(long)] rejects it when the row is read. + checkViewsExtent(n); for (long i = 0; i < n; i++) { c.accept(views.get(VortexFormat.LE_INT, i * VIEW_SIZE)); } @@ -665,7 +804,43 @@ public VarBinArray limited(long rows) { if (rows >= length) { return this; } + checkViewsExtent(rows); return new ViewMode(dtype, rows, views.asSlice(0, rows * VIEW_SIZE), dataBufs); } + + /// Byte offset of view `i`, rejecting a row the views segment does not cover. + /// + /// @param i zero-based row index + /// @return the byte offset of the 16-byte view for row `i` + private long viewOffset(long i) { + long off = i * VIEW_SIZE; + if (i < 0 || off + VIEW_SIZE > views.byteSize()) { + throw new VortexException("varbin view index " + i + + " out of range for a views segment of " + views.byteSize() + " bytes"); + } + return off; + } + + /// Verifies the views segment holds `rows` complete 16-byte views. + /// + /// @param rows number of rows whose views are about to be read + private void checkViewsExtent(long rows) { + if (rows > views.byteSize() / VIEW_SIZE) { + throw new VortexException("varbin views segment of " + views.byteSize() + + " bytes holds fewer than " + rows + " views"); + } + } + + /// Rejects a negative element size read from a view header, which would otherwise + /// reach `new byte[size]` as a `NegativeArraySizeException` (ADR 0003). + /// + /// @param size element size read from the view + /// @return `size` when it is non-negative + private static int checkedSize(int size) { + if (size < 0) { + throw new VortexException("negative varbin view size " + size); + } + return size; + } } } diff --git a/reader/src/main/java/io/github/dfa1/vortex/reader/decode/AlpEncodingDecoder.java b/reader/src/main/java/io/github/dfa1/vortex/reader/decode/AlpEncodingDecoder.java index 91eb16b34..beff79dba 100644 --- a/reader/src/main/java/io/github/dfa1/vortex/reader/decode/AlpEncodingDecoder.java +++ b/reader/src/main/java/io/github/dfa1/vortex/reader/decode/AlpEncodingDecoder.java @@ -82,6 +82,7 @@ public Array decode(DecodeContext ctx) { private static Array decodeF64(DecodeContext ctx, ProtoALPMetadata meta, int expE, int expF, long n, MemorySegment src) { + checkExponents(expE, expF, F10_F64.length); // Decode formula mirrors the Rust reference (`ALPFloat::decode_single`): two-step // `encoded * F10[f] * IF10[e]`. A pre-multiplied `scale = F10[f] * IF10[e]` // gives different IEEE rounding for non-trivial `expF`, breaking round-trip with @@ -89,6 +90,7 @@ private static Array decodeF64(DecodeContext ctx, ProtoALPMetadata meta, int exp double df = F10_F64[expF]; double de = IF10_F64[expE]; long srcCap = SegmentBroadcast.capacity(src, 8); + checkSource(srcCap, n); if (meta.patches() == null) { if (srcCap >= n) { @@ -116,9 +118,11 @@ private static Array decodeF64(DecodeContext ctx, ProtoALPMetadata meta, int exp private static Array decodeF32(DecodeContext ctx, ProtoALPMetadata meta, int expE, int expF, long n, MemorySegment src) { + checkExponents(expE, expF, F10_F32.length); float df = F10_F32[expF]; float de = IF10_F32[expE]; long srcCap = SegmentBroadcast.capacity(src, 4); + checkSource(srcCap, n); if (meta.patches() == null) { if (srcCap >= n) { @@ -144,30 +148,88 @@ private static Array decodeF32(DecodeContext ctx, ProtoALPMetadata meta, int exp return new MaterializedFloatArray(ctx.dtype(), n, buf.asReadOnly()); } + /// Rejects out-of-range ALP exponents before they index the power-of-ten tables. + /// + /// `exp_e` and `exp_f` are untrusted metadata read verbatim from the file; a negative or + /// oversized exponent used to escape as an `ArrayIndexOutOfBoundsException` from the + /// table lookup rather than a [VortexException] (ADR 0003). The check is O(1) and runs + /// once per decode, outside every loop. + /// + /// @param expE the `e` exponent, an index into the inverse power-of-ten table + /// @param expF the `f` exponent, an index into the power-of-ten table + /// @param maxExp exclusive upper bound — the table length for this float width + private static void checkExponents(int expE, int expF, int maxExp) { + if (expE < 0 || expE >= maxExp || expF < 0 || expF >= maxExp) { + throw new VortexException(EncodingId.VORTEX_ALP, + "exponents (e=" + expE + ", f=" + expF + ") out of range [0," + maxExp + ")"); + } + } + + /// Rejects an encoded child that carries no element at all. + /// + /// Every path below reads at least element 0 — the broadcast path reads exactly it, and + /// the row-wise paths wrap with `% srcCap` — so a zero-length child would either read + /// off the segment or divide by zero. Checked once, outside the row loops. + /// + /// @param srcCap number of elements physically present in the encoded child + /// @param n logical row count + private static void checkSource(long srcCap, long n) { + if (srcCap == 0 && n > 0) { + throw new VortexException(EncodingId.VORTEX_ALP, + "empty encoded child for " + n + " rows"); + } + } + private static void applyPatches(DecodeContext ctx, ProtoPatchesMetadata pm, MemorySegment out, int elemBytes) { long numPatches = pm.len(); + if (numPatches == 0) { + return; + } long offset = pm.offset(); PType idxPtype = PType.fromOrdinal(pm.indices_ptype().value()); int idxBytes = idxPtype.byteSize(); + long n = out.byteSize() / elemBytes; MemorySegment idxSeg = ctx.decodeChildSegment(1, new DType.Primitive(idxPtype, false), numPatches); MemorySegment valSeg = ctx.decodeChildSegment(2, ctx.dtype(), numPatches); long idxCap = SegmentBroadcast.capacity(idxSeg, idxBytes); long valCap = SegmentBroadcast.capacity(valSeg, elemBytes); + if (idxCap == 0 || valCap == 0) { + throw new VortexException(EncodingId.VORTEX_ALP, + "empty patch child for " + numPatches + " declared patches (indices=" + + idxSeg.byteSize() + " bytes, values=" + valSeg.byteSize() + " bytes)"); + } if (idxCap >= numPatches && valCap >= numPatches) { for (long i = 0; i < numPatches; i++) { long absIdx = readUnsigned(idxSeg, i * idxBytes, idxPtype) - offset; + checkPatchIndex(absIdx, n); MemorySegment.copy(valSeg, i * elemBytes, out, absIdx * elemBytes, elemBytes); } } else { for (long i = 0; i < numPatches; i++) { long absIdx = readUnsigned(idxSeg, (i % idxCap) * idxBytes, idxPtype) - offset; + checkPatchIndex(absIdx, n); MemorySegment.copy(valSeg, (i % valCap) * elemBytes, out, absIdx * elemBytes, elemBytes); } } } + /// Guards a patch scatter-write against an untrusted index, mirroring the identical + /// check in [BitpackedEncodingDecoder]: a patch index outside the row range (or one + /// pushed negative by `patches.offset`) must fail as a [VortexException] rather than as + /// a raw `IndexOutOfBoundsException` from the copy. Patches are sparse, so the per-patch + /// test costs nothing on the row-wise decode loops. + /// + /// @param absIdx absolute row index of the patch, after subtracting `patches.offset` + /// @param n logical row count + private static void checkPatchIndex(long absIdx, long n) { + if (absIdx < 0 || absIdx >= n) { + throw new VortexException(EncodingId.VORTEX_ALP, + "patch index " + absIdx + " out of range [0," + n + ")"); + } + } + private static long readUnsigned(MemorySegment seg, long off, PType ptype) { return switch (ptype) { case U8 -> Byte.toUnsignedLong(seg.get(ValueLayout.JAVA_BYTE, off)); diff --git a/reader/src/main/java/io/github/dfa1/vortex/reader/decode/BitpackedEncodingDecoder.java b/reader/src/main/java/io/github/dfa1/vortex/reader/decode/BitpackedEncodingDecoder.java index 46e9f4e56..f012027c5 100644 --- a/reader/src/main/java/io/github/dfa1/vortex/reader/decode/BitpackedEncodingDecoder.java +++ b/reader/src/main/java/io/github/dfa1/vortex/reader/decode/BitpackedEncodingDecoder.java @@ -23,6 +23,10 @@ public final class BitpackedEncodingDecoder implements EncodingDecoder { private static final int[] FL_ORDER = {0, 4, 2, 6, 1, 5, 3, 7}; + /// Elements per FastLanes block; also the exclusive upper bound of the in-block + /// `offset` carried in the metadata. + private static final int FL_BLOCK_SIZE = 1024; + @Override public EncodingId encodingId() { return EncodingId.FASTLANES_BITPACKED; @@ -52,9 +56,34 @@ public Array decode(DecodeContext ctx) { int typeBits = ptype.bits(); long rowCount = ctx.rowCount(); + // bit_width and offset come straight off the wire and drive every byte offset the + // unpack loops compute, so both are range-checked up front (O(1), outside any + // loop). The width bound is the column's own element width, not a flat 64: a + // 40-bit width on an I8 column is just as malformed as a 65-bit one. The offset is + // a position inside the first 1024-element FastLanes block, and an oversized one + // would spin through billions of skipped lanes before failing. + if (bitWidth < 0 || bitWidth > typeBits) { + throw new VortexException(EncodingId.FASTLANES_BITPACKED, + "bit width " + bitWidth + " out of range [0," + typeBits + "] for " + ptype); + } + if (offset < 0 || offset >= FL_BLOCK_SIZE) { + throw new VortexException(EncodingId.FASTLANES_BITPACKED, + "offset " + offset + " out of range [0," + FL_BLOCK_SIZE + ")"); + } + MemorySegment packed = ctx.buffer(0); MemorySegment output = ctx.arena().allocate(rowCount * ptype.byteSize()); - fastlanesUnpackToSeg(packed, bitWidth, offset, typeBits, rowCount, output); + // A packed buffer shorter than the block/lane math requires overruns `packed`. + // Computing the exact required length here would duplicate that math for every + // width, so the overrun is caught at the call boundary and reported as a + // VortexException instead of a raw IndexOutOfBoundsException (ADR 0003). + try { + fastlanesUnpackToSeg(packed, bitWidth, offset, typeBits, rowCount, output); + } catch (IndexOutOfBoundsException e) { + throw new VortexException(EncodingId.FASTLANES_BITPACKED, + "packed buffer of " + packed.byteSize() + " bytes is too small for " + + rowCount + " rows at bit width " + bitWidth, e); + } if (meta.patches() != null) { applyPatches(ctx, meta.patches(), output, ptype.byteSize()); diff --git a/reader/src/main/java/io/github/dfa1/vortex/reader/decode/ChunkedEncodingDecoder.java b/reader/src/main/java/io/github/dfa1/vortex/reader/decode/ChunkedEncodingDecoder.java index 4084c4794..a95e99308 100644 --- a/reader/src/main/java/io/github/dfa1/vortex/reader/decode/ChunkedEncodingDecoder.java +++ b/reader/src/main/java/io/github/dfa1/vortex/reader/decode/ChunkedEncodingDecoder.java @@ -36,7 +36,16 @@ public Array decode(DecodeContext ctx) { "needs at least one child (chunk offsets)"); } int nchunks = nchildren - 1; - long[] offsets = readOffsets(ctx, nchunks); + // A chunked node with only the offsets child carries no data at all, so it can never + // describe a non-empty column. Rejecting it here names the real defect uniformly; + // letting it through would surface as the less direct `Chunked*Array` "empty chunk + // list" error for most dtypes, and as a silently empty zero-field struct for + // DType.Struct (any struct with a field routes through the same "empty chunk list"). + if (nchunks == 0 && ctx.rowCount() > 0) { + throw new VortexException(EncodingId.VORTEX_CHUNKED, + "no chunks for " + ctx.rowCount() + " row(s)"); + } + long[] offsets = readOffsets(ctx, nchunks, ctx.rowCount()); DType dtype = ctx.dtype(); List chunks = new ArrayList<>(nchunks); @@ -48,12 +57,40 @@ public Array decode(DecodeContext ctx) { return wrap(chunks, dtype, ctx.rowCount()); } - private static long[] readOffsets(DecodeContext ctx, int nchunks) { + /// Reads the `nchunks + 1` cumulative chunk offsets and validates them. + /// + /// The offsets buffer is untrusted: an empty one would make the broadcast wrap divide by + /// zero, and a non-monotonic or negative pair yields a negative chunk length that flows + /// into the child decode and then into a `Chunked*Array` whose `offsets` are no longer + /// sorted — making its binary-search dispatch index a chunk at a negative row. Both must + /// fail as a [VortexException], never as an `ArithmeticException` or a raw + /// `IndexOutOfBoundsException` (ADR 0003). The scan is O(nchunks), not per row. + /// + /// The final offset is also checked against `rowCount`: without this, an offsets pair like + /// `[0, 1 << 40]` for a 4-row array passes the monotonic check yet hands a terabyte-sized + /// chunk length to `decodeChild`, which allocates it before anything downstream can reject + /// it — an `OutOfMemoryError` rather than a `VortexException` (ADR 0003). + /// + /// @param ctx decode context + /// @param nchunks number of data chunks (children minus the offsets child) + /// @param rowCount logical row count the chunk offsets must sum to + /// @return the `nchunks + 1` monotonic, non-negative offsets spanning exactly `rowCount` + private static long[] readOffsets(DecodeContext ctx, int nchunks, long rowCount) { MemorySegment offsetsBuf = ctx.decodeChildSegment(0, DType.U64, nchunks + 1L); - long cap = SegmentBroadcast.capacity(offsetsBuf, 8); long[] offsets = new long[nchunks + 1]; for (int i = 0; i <= nchunks; i++) { - offsets[i] = offsetsBuf.get(LE_LONG, (i % cap) * 8); + offsets[i] = offsetsBuf.get(LE_LONG, SegmentBroadcast.elementOffset(offsetsBuf, i, 8)); + long previous = i == 0 ? 0 : offsets[i - 1]; + if (offsets[i] < previous) { + throw new VortexException(EncodingId.VORTEX_CHUNKED, + "chunk offsets must be non-negative and non-decreasing, got offsets[" + + i + "]=" + offsets[i] + " after " + previous); + } + } + long span = offsets[nchunks] - offsets[0]; + if (span != rowCount) { + throw new VortexException(EncodingId.VORTEX_CHUNKED, + "chunk offsets span " + span + " rows, expected " + rowCount); } return offsets; } diff --git a/reader/src/main/java/io/github/dfa1/vortex/reader/decode/DecodeContext.java b/reader/src/main/java/io/github/dfa1/vortex/reader/decode/DecodeContext.java index dbcd482ec..3e437f655 100644 --- a/reader/src/main/java/io/github/dfa1/vortex/reader/decode/DecodeContext.java +++ b/reader/src/main/java/io/github/dfa1/vortex/reader/decode/DecodeContext.java @@ -1,6 +1,7 @@ package io.github.dfa1.vortex.reader.decode; import io.github.dfa1.vortex.core.model.DType; +import io.github.dfa1.vortex.core.error.VortexException; import io.github.dfa1.vortex.reader.array.Array; import io.github.dfa1.vortex.reader.ReadRegistry; @@ -35,8 +36,7 @@ public record DecodeContext( /// @param i zero-based child index within this node's children array /// @return the decoded [Array] for child `i` public Array decodeChild(int i) { - ArrayNode child = node.children()[i]; - var childCtx = new DecodeContext(child, dtype, rowCount, segmentBuffers, registry, arena); + var childCtx = new DecodeContext(child(i), dtype, rowCount, segmentBuffers, registry, arena); return registry.decode(childCtx); } @@ -50,8 +50,7 @@ public Array decodeChild(int i) { /// @param rowCount number of logical rows for the child /// @return the decoded [Array] for child `i` public Array decodeChild(int i, DType dtype, long rowCount) { - ArrayNode child = node.children()[i]; - var childCtx = new DecodeContext(child, dtype, rowCount, segmentBuffers, registry, arena); + var childCtx = new DecodeContext(child(i), dtype, rowCount, segmentBuffers, registry, arena); return registry.decode(childCtx); } @@ -60,8 +59,7 @@ public Array decodeChild(int i, DType dtype, long rowCount) { /// @param i zero-based child index within this node's children array /// @return the primary [MemorySegment] of the decoded child public MemorySegment decodeChildSegment(int i) { - ArrayNode child = node.children()[i]; - var childCtx = new DecodeContext(child, dtype, rowCount, segmentBuffers, registry, arena); + var childCtx = new DecodeContext(child(i), dtype, rowCount, segmentBuffers, registry, arena); return registry.decodeAsSegment(childCtx); } @@ -72,11 +70,41 @@ public MemorySegment decodeChildSegment(int i) { /// @param rowCount number of logical rows for the child /// @return the primary [MemorySegment] of the decoded child public MemorySegment decodeChildSegment(int i, DType dtype, long rowCount) { - ArrayNode child = node.children()[i]; - var childCtx = new DecodeContext(child, dtype, rowCount, segmentBuffers, registry, arena); + var childCtx = new DecodeContext(child(i), dtype, rowCount, segmentBuffers, registry, arena); return registry.decodeAsSegment(childCtx); } + /// Returns buffer `bufferPosition` of child `i` without decoding that child. + /// + /// Package-private: it exists only so decoders that read a child's raw segment + /// directly (the legacy dict layout) go through the same bounds-checked accessors as + /// every other read instead of indexing the children and segment arrays by hand. + /// + /// @param i zero-based child index within this node's children array + /// @param bufferPosition zero-based index into the child's `bufferIndices` array + /// @return the child's [MemorySegment] at that position + MemorySegment childBuffer(int i, int bufferPosition) { + var childCtx = new DecodeContext(child(i), dtype, rowCount, segmentBuffers, registry, arena); + return childCtx.buffer(bufferPosition); + } + + /// Returns child `i` of this node, rejecting an index the node does not have. + /// + /// The child vector is shaped by untrusted file data — a node may declare fewer + /// children than its encoding requires — so a missing child must fail as a + /// [VortexException] rather than a raw `ArrayIndexOutOfBoundsException` (ADR 0003). + /// + /// @param i zero-based child index within this node's children array + /// @return the child [ArrayNode] + private ArrayNode child(int i) { + ArrayNode[] children = node.children(); + if (i < 0 || i >= children.length) { + throw new VortexException(node.encodingId(), + "child index " + i + " out of bounds for " + children.length + " child(ren)"); + } + return children[i]; + } + /// Materializes an already-decoded array into a flat primary segment, allocating lazy /// variants from this context's arena. /// @@ -90,12 +118,34 @@ public MemorySegment materialize(Array arr) { return arr.materialize(arena); } + /// Returns the number of segment buffers available to this context. + /// + /// @return the segment buffer count + public int bufferCount() { + return segmentBuffers.length; + } + /// Returns the buffer at position `i` in this node's bufferIndices. /// + /// Both the position and the segment index it holds come from untrusted file data: a + /// node may declare fewer buffers than its encoding reads, or point at a segment that + /// does not exist. Either way the read fails as a [VortexException] rather than a raw + /// `ArrayIndexOutOfBoundsException` (ADR 0003). + /// /// @param i zero-based index into this node's `bufferIndices` array /// @return the [MemorySegment] for the referenced segment buffer public MemorySegment buffer(int i) { - return segmentBuffers[node.bufferIndices()[i]]; + int[] bufferIndices = node.bufferIndices(); + if (i < 0 || i >= bufferIndices.length) { + throw new VortexException(node.encodingId(), + "buffer position " + i + " out of bounds for " + bufferIndices.length + " declared buffer(s)"); + } + int segmentIndex = bufferIndices[i]; + if (segmentIndex < 0 || segmentIndex >= bufferCount()) { + throw new VortexException(node.encodingId(), + "buffer index " + segmentIndex + " out of bounds for " + bufferCount() + " segment(s)"); + } + return segmentBuffers[segmentIndex]; } /// Returns the encoding-specific metadata bytes for this node, or `null` if absent. diff --git a/reader/src/main/java/io/github/dfa1/vortex/reader/decode/DictEncodingDecoder.java b/reader/src/main/java/io/github/dfa1/vortex/reader/decode/DictEncodingDecoder.java index 371067cc8..1209fc201 100644 --- a/reader/src/main/java/io/github/dfa1/vortex/reader/decode/DictEncodingDecoder.java +++ b/reader/src/main/java/io/github/dfa1/vortex/reader/decode/DictEncodingDecoder.java @@ -74,18 +74,13 @@ private static Array decodeLegacyJava(DecodeContext ctx, byte codeTypeByte) { int elemSize = valPType.byteSize(); long rowCount = ctx.rowCount(); - MemorySegment valuesBuf = ctx.segmentBuffers()[ctx.node().children()[0].bufferIndices()[0]]; + MemorySegment valuesBuf = ctx.childBuffer(0, 0); DType codesDtype = new DType.Primitive(codePType, false); MemorySegment codesBuf = ctx.decodeChildSegment(1, codesDtype, rowCount); MemorySegment out = ctx.arena().allocate(rowCount * elemSize); - switch (codePType) { - case U8 -> expandU8(codesBuf, valuesBuf, out, rowCount, elemSize); - case U16 -> expandU16(codesBuf, valuesBuf, out, rowCount, elemSize); - case U32 -> expandU32(codesBuf, valuesBuf, out, rowCount, elemSize); - default -> throw new VortexException(EncodingId.VORTEX_DICT, "unexpected code type: " + codePType); - } + expand(codePType, codesBuf, valuesBuf, out, rowCount, elemSize); return typedArray(ctx.dtype(), valPType, rowCount, out.asReadOnly()); } @@ -127,17 +122,49 @@ private static Array decodeRustProto(DecodeContext ctx, MemorySegment metaBuf) { MemorySegment valuesBuf = ctx.materialize(rawValues); MemorySegment out = ctx.arena().allocate(rowCount * elemSize); - switch (codePType) { - case U8 -> expandU8(codesBuf, valuesBuf, out, rowCount, elemSize); - case U16 -> expandU16(codesBuf, valuesBuf, out, rowCount, elemSize); - case U32 -> expandU32(codesBuf, valuesBuf, out, rowCount, elemSize); - default -> throw new VortexException(EncodingId.VORTEX_DICT, "unexpected code type: " + codePType); - } + expand(codePType, codesBuf, valuesBuf, out, rowCount, elemSize); Array values = typedArray(ctx.dtype(), valPType, rowCount, out.asReadOnly()); BoolArray rowValidity = rowValidity(ctx, codesBuf, codePType, codesValidity, poolValidity, rowCount); return rowValidity == null ? values : new MaskedArray(values, rowValidity); } + /// Expands `codes` against the values pool at the width of `codePType`. + /// + /// The codes buffer is untrusted: an entry pointing past the end of the pool indexes + /// `values` out of bounds. The expand loops must stay uniform to vectorize (CLAUDE.md + /// hot-loop rule), so the guard is a boundary catch-and-wrap around the whole call + /// instead of a per-element range test — the malformed file still fails as a + /// [VortexException] rather than a raw `IndexOutOfBoundsException` (ADR 0003). + /// The broadcast (slow) branches wrap codes with `% valuesCap`, so they cannot overrun + /// — but an empty codes or values child would make that wrap divide by zero, which is + /// rejected up front (O(1), outside the loops). + /// + /// @param codePType unsigned code ptype (U8/U16/U32) + /// @param codes raw codes buffer + /// @param values raw values pool + /// @param out expanded output buffer of `rowCount * elemSize` bytes + /// @param rowCount logical row count + /// @param elemSize value element width in bytes + private static void expand(PType codePType, MemorySegment codes, MemorySegment values, + MemorySegment out, long rowCount, int elemSize) { + if (rowCount > 0 && (codes.byteSize() < codePType.byteSize() || values.byteSize() < elemSize)) { + throw new VortexException(EncodingId.VORTEX_DICT, + "empty dict child for " + rowCount + " rows (codes=" + codes.byteSize() + + " bytes, values=" + values.byteSize() + " bytes)"); + } + try { + switch (codePType) { + case U8 -> expandU8(codes, values, out, rowCount, elemSize); + case U16 -> expandU16(codes, values, out, rowCount, elemSize); + case U32 -> expandU32(codes, values, out, rowCount, elemSize); + default -> throw new VortexException(EncodingId.VORTEX_DICT, "unexpected code type: " + codePType); + } + } catch (IndexOutOfBoundsException e) { + throw new VortexException(EncodingId.VORTEX_DICT, + "code out of range for a values pool of " + values.byteSize() + " bytes", e); + } + } + /// Combines codes-side and pool-side validity into per-row validity: row `i` is /// valid iff its code is valid and the pool slot the code references is valid. /// Returns `null` when neither side carries validity (all rows valid), and the diff --git a/reader/src/main/java/io/github/dfa1/vortex/reader/decode/PrimitiveEncodingDecoder.java b/reader/src/main/java/io/github/dfa1/vortex/reader/decode/PrimitiveEncodingDecoder.java index f5760b7be..482e70f2c 100644 --- a/reader/src/main/java/io/github/dfa1/vortex/reader/decode/PrimitiveEncodingDecoder.java +++ b/reader/src/main/java/io/github/dfa1/vortex/reader/decode/PrimitiveEncodingDecoder.java @@ -30,7 +30,14 @@ public Array decode(DecodeContext ctx) { MemorySegment buf = ctx.buffer(0); long n = ctx.rowCount(); DType dt = ctx.dtype(); - PType ptype = ((DType.Primitive) dt).ptype(); + // decodeChild dispatches on the child node's own encoding id, not on the dtype the + // parent expects — a crafted file can put a `vortex.primitive` node where a non- + // primitive dtype (e.g. Utf8) is requested, which would otherwise leak a raw + // ClassCastException here instead of a VortexException (ADR 0003). + if (!(dt instanceof DType.Primitive primitiveDt)) { + throw new VortexException(EncodingId.VORTEX_PRIMITIVE, "expected primitive dtype, got " + dt); + } + PType ptype = primitiveDt.ptype(); Array values = switch (ptype) { case I64, U64 -> new MaterializedLongArray(dt, n, buf); case I32, U32 -> new MaterializedIntArray(dt, n, buf); diff --git a/reader/src/main/java/io/github/dfa1/vortex/reader/decode/SparseEncodingDecoder.java b/reader/src/main/java/io/github/dfa1/vortex/reader/decode/SparseEncodingDecoder.java index 04fdc76e2..20a90b1a9 100644 --- a/reader/src/main/java/io/github/dfa1/vortex/reader/decode/SparseEncodingDecoder.java +++ b/reader/src/main/java/io/github/dfa1/vortex/reader/decode/SparseEncodingDecoder.java @@ -59,11 +59,27 @@ public Array decode(DecodeContext ctx) { } ProtoPatchesMetadata patches = sparseMeta.patches(); + if (patches == null) { + // proto3 elides an unset message field entirely; a sparse array with no patches + // metadata at all is not a legal encoding (even zero patches must say so + // explicitly), and dereferencing null here would leak a raw NullPointerException. + throw new VortexException(EncodingId.VORTEX_SPARSE, "missing patches metadata"); + } long numPatches = patches.len(); long offset = patches.offset(); PType indicesPtype = PType.fromOrdinal(patches.indices_ptype().value()); long n = ctx.rowCount(); + // Patches sit at distinct positions inside the array, so there can never be more of + // them than rows — the same invariant the Rust reference asserts in `Patches::new` + // (`indices.len() <= array_len`). The count comes from untrusted metadata and drives + // both child decodes and the row-validity bitmap sizing, so an absurd or negative + // value must fail here as a VortexException rather than as an OutOfMemoryError from + // the `allValid` allocation further down (ADR 0003). + if (numPatches < 0 || numPatches > n) { + throw new VortexException(EncodingId.VORTEX_SPARSE, + "patch count " + numPatches + " out of range for " + n + " row(s)"); + } // Row validity mirrors the Rust reference `ValidityVTable`: it is a sparse // bool array whose fill is `fill_value.is_valid()` and whose per-patch value is the @@ -93,8 +109,11 @@ public Array decode(DecodeContext ctx) { valData = m.inner(); patchValidity = m.validity(); } + checkPatchChild(idxData, numPatches, "indices"); + checkPatchChild(valData, numPatches, "values"); boolean fillValue = Boolean.TRUE.equals(fillScalar.bool_value()); - Array result = new LazySparseBoolArray(ctx.dtype(), n, fillValue, (BoolArray) valData, idxData, offset); + BoolArray boolValues = checkedCast(valData, BoolArray.class, "values"); + Array result = new LazySparseBoolArray(ctx.dtype(), n, fillValue, boolValues, idxData, offset); return withSparseValidity(ctx, result, fillValid, patchValidity, idxData, numPatches, n, offset); } @@ -117,24 +136,26 @@ public Array decode(DecodeContext ctx) { valData = m.inner(); patchValidity = m.validity(); } + checkPatchChild(idxData, numPatches, "indices"); + checkPatchChild(valData, numPatches, "values"); Array result = switch (valuePtype) { case I64, U64 -> new LazySparseLongArray(ctx.dtype(), n, fillBits, - (LongArray) valData, idxData, offset); + checkedCast(valData, LongArray.class, "values"), idxData, offset); case I32, U32 -> new LazySparseIntArray(ctx.dtype(), n, (int) fillBits, - (IntArray) valData, idxData, offset); + checkedCast(valData, IntArray.class, "values"), idxData, offset); case F64 -> new LazySparseDoubleArray(ctx.dtype(), n, Double.longBitsToDouble(fillBits), - (DoubleArray) valData, idxData, offset); + checkedCast(valData, DoubleArray.class, "values"), idxData, offset); case F32 -> new LazySparseFloatArray(ctx.dtype(), n, Float.intBitsToFloat((int) fillBits), - (FloatArray) valData, idxData, offset); + checkedCast(valData, FloatArray.class, "values"), idxData, offset); case I16 -> new LazySparseShortArray(ctx.dtype(), n, (short) fillBits, (short) fillBits, - (ShortArray) valData, idxData, offset); + checkedCast(valData, ShortArray.class, "values"), idxData, offset); case U16 -> new LazySparseShortArray(ctx.dtype(), n, (short) fillBits, (int) (fillBits & 0xFFFFL), - (ShortArray) valData, idxData, offset); + checkedCast(valData, ShortArray.class, "values"), idxData, offset); case I8 -> new LazySparseByteArray(ctx.dtype(), n, (byte) fillBits, (byte) fillBits, - (ByteArray) valData, idxData, offset); + checkedCast(valData, ByteArray.class, "values"), idxData, offset); case U8 -> new LazySparseByteArray(ctx.dtype(), n, (byte) fillBits, (int) (fillBits & 0xFFL), - (ByteArray) valData, idxData, offset); + checkedCast(valData, ByteArray.class, "values"), idxData, offset); default -> throw new VortexException(EncodingId.VORTEX_SPARSE, "unsupported ptype " + valuePtype); }; return withSparseValidity(ctx, result, fillValid, patchValidity, idxData, numPatches, n, offset); @@ -173,6 +194,46 @@ private static BoolArray allValid(DecodeContext ctx, long len) { return new MaterializedBoolArray(DType.BOOL, len, bits.asReadOnly()); } + /// Rejects a patch child whose physical buffer holds no elements at all while the + /// metadata claims patches. + /// + /// The `Materialized*` accessors deliberately broadcast an undersized buffer with + /// `i % elementCount` (the `ConstantEncoding` fan-out), so a zero-element buffer would + /// divide by zero — an `ArithmeticException`, not a [VortexException] (ADR 0003). The + /// probe is O(1) and non-allocating: lazy children report no segment and are skipped, + /// exactly like the equivalent guard in [DictEncodingDecoder]. + /// + /// @param child decoded patch child (indices or values) + /// @param numPatches number of patches the metadata declares + /// @param role `"indices"` or `"values"`, for the error message + private static void checkPatchChild(Array child, long numPatches, String role) { + if (numPatches > 0 && child.segmentIfPresent().filter(s -> s.byteSize() == 0).isPresent()) { + throw new VortexException(EncodingId.VORTEX_SPARSE, + "empty patch " + role + " child for " + numPatches + " patch(es)"); + } + } + + /// Casts a decoded patch child to the type its declared ptype/dtype demands, rejecting a + /// mismatch as a [VortexException]. + /// + /// `decodeChild` dispatches on the *child node's own* encoding id, not on the dtype this + /// decoder asked for — a crafted file can put e.g. a `vortex.bool` node where an `i64` + /// sparse array expects its values child, which decodes without error and would otherwise + /// blow up as a raw `ClassCastException` at the unchecked cast site (ADR 0003). + /// + /// @param child decoded patch child (indices or values) + /// @param type the concrete [Array] subtype required at this call site + /// @param role `"indices"` or `"values"`, for the error message + /// @param the required array type + /// @return `child`, cast to `type` + private static T checkedCast(Array child, Class type, String role) { + if (!type.isInstance(child)) { + throw new VortexException(EncodingId.VORTEX_SPARSE, + "patch " + role + " child decoded to unexpected type: " + child.getClass().getSimpleName()); + } + return type.cast(child); + } + private static ProtoScalarValue decodeFill(MemorySegment fillBuf) { try { return ProtoScalarValue.decode(fillBuf, 0, fillBuf.byteSize()); @@ -203,6 +264,7 @@ private static Array decodeVarBin( DType indicesDtype = new DType.Primitive(indicesPtype, false); Array patchIndices = ctx.decodeChild(0, indicesDtype, numPatches); Array idxData = patchIndices instanceof MaskedArray m ? m.inner() : patchIndices; + checkPatchChild(idxData, numPatches, "indices"); MemorySegment outOffsets = ctx.arena().allocate((n + 1) * 4L, 4); if (numPatches == 0) { @@ -220,37 +282,59 @@ private static Array decodeVarBin( valData = m.inner(); patchValidity = m.validity(); } - VarBinArray.OffsetMode varBin = VarBinArray.toOffsetMode((VarBinArray) valData, ctx.arena()); + VarBinArray.OffsetMode varBin = VarBinArray.toOffsetMode( + checkedCast(valData, VarBinArray.class, "values"), ctx.arena()); MemorySegment valBytes = varBin.bytesSegment(); MemorySegment valOffsets = varBin.offsetsSegment(); PType valOffPtype = varBin.offsetsPtype(); MemorySegment idxSeg = ctx.materialize(idxData); int idxBytes = indicesPtype.byteSize(); - long totalBytes = 0; - for (long i = 0; i < numPatches; i++) { - totalBytes += readVarBinOffset(valOffsets, i + 1, valOffPtype) - - readVarBinOffset(valOffsets, i, valOffPtype); - } - - MemorySegment outBytes = ctx.arena().allocate(Math.max(1, totalBytes)); - long patchCursor = 0; - long bytePos = 0; - for (long pos = 0; pos < n; pos++) { - if (patchCursor < numPatches) { - long patchPos = readUnsignedIdx(idxSeg, SegmentBroadcast.elementOffset(idxSeg, patchCursor, idxBytes), indicesPtype) - offset; - if (patchPos == pos) { - long strStart = readVarBinOffset(valOffsets, patchCursor, valOffPtype); - long strEnd = readVarBinOffset(valOffsets, patchCursor + 1, valOffPtype); - long strLen = strEnd - strStart; - if (strLen > 0) { - MemorySegment.copy(valBytes, strStart, outBytes, bytePos, strLen); - bytePos += strLen; + MemorySegment outBytes; + // The patch-value offsets and the declared patch count come from untrusted metadata and + // are deliberately not cross-validated up front (VarBin decode stays lazy). Both loops + // below index `valOffsets` at `numPatches + 1` and copy `valBytes` sub-ranges, so an + // over-long patch count or a non-monotonic offsets pair walks off a segment. The guard + // is a boundary catch-and-wrap around the whole merge rather than a per-element range + // test, so the copy loop stays uniform (CLAUDE.md hot-loop rule) — the malformed file + // still fails as a VortexException, never a raw IndexOutOfBoundsException (ADR 0003). + try { + long totalBytes = 0; + for (long i = 0; i < numPatches; i++) { + totalBytes += readVarBinOffset(valOffsets, i + 1, valOffPtype) + - readVarBinOffset(valOffsets, i, valOffPtype); + } + // Patched bytes are a subset of the value buffer, so a total outside it means the + // offsets disagree with the payload; catching it here also keeps the allocation + // below bounded by data that actually exists (no OutOfMemoryError zip bomb). + if (totalBytes < 0 || totalBytes > valBytes.byteSize()) { + throw new VortexException(EncodingId.VORTEX_SPARSE, + "patch bytes " + totalBytes + " out of range for a value buffer of " + + valBytes.byteSize() + " byte(s)"); + } + outBytes = ctx.arena().allocate(Math.max(1, totalBytes)); + long patchCursor = 0; + long bytePos = 0; + for (long pos = 0; pos < n; pos++) { + if (patchCursor < numPatches) { + long patchPos = readUnsignedIdx(idxSeg, SegmentBroadcast.elementOffset(idxSeg, patchCursor, idxBytes), indicesPtype) - offset; + if (patchPos == pos) { + long strStart = readVarBinOffset(valOffsets, patchCursor, valOffPtype); + long strEnd = readVarBinOffset(valOffsets, patchCursor + 1, valOffPtype); + long strLen = strEnd - strStart; + if (strLen > 0) { + MemorySegment.copy(valBytes, strStart, outBytes, bytePos, strLen); + bytePos += strLen; + } + patchCursor++; } - patchCursor++; } + outOffsets.setAtIndex(VortexFormat.LE_INT, pos + 1, (int) bytePos); } - outOffsets.setAtIndex(VortexFormat.LE_INT, pos + 1, (int) bytePos); + } catch (IndexOutOfBoundsException e) { + throw new VortexException(EncodingId.VORTEX_SPARSE, + "patch value offsets out of range for " + numPatches + " patch(es) over a " + + valOffsets.byteSize() + "-byte offsets buffer", e); } Array result = new VarBinArray.OffsetMode(ctx.dtype(), n, outBytes, outOffsets, PType.I32); diff --git a/reader/src/test/java/io/github/dfa1/vortex/reader/array/LazySparseArrayTest.java b/reader/src/test/java/io/github/dfa1/vortex/reader/array/LazySparseArrayTest.java index 0e36eea57..21eb5df41 100644 --- a/reader/src/test/java/io/github/dfa1/vortex/reader/array/LazySparseArrayTest.java +++ b/reader/src/test/java/io/github/dfa1/vortex/reader/array/LazySparseArrayTest.java @@ -1,5 +1,6 @@ package io.github.dfa1.vortex.reader.array; +import io.github.dfa1.vortex.core.error.VortexException; import org.junit.jupiter.api.Nested; import org.junit.jupiter.api.Test; @@ -25,6 +26,7 @@ import static io.github.dfa1.vortex.reader.array.TestArrays.longs; import static io.github.dfa1.vortex.reader.array.TestArrays.shorts; import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; /// Unit tests for the lazy Sparse records. Covers fill vs patch dispatch, ordered /// forEach iteration, fold reduction, and offset slicing semantics. @@ -318,4 +320,41 @@ void nullPatchesIsAllFill() { assertThat(sut.fold(0.0, java.lang.Double::sum)).isEqualTo(126.0); } } + + /// Malformed-input cases (TODO.md §Security, ADR 0003): `patchIndices` is untrusted file + /// data the format requires to be sorted ascending. `walkPatches` (the shared `forEach` + /// engine) advances its cursor to `patchAbs + 1` per patch and assumes it never sees a + /// smaller value again — before the guard landed, an out-of-order index moved the cursor + /// backwards and the walk re-covered already-emitted positions, emitting more callbacks + /// than the array's own `length()`. A caller sizing a buffer from `length()` (as every real + /// `forEach*` caller does) then overran it with a raw `IndexOutOfBoundsException`. + @Nested + class AdversarialInput { + + @Test + void forEachLong_unsortedPatchIndices_throws() { + // Given — length=10, patch indices out of order (5 before 1) + LongArray values = longs(50L, 10L); + Array indices = ints(5, 1); + var sut = new LazySparseLongArray(I64, 10, 0L, values, indices, 0L); + + // When / Then + assertThatThrownBy(() -> sut.forEachLong(v -> { })) + .isInstanceOf(VortexException.class) + .hasMessageContaining("not sorted"); + } + + @Test + void fold_unsortedPatchIndices_throws() { + // Given — same out-of-order indices, reached through the fold walker instead + LongArray values = longs(50L, 10L); + Array indices = ints(5, 1); + var sut = new LazySparseLongArray(I64, 10, 0L, values, indices, 0L); + + // When / Then + assertThatThrownBy(() -> sut.fold(0L, java.lang.Long::sum)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("not sorted"); + } + } } diff --git a/reader/src/test/java/io/github/dfa1/vortex/reader/decode/AlpEncodingDecoderTest.java b/reader/src/test/java/io/github/dfa1/vortex/reader/decode/AlpEncodingDecoderTest.java index 9237af354..3eedb1ee2 100644 --- a/reader/src/test/java/io/github/dfa1/vortex/reader/decode/AlpEncodingDecoderTest.java +++ b/reader/src/test/java/io/github/dfa1/vortex/reader/decode/AlpEncodingDecoderTest.java @@ -1,8 +1,10 @@ package io.github.dfa1.vortex.reader.decode; +import io.github.dfa1.vortex.core.error.VortexException; import io.github.dfa1.vortex.core.model.DType; import io.github.dfa1.vortex.core.model.EncodingId; import io.github.dfa1.vortex.core.proto.ProtoALPMetadata; +import io.github.dfa1.vortex.core.proto.ProtoPType; import io.github.dfa1.vortex.core.proto.ProtoPatchesMetadata; import io.github.dfa1.vortex.reader.ReadRegistry; import io.github.dfa1.vortex.reader.array.Array; @@ -125,7 +127,7 @@ void decode_f32_broadcastNoPatches_returnsConstant() { void decode_f64_patches_withU8Indices() { // Given patches whose index child uses U8 storage — exercises the U8 arm of // readUnsigned (the encoder always emits U32 indices) - ProtoPatchesMetadata pm = new ProtoPatchesMetadata(1L, 0L, io.github.dfa1.vortex.core.proto.ProtoPType.U8, null, null, null); + ProtoPatchesMetadata pm = new ProtoPatchesMetadata(1L, 0L, ProtoPType.U8, null, null, null); byte[] meta = new ProtoALPMetadata(2, 0, pm).encode(); // *0.01 ArrayNode enc = new ArrayNode(EncodingId.VORTEX_PRIMITIVE, null, new ArrayNode[0], new int[]{0}); @@ -150,7 +152,7 @@ void decode_f64_patches_withU8Indices() { @Test void decode_patches_nonUnsignedIndexPtype_throws() { // Given a signed (I32) patch-index ptype — readUnsigned rejects it - ProtoPatchesMetadata pm = new ProtoPatchesMetadata(1L, 0L, io.github.dfa1.vortex.core.proto.ProtoPType.I32, null, null, null); + ProtoPatchesMetadata pm = new ProtoPatchesMetadata(1L, 0L, ProtoPType.I32, null, null, null); byte[] meta = new ProtoALPMetadata(2, 0, pm).encode(); ArrayNode enc = new ArrayNode(EncodingId.VORTEX_PRIMITIVE, null, new ArrayNode[0], new int[]{0}); @@ -166,6 +168,134 @@ void decode_patches_nonUnsignedIndexPtype_throws() { assertThatThrownBy(() -> SUT.decode(ctx)).hasMessageContaining("non-unsigned patch index ptype"); } + /// Adversarial metadata from an untrusted file (TODO.md §Security, per-encoding + /// adversarial tests). `exp_e`/`exp_f` were read verbatim and used to index the + /// power-of-ten tables, so an out-of-range exponent escaped as an + /// `ArrayIndexOutOfBoundsException`; patch indices were scatter-written with no range + /// check at all (unlike [BitpackedEncodingDecoder]), so a patch pointing outside the + /// row range escaped as a raw `IndexOutOfBoundsException` from the copy, and an empty + /// patch child divided by zero in the broadcast branch. + /// + /// The float-vs-double discriminator is not part of ALP metadata here — it is derived + /// from the column's dtype — and an unsupported ptype already fails as a + /// [VortexException] ("unsupported dtype"), so there is + /// no out-of-enum-range byte to fuzz. + @Nested + class AdversarialMetadata { + + @Test + void negativeExponent_f64_throws() { + // Given exp_e = -1, which would index the inverse power-of-ten table below zero + DecodeContext ctx = ctx(F64, new ProtoALPMetadata(-1, 0, null), leLongs(1L, 2L), 2); + + // When / Then + assertThatThrownBy(() -> SUT.decode(ctx)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("exponents (e=-1, f=0) out of range [0,24)"); + } + + @Test + void exponentPastTable_f64_throws() { + // Given exp_e = 24, one past the last entry of the 24-wide f64 table + DecodeContext ctx = ctx(F64, new ProtoALPMetadata(24, 0, null), leLongs(1L, 2L), 2); + + // When / Then + assertThatThrownBy(() -> SUT.decode(ctx)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("out of range [0,24)"); + } + + @Test + void exponentPastTable_f32_throws() { + // Given exp_f = 11 on an f32 column: valid for f64 but one past the narrower + // 11-wide f32 table, so the width-specific bound is what matters + DecodeContext ctx = ctx(F32, new ProtoALPMetadata(0, 11, null), leInts(1, 2), 2); + + // When / Then + assertThatThrownBy(() -> SUT.decode(ctx)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("out of range [0,11)"); + } + + @Test + void patchIndexPastRowCount_throws() { + // Given a single patch pointing at row 9 of a 2-row array + ProtoPatchesMetadata pm = new ProtoPatchesMetadata(1L, 0L, + ProtoPType.U8, null, null, null); + DecodeContext ctx = patchedCtx(pm, leLongs(100L, 200L), + MemorySegment.ofArray(new byte[]{9}), leDoubles(9.0), 2); + + // When / Then + assertThatThrownBy(() -> SUT.decode(ctx)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("patch index 9 out of range [0,2)"); + } + + @Test + void patchOffsetPushesIndexNegative_throws() { + // Given patches.offset = 5 against a patch index of 1, so the absolute row index + // is -4 and the copy would write before the start of the output buffer + ProtoPatchesMetadata pm = new ProtoPatchesMetadata(1L, 5L, + ProtoPType.U8, null, null, null); + DecodeContext ctx = patchedCtx(pm, leLongs(100L, 200L), + MemorySegment.ofArray(new byte[]{1}), leDoubles(9.0), 2); + + // When / Then + assertThatThrownBy(() -> SUT.decode(ctx)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("patch index -4 out of range [0,2)"); + } + + @Test + void patchCountGreaterThanRowCount_throws() { + // Given 3 declared patches over a 2-row array: the third index necessarily falls + // outside the rows, which is exactly what the range guard is for + ProtoPatchesMetadata pm = new ProtoPatchesMetadata(3L, 0L, + ProtoPType.U8, null, null, null); + DecodeContext ctx = patchedCtx(pm, leLongs(100L, 200L), + MemorySegment.ofArray(new byte[]{0, 1, 2}), leDoubles(9.0, 9.0, 9.0), 2); + + // When / Then + assertThatThrownBy(() -> SUT.decode(ctx)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("patch index 2 out of range [0,2)"); + } + + @Test + void emptyPatchIndexChild_throws() { + // Given patches declared but a zero-length index child: the broadcast branch used + // to compute `i % 0` and die with an ArithmeticException + ProtoPatchesMetadata pm = new ProtoPatchesMetadata(1L, 0L, + ProtoPType.U8, null, null, null); + DecodeContext ctx = patchedCtx(pm, leLongs(100L, 200L), + MemorySegment.ofArray(new byte[0]), leDoubles(9.0), 2); + + // When / Then + assertThatThrownBy(() -> SUT.decode(ctx)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("empty patch child"); + } + + private DecodeContext ctx(DType dtype, ProtoALPMetadata meta, MemorySegment encoded, long rowCount) { + ArrayNode enc = new ArrayNode(EncodingId.VORTEX_PRIMITIVE, null, new ArrayNode[0], new int[]{0}); + ArrayNode node = new ArrayNode(EncodingId.VORTEX_ALP, MemorySegment.ofArray(meta.encode()), + new ArrayNode[]{enc}, new int[0]); + return new DecodeContext(node, dtype, rowCount, new MemorySegment[]{encoded}, REGISTRY, Arena.ofAuto()); + } + + private DecodeContext patchedCtx(ProtoPatchesMetadata pm, MemorySegment encoded, + MemorySegment indices, MemorySegment values, long rowCount) { + byte[] meta = new ProtoALPMetadata(2, 0, pm).encode(); + ArrayNode enc = new ArrayNode(EncodingId.VORTEX_PRIMITIVE, null, new ArrayNode[0], new int[]{0}); + ArrayNode idx = new ArrayNode(EncodingId.VORTEX_PRIMITIVE, null, new ArrayNode[0], new int[]{1}); + ArrayNode val = new ArrayNode(EncodingId.VORTEX_PRIMITIVE, null, new ArrayNode[0], new int[]{2}); + ArrayNode node = new ArrayNode(EncodingId.VORTEX_ALP, MemorySegment.ofArray(meta), + new ArrayNode[]{enc, idx, val}, new int[0]); + return new DecodeContext(node, F64, rowCount, + new MemorySegment[]{encoded, indices, values}, REGISTRY, Arena.ofAuto()); + } + } + /// An ALP array's validity IS its encoded child's (`ValidityChild`, #210): a nullable /// encoded primitive child surfaces as a [MaskedArray], and that mask must ride through both the /// per-row lazy path and the single-value constant-broadcast path rather than being flattened. diff --git a/reader/src/test/java/io/github/dfa1/vortex/reader/decode/BitpackedEncodingDecoderTest.java b/reader/src/test/java/io/github/dfa1/vortex/reader/decode/BitpackedEncodingDecoderTest.java index 0356df133..b32b96a16 100644 --- a/reader/src/test/java/io/github/dfa1/vortex/reader/decode/BitpackedEncodingDecoderTest.java +++ b/reader/src/test/java/io/github/dfa1/vortex/reader/decode/BitpackedEncodingDecoderTest.java @@ -180,6 +180,96 @@ void unexpectedChildCount_throwsWithExpectedCounts() { } } + /// Adversarial metadata and buffers from an untrusted file (TODO.md §Security, + /// per-encoding adversarial tests). `bit_width` was read straight off the wire and fed + /// into the shift and byte-offset math of the unpack loops, so a negative or oversized + /// width — or a `packed` buffer too short for the declared row count — escaped as a raw + /// `IndexOutOfBoundsException` from the segment reads instead of a [VortexException]. + @Nested + class AdversarialMetadata { + + @Test + void negativeBitWidth_throws() { + // Given bit_width = -1, which makes every shift and mask nonsensical + DecodeContext ctx = ctxWithMeta(new ProtoBitPackedMetadata(-1, 0, null), packedBytes(64), 8); + + // When / Then + assertThatThrownBy(() -> SUT.decode(ctx)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("bit width -1 out of range [0,32] for I32"); + } + + @Test + void bitWidthAboveElementWidth_throws() { + // Given bit_width = 40 on an I32 column: below the 64-bit ceiling but wider than + // the column's own elements, so the bound has to be the element width + DecodeContext ctx = ctxWithMeta(new ProtoBitPackedMetadata(40, 0, null), packedBytes(64), 8); + + // When / Then + assertThatThrownBy(() -> SUT.decode(ctx)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("bit width 40 out of range [0,32] for I32"); + } + + @Test + void bitWidthAboveSixtyFour_throws() { + // Given bit_width = 65, past even the widest supported element + DecodeContext ctx = ctxWithMeta(new ProtoBitPackedMetadata(65, 0, null), packedBytes(64), 8); + + // When / Then + assertThatThrownBy(() -> SUT.decode(ctx)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("bit width 65 out of range"); + } + + @Test + void negativeOffset_throws() { + // Given a negative in-block offset (a u32 large enough to wrap when read as int) + DecodeContext ctx = ctxWithMeta(new ProtoBitPackedMetadata(8, -1, null), packedBytes(64), 8); + + // When / Then + assertThatThrownBy(() -> SUT.decode(ctx)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("offset -1 out of range [0,1024)"); + } + + @Test + void offsetPastBlockSize_throws() { + // Given an in-block offset far past the 1024-element FastLanes block: it is not + // just wrong, it inflates the block count so the decoder would grind through + // billions of skipped lanes from a single metadata varint before failing + DecodeContext ctx = ctxWithMeta(new ProtoBitPackedMetadata(8, Integer.MAX_VALUE, null), + packedBytes(64), 4); + + // When / Then + assertThatThrownBy(() -> SUT.decode(ctx)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("out of range [0,1024)"); + } + + @Test + void packedBufferTooSmallForRowCount_throws() { + // Given 1024 rows at bit_width 8 (one full 1024-element block = 1024 packed + // bytes for I32) but only 4 bytes of packed data on the wire + DecodeContext ctx = ctxWithMeta(new ProtoBitPackedMetadata(8, 0, null), packedBytes(4), 1024); + + // When / Then + assertThatThrownBy(() -> SUT.decode(ctx)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("too small for 1024 rows at bit width 8"); + } + + private DecodeContext ctxWithMeta(ProtoBitPackedMetadata meta, MemorySegment packed, long rowCount) { + ArrayNode node = new ArrayNode(BITPACKED, MemorySegment.ofArray(meta.encode()), + new ArrayNode[0], new int[]{0}); + return new DecodeContext(node, I32, rowCount, new MemorySegment[]{packed}, REGISTRY, Arena.ofAuto()); + } + + private MemorySegment packedBytes(int n) { + return MemorySegment.ofArray(new byte[n]); + } + } + // ── helpers ───────────────────────────────────────────────────────────────── private static MaskedArray assertMasked(Array result) { diff --git a/reader/src/test/java/io/github/dfa1/vortex/reader/decode/ChunkedEncodingDecoderTest.java b/reader/src/test/java/io/github/dfa1/vortex/reader/decode/ChunkedEncodingDecoderTest.java new file mode 100644 index 000000000..4e84c549a --- /dev/null +++ b/reader/src/test/java/io/github/dfa1/vortex/reader/decode/ChunkedEncodingDecoderTest.java @@ -0,0 +1,148 @@ +package io.github.dfa1.vortex.reader.decode; + +import io.github.dfa1.vortex.core.error.VortexException; +import io.github.dfa1.vortex.core.model.DType; +import io.github.dfa1.vortex.core.model.EncodingId; +import io.github.dfa1.vortex.core.testing.TestSegments; +import io.github.dfa1.vortex.reader.ReadRegistry; +import io.github.dfa1.vortex.reader.array.Array; +import io.github.dfa1.vortex.reader.array.LongArray; +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; + +import java.lang.foreign.Arena; +import java.lang.foreign.MemorySegment; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +class ChunkedEncodingDecoderTest { + + private static final ChunkedEncodingDecoder SUT = new ChunkedEncodingDecoder(); + private static final ReadRegistry REGISTRY = TestRegistry.ofDecoders( + SUT, new PrimitiveEncodingDecoder(), new BoolEncodingDecoder()); + + @Test + void encodingId_isVortexChunked() { + // Given / When / Then + assertThat(SUT.encodingId()).isEqualTo(EncodingId.VORTEX_CHUNKED); + } + + @Test + void decode_twoChunks_viewsBothWithoutConcatenating() { + // Given — chunk offsets 0/2/5 over a 2-row and a 3-row i64 chunk + MemorySegment[] segs = { + TestSegments.leLongs(0, 2, 5), + TestSegments.leLongs(10, 11), + TestSegments.leLongs(12, 13, 14) + }; + + // When + Array result = decode(DType.I64, 5, segs, primitiveNode(0), primitiveNode(1), primitiveNode(2)); + + // Then — a single logical i64 view spanning both chunks + assertThat(result).isInstanceOf(LongArray.class); + LongArray longs = (LongArray) result; + assertThat(longs.length()).isEqualTo(5); + assertThat(longs.getLong(0)).isEqualTo(10); + assertThat(longs.getLong(2)).isEqualTo(12); + assertThat(longs.getLong(4)).isEqualTo(14); + } + + /// Malformed-input cases for `vortex.chunked` (TODO.md §Security, ADR 0003). The child count + /// and the chunk-offsets buffer both come from untrusted file bytes; the class named in each + /// comment is the raw JDK exception the reader leaked before the guards landed. + /// + /// The TODO's other Chunked item — a maliciously deep child tree — is guarded and tested one + /// layer up, at the point the `ArrayNode` tree is built from the file's FlatBuffer: see + /// [io.github.dfa1.vortex.reader.SerializedArrayDecoder] `MAX_ARRAY_TREE_DEPTH` and + /// `ArrayNodeDepthBombSecurityTest`. FlatBuffers cannot encode a true cycle, so the depth cap + /// is the whole defense and nothing here can reproduce it from a hand-built node. + @Nested + class AdversarialInput { + + @Test + void noChildren_throws() { + // Given — a chunked node with no children at all, not even the offsets child + ArrayNode node = new ArrayNode(EncodingId.VORTEX_CHUNKED, null, new ArrayNode[0], new int[0]); + DecodeContext ctx = new DecodeContext(node, DType.I64, 3, new MemorySegment[0], + REGISTRY, Arena.ofAuto()); + + // When / Then + assertThatThrownBy(() -> SUT.decode(ctx)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("at least one child"); + } + + /// The TODO's "zero children with non-zero row_count": a node carrying only the offsets + /// child describes no data, yet claims rows. The primitive families surfaced this as the + /// `Chunked*Array` "empty chunk list" error and a struct dtype swallowed it silently, so + /// the decoder now names the real defect uniformly. + @Test + void zeroChunksWithNonZeroRowCount_throws() { + // Given — only the offsets child, but 10 rows claimed + MemorySegment[] segs = {TestSegments.leLongs(0)}; + + // When / Then + assertThatThrownBy(() -> decode(DType.I64, 10, segs, primitiveNode(0))) + .isInstanceOf(VortexException.class) + .hasMessageContaining("no chunks for 10 row(s)"); + } + + /// `ArithmeticException: / by zero` — the offsets read broadcasts an undersized buffer + /// with `i % capacity`, and an empty offsets child makes that capacity zero. + @Test + void emptyOffsetsChild_throws() { + // Given — a zero-byte offsets segment in front of one real chunk + MemorySegment[] segs = {MemorySegment.ofArray(new byte[0]), TestSegments.leLongs(1, 2, 3)}; + + // When / Then + assertThatThrownBy(() -> decode(DType.I64, 3, segs, primitiveNode(0), primitiveNode(1))) + .isInstanceOf(VortexException.class) + .hasMessageContaining("decoded child segment is empty"); + } + + /// `IndexOutOfBoundsException` — a decreasing offsets pair yields a negative chunk length. + /// The chunk lengths still summed to the declared row count here, so the `Chunked*Array` + /// row-count check passed and left an unsorted `offsets` array behind, whose binary-search + /// dispatch then indexed a chunk at a negative row. + @Test + void nonMonotonicOffsets_throws() { + // Given — offsets 0/5/3: chunk 0 spans 5 rows, chunk 1 spans -2, summing to the + // declared 3 rows so nothing downstream catches the inversion. + MemorySegment[] segs = { + TestSegments.leLongs(0, 5, 3), + TestSegments.leLongs(1, 2, 3, 4, 5), + TestSegments.leLongs(9, 9) + }; + + // When / Then + assertThatThrownBy(() -> decode(DType.I64, 3, segs, + primitiveNode(0), primitiveNode(1), primitiveNode(2))) + .isInstanceOf(VortexException.class) + .hasMessageContaining("non-decreasing"); + } + + /// The same guard's lower bound: offsets are cumulative row counts, so the first one is + /// never negative. A negative start would make every chunk length nonsense. + @Test + void negativeFirstOffset_throws() { + // Given — a first offset of -1 + MemorySegment[] segs = {TestSegments.leLongs(-1, 2), TestSegments.leLongs(1, 2, 3)}; + + // When / Then + assertThatThrownBy(() -> decode(DType.I64, 3, segs, primitiveNode(0), primitiveNode(1))) + .isInstanceOf(VortexException.class) + .hasMessageContaining("non-decreasing"); + } + } + + private static Array decode(DType dtype, long rowCount, MemorySegment[] segs, ArrayNode... children) { + ArrayNode node = new ArrayNode(EncodingId.VORTEX_CHUNKED, null, children, new int[0]); + return SUT.decode(new DecodeContext(node, dtype, rowCount, segs, REGISTRY, Arena.ofAuto())); + } + + private static ArrayNode primitiveNode(int bufferIndex) { + return new ArrayNode(EncodingId.VORTEX_PRIMITIVE, null, new ArrayNode[0], new int[]{bufferIndex}); + } +} diff --git a/reader/src/test/java/io/github/dfa1/vortex/reader/decode/DecodeContextTest.java b/reader/src/test/java/io/github/dfa1/vortex/reader/decode/DecodeContextTest.java new file mode 100644 index 000000000..7bc3771d7 --- /dev/null +++ b/reader/src/test/java/io/github/dfa1/vortex/reader/decode/DecodeContextTest.java @@ -0,0 +1,147 @@ +package io.github.dfa1.vortex.reader.decode; + +import io.github.dfa1.vortex.core.error.VortexException; +import io.github.dfa1.vortex.core.model.DType; +import io.github.dfa1.vortex.core.model.EncodingId; +import org.junit.jupiter.api.Test; + +import java.lang.foreign.Arena; +import java.lang.foreign.MemorySegment; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +class DecodeContextTest { + + @Test + void buffer_indexPastSegmentCount_throwsVortexException() { + // Given a node claiming buffer index 1 while only 1 segment (index 0) exists + ArrayNode node = new ArrayNode(EncodingId.VORTEX_PRIMITIVE, null, new ArrayNode[0], new int[]{1}); + DecodeContext sut = TestDecodeContexts.of(node, DType.UTF8) + .segments(MemorySegment.ofArray(new byte[4])) + .arena(Arena.ofAuto()) + .build(); + + // When / Then + assertThatThrownBy(() -> sut.buffer(0)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("out of bounds"); + } + + @Test + void buffer_negativeIndex_throwsVortexException() { + // Given a node with a negative buffer index (e.g. corrupted wire data) + ArrayNode node = new ArrayNode(EncodingId.VORTEX_PRIMITIVE, null, new ArrayNode[0], new int[]{-1}); + DecodeContext sut = TestDecodeContexts.of(node, DType.UTF8) + .segments(MemorySegment.ofArray(new byte[4])) + .arena(Arena.ofAuto()) + .build(); + + // When / Then + assertThatThrownBy(() -> sut.buffer(0)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("out of bounds"); + } + + @Test + void buffer_positionPastDeclaredBuffers_throwsVortexException() { + // Given a node that declares no buffers at all, as a truncated legacy dict layout + // would: the position itself is out of bounds before its value can be checked + ArrayNode node = new ArrayNode(EncodingId.VORTEX_DICT, null, new ArrayNode[0], new int[0]); + DecodeContext sut = TestDecodeContexts.of(node, DType.UTF8) + .segments(MemorySegment.ofArray(new byte[4])) + .arena(Arena.ofAuto()) + .build(); + + // When / Then + assertThatThrownBy(() -> sut.buffer(0)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("buffer position 0 out of bounds for 0 declared buffer(s)"); + } + + @Test + void buffer_negativePosition_throwsVortexException() { + // Given a valid node but a negative position + ArrayNode node = new ArrayNode(EncodingId.VORTEX_PRIMITIVE, null, new ArrayNode[0], new int[]{0}); + DecodeContext sut = TestDecodeContexts.of(node, DType.UTF8) + .segments(MemorySegment.ofArray(new byte[4])) + .arena(Arena.ofAuto()) + .build(); + + // When / Then + assertThatThrownBy(() -> sut.buffer(-1)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("buffer position -1 out of bounds"); + } + + @Test + void decodeChild_indexPastChildCount_throwsVortexException() { + // Given a node with no children, as an ALP node stripped of its encoded child would + // be: the raw children array access must not escape as an AIOOBE + ArrayNode node = new ArrayNode(EncodingId.VORTEX_ALP, null, new ArrayNode[0], new int[0]); + DecodeContext sut = TestDecodeContexts.of(node, DType.F64).arena(Arena.ofAuto()).build(); + + // When / Then + assertThatThrownBy(() -> sut.decodeChild(0)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("child index 0 out of bounds for 0 child(ren)"); + } + + @Test + void decodeChildSegment_indexPastChildCount_throwsVortexException() { + // Given the same childless node reached through the segment overload + ArrayNode node = new ArrayNode(EncodingId.VORTEX_ALP, null, new ArrayNode[0], new int[0]); + DecodeContext sut = TestDecodeContexts.of(node, DType.F64).arena(Arena.ofAuto()).build(); + + // When / Then + assertThatThrownBy(() -> sut.decodeChildSegment(1, DType.F64, 1)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("child index 1 out of bounds"); + } + + @Test + void decodeChild_negativeIndex_throwsVortexException() { + // Given a node with one child but a negative index + ArrayNode child = new ArrayNode(EncodingId.VORTEX_PRIMITIVE, null, new ArrayNode[0], new int[]{0}); + ArrayNode node = new ArrayNode(EncodingId.VORTEX_ALP, null, new ArrayNode[]{child}, new int[0]); + DecodeContext sut = TestDecodeContexts.of(node, DType.F64).arena(Arena.ofAuto()).build(); + + // When / Then + assertThatThrownBy(() -> sut.decodeChild(-1, DType.F64, 1)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("child index -1 out of bounds"); + } + + @Test + void buffer_validIndex_returnsSegment() { + // Given a node whose buffer index correctly references the single available segment + MemorySegment segment = MemorySegment.ofArray(new byte[4]); + ArrayNode node = new ArrayNode(EncodingId.VORTEX_PRIMITIVE, null, new ArrayNode[0], new int[]{0}); + DecodeContext sut = TestDecodeContexts.of(node, DType.UTF8) + .segments(segment) + .arena(Arena.ofAuto()) + .build(); + + // When + MemorySegment result = sut.buffer(0); + + // Then + assertThat(result).isSameAs(segment); + } + + @Test + void bufferCount_returnsSegmentBufferLength() { + // Given two segment buffers + ArrayNode node = new ArrayNode(EncodingId.VORTEX_PRIMITIVE, null, new ArrayNode[0], new int[]{0}); + DecodeContext sut = TestDecodeContexts.of(node, DType.UTF8) + .segments(MemorySegment.ofArray(new byte[4]), MemorySegment.ofArray(new byte[4])) + .arena(Arena.ofAuto()) + .build(); + + // When + int result = sut.bufferCount(); + + // Then + assertThat(result).isEqualTo(2); + } +} diff --git a/reader/src/test/java/io/github/dfa1/vortex/reader/decode/DictEncodingDecoderTest.java b/reader/src/test/java/io/github/dfa1/vortex/reader/decode/DictEncodingDecoderTest.java index 1ebf7625b..d8700960e 100644 --- a/reader/src/test/java/io/github/dfa1/vortex/reader/decode/DictEncodingDecoderTest.java +++ b/reader/src/test/java/io/github/dfa1/vortex/reader/decode/DictEncodingDecoderTest.java @@ -6,6 +6,7 @@ import io.github.dfa1.vortex.core.error.VortexException; import io.github.dfa1.vortex.core.model.EncodingId; import io.github.dfa1.vortex.core.proto.ProtoDictMetadata; +import io.github.dfa1.vortex.core.proto.ProtoPType; import io.github.dfa1.vortex.core.proto.ProtoVarBinMetadata; import io.github.dfa1.vortex.reader.ReadRegistry; import io.github.dfa1.vortex.reader.array.Array; @@ -21,6 +22,7 @@ import org.junit.jupiter.api.Test; import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.EnumSource; import org.junit.jupiter.params.provider.MethodSource; import java.lang.foreign.Arena; @@ -181,7 +183,7 @@ void malformedProtoMetadata_throws() { class PrimitiveLegacy { @ParameterizedTest(name = "codes={0}") - @org.junit.jupiter.params.provider.EnumSource(value = PType.class, names = {"U8", "U16", "U32"}) + @EnumSource(value = PType.class, names = {"U8", "U16", "U32"}) void singleByteMetadata_decodesViaLegacyPath(PType codePType) { // Given — legacy layout: 1-byte metadata (code ptype), child[0]=values, child[1]=codes long[] dict = {100, 200, 300}; @@ -216,7 +218,7 @@ void nonStandardCodeType_throws() { class ExpandGenericElemSize { @ParameterizedTest(name = "codes={0}") - @org.junit.jupiter.params.provider.EnumSource(value = PType.class, names = {"U8", "U16", "U32"}) + @EnumSource(value = PType.class, names = {"U8", "U16", "U32"}) void fastPath_copiesPerElement(PType codePType) { // Given — two 3-byte elements, codes [0, 1] (codesCap == rowCount, valuesCap > 1) MemorySegment values = bytes(1, 2, 3, 4, 5, 6); @@ -231,7 +233,7 @@ void fastPath_copiesPerElement(PType codePType) { } @ParameterizedTest(name = "codes={0}") - @org.junit.jupiter.params.provider.EnumSource(value = PType.class, names = {"U8", "U16", "U32"}) + @EnumSource(value = PType.class, names = {"U8", "U16", "U32"}) void slowPath_broadcastsSingleElement(PType codePType) { // Given — one 3-byte element (valuesCap == 1) forces the broadcast branch MemorySegment values = bytes(7, 8, 9); @@ -246,7 +248,7 @@ void slowPath_broadcastsSingleElement(PType codePType) { } @ParameterizedTest(name = "codes={0}") - @org.junit.jupiter.params.provider.EnumSource(value = PType.class, names = {"U8", "U16", "U32"}) + @EnumSource(value = PType.class, names = {"U8", "U16", "U32"}) void broadcastCodes_whenCodesShorterThanRowCount(PType codePType) { // Given — a single code element (codesCap < rowCount) takes the codes-broadcast branch MemorySegment values = bytes(10, 11, 12, 20, 21, 22); @@ -532,7 +534,7 @@ void codesBroadcast_slowPathSetsValidityBits() { } @ParameterizedTest(name = "codes={0}") - @org.junit.jupiter.params.provider.EnumSource(value = PType.class, names = {"U16", "U32"}) + @EnumSource(value = PType.class, names = {"U16", "U32"}) void poolNull_readsWiderCodeWidths(PType codePType) { // Given — codes at the wider widths (U16/U32) with a pool-null slot. rowValidity must // read each code at the correct stride via readCode's U16/U32 arms (only exercised when @@ -603,6 +605,237 @@ private MemorySegment boolBitmap(boolean... valid) { } } + /// Adversarial codes and dictionary offsets from an untrusted file (TODO.md §Security, + /// per-encoding adversarial tests). `poolValid` already guarded codes against the + /// row-validity pool, but the value **expansion** loops indexed the values pool with no + /// bounds check at all: a code pointing past the pool escaped as a raw + /// `IndexOutOfBoundsException` from the segment access instead of a [VortexException]. + /// + /// A codes ptype that cannot address the whole pool (u8 codes, pool longer than 256) is + /// deliberately NOT an error: the writer picks the codes width from the codes actually + /// emitted, so a pool whose tail is unreachable is well-formed, and neither the format + /// spec nor the Rust reference rejects it. + @Nested + class AdversarialCodes { + + @ParameterizedTest(name = "codes={0}") + @EnumSource(value = PType.class, names = {"U8", "U16", "U32"}) + void codePastValuesPool_protoFastPath_throws(PType codePType) { + // Given a 2-entry I32 pool and a code of 7; codesCap == rowCount and valuesCap > 1, + // so the un-clamped fast expansion branch runs and indexes past the pool + MemorySegment codes = codeSegment(codePType, new long[]{0, 7}); + MemorySegment values = TestSegments.leInts(10, 20); + + // When / Then + assertThatThrownBy(() -> decodeProtoSegments(DType.I32, codePType, codes, values, 2, 2)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("out of range for a values pool"); + } + + @Test + void codePastValuesPool_legacyPath_throws() { + // Given the legacy 1-byte-metadata layout with the same overrunning code + MemorySegment values = TestSegments.leLongs(1, 2); + MemorySegment codes = u8Codes(0, 9); + + // When / Then + assertThatThrownBy(() -> decodeLegacy(DType.I64, PType.U8, values, codes, 2)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("out of range for a values pool"); + } + + @Test + void codePastDictionaryOffsets_utf8_throws() { + // Given a 2-entry string dictionary and a code of 5, which walks off the + // dictionary offsets segment when the row is read + VarBinArray array = decodeLegacyUtf8("abcde", TestSegments.leLongs(0, 2, 5), u8Codes(5)); + + // When / Then + assertThatThrownBy(() -> array.getString(0)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("dict value offset index"); + } + + @Test + void nonMonotonicDictionaryOffsets_utf8_throws() { + // Given dictionary offsets [0, 5, 2]: entry 1 spans [5, 2), a negative length that + // used to reach `new byte[end - start]` as a NegativeArraySizeException + VarBinArray array = decodeLegacyUtf8("abcde", TestSegments.leLongs(0, 5, 2), u8Codes(1)); + + // When / Then + assertThatThrownBy(() -> array.getString(0)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("out of range for a data buffer"); + } + + @Test + void dictionaryOffsetPastBytes_utf8_throws() { + // Given an entry ending at 100 over a 5-byte dictionary bytes buffer + VarBinArray array = decodeLegacyUtf8("abcde", TestSegments.leLongs(0, 100), u8Codes(0)); + + // When / Then + assertThatThrownBy(() -> array.getString(0)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("out of range for a data buffer"); + } + + @Test + void truncatedCodesSegment_utf8_throws() { + // Given 3 declared rows but only 2 codes on the wire: the third row reads past + // the codes segment, which must fail loudly rather than as a raw IOOBE + MemorySegment bytes = MemorySegment.ofArray("abcde".getBytes(StandardCharsets.UTF_8)); + MemorySegment meta = MemorySegment.ofArray(new byte[]{(byte) PType.U8.ordinal()}); + ArrayNode node = new ArrayNode(EncodingId.VORTEX_DICT, meta, new ArrayNode[0], new int[]{0, 1, 2}); + DecodeContext ctx = new DecodeContext(node, DType.UTF8, 3, + new MemorySegment[]{bytes, TestSegments.leLongs(0, 2, 5), u8Codes(0, 1)}, + REGISTRY, Arena.ofAuto()); + VarBinArray array = (VarBinArray) SUT.decode(ctx); + + // When / Then + assertThatThrownBy(() -> array.getString(2)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("out of range for a codes segment"); + } + + @Test + void emptyValuesPool_throws() { + // Given a zero-length values child: the broadcast branch wraps codes with + // `% valuesCap`, so an empty pool used to die with ArithmeticException + // (values_len stays 1 so the metadata is not elided to an empty proto payload) + MemorySegment values = MemorySegment.ofArray(new byte[0]); + + // When / Then + assertThatThrownBy(() -> decodeProtoSegments(DType.I32, PType.U8, u8Codes(0, 0), values, 1, 2)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("empty dict child"); + } + + @Test + void emptyCodesChild_throws() { + // Given a zero-length codes child, which makes the `% codesCap` wrap divide by zero + MemorySegment codes = MemorySegment.ofArray(new byte[0]); + + // When / Then + assertThatThrownBy(() -> decodeProtoSegments(DType.I32, PType.U8, codes, TestSegments.leInts(1, 2), 2, 2)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("empty dict child"); + } + + @Test + void legacyPathWithoutValuesChild_throws() { + // Given the legacy layout with an empty children vector: the values buffer used + // to be reached by three raw array indexes on untrusted data + MemorySegment meta = MemorySegment.ofArray(new byte[]{(byte) PType.U8.ordinal()}); + ArrayNode node = new ArrayNode(EncodingId.VORTEX_DICT, meta, new ArrayNode[0], new int[]{}); + DecodeContext ctx = new DecodeContext(node, DType.I64, 2, + new MemorySegment[]{TestSegments.leLongs(1, 2)}, REGISTRY, Arena.ofAuto()); + + // When / Then + assertThatThrownBy(() -> SUT.decode(ctx)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("child index 0 out of bounds"); + } + + @Test + void truncatedCodesSegment_limited_throws() { + // Given 5 declared rows but 2 codes on the wire, cut to 3 rows by a scan limit: + // slicing the codes segment used to escape as a raw IOOBE + MemorySegment bytes = MemorySegment.ofArray("abcde".getBytes(StandardCharsets.UTF_8)); + MemorySegment meta = MemorySegment.ofArray(new byte[]{(byte) PType.U8.ordinal()}); + ArrayNode node = new ArrayNode(EncodingId.VORTEX_DICT, meta, new ArrayNode[0], new int[]{0, 1, 2}); + DecodeContext ctx = new DecodeContext(node, DType.UTF8, 5, + new MemorySegment[]{bytes, TestSegments.leLongs(0, 2, 5), u8Codes(0, 1)}, + REGISTRY, Arena.ofAuto()); + VarBinArray array = (VarBinArray) SUT.decode(ctx); + + // When / Then + assertThatThrownBy(() -> array.limited(3)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("holds fewer than 3 U8 codes"); + } + + @Test + void codePastDictionaryOffsets_forEachByteLength_blamesTheCode() { + // Given a code past a dictionary with I32 value offsets — the shape FSST + // decompression produces, and the only one that takes the branch-free bulk + // length path. That path catches the overrun at the loop boundary, and the cold + // re-validation must name the offending code rather than blaming the offsets + // segment it happened to read. + VarBinArray array = decodeProtoUtf8("fizzbuzz", TestSegments.leInts(0, 4, 8), u8Codes(5), 2); + + // When / Then + assertThatThrownBy(() -> array.forEachByteLength(len -> { + })) + .isInstanceOf(VortexException.class) + .hasMessageContaining("dict code 5 at row 0 out of range for 3 value offsets"); + } + + @Test + void consumerFailure_forEachByteLength_propagatesUnchanged() { + // Given well-formed dictionary data and a consumer that throws its own + // IndexOutOfBoundsException: the boundary catch must not relabel a caller bug as + // malformed input + VarBinArray array = decodeLegacyUtf8("abcde", TestSegments.leLongs(0, 2, 5), u8Codes(1)); + + // When / Then + assertThatThrownBy(() -> array.forEachByteLength(len -> { + throw new IndexOutOfBoundsException("sink says no"); + })) + .isInstanceOf(IndexOutOfBoundsException.class) + .isNotInstanceOf(VortexException.class) + .hasMessage("sink says no"); + } + + @Test + void poolWiderThanCodePType_decodesTheAddressableEntries() { + // Given a 300-entry pool with u8 codes: entries 256+ are unreachable, which is + // well-formed rather than an error (the writer sizes codes from what it emits), + // so decode must succeed for the entries the codes can address. + int[] pool = new int[300]; + for (int i = 0; i < pool.length; i++) { + pool[i] = i * 10; + } + MemorySegment values = TestSegments.leInts(pool); + + // When + Array result = decodeProtoSegments(DType.I32, PType.U8, u8Codes(0, 255), values, 300, 2); + + // Then + assertThat(((IntArray) result).getInt(0)).isZero(); + assertThat(((IntArray) result).getInt(1)).isEqualTo(2550); + } + + /// Builds the proto utf8 dict layout with a VarBin values child carrying I32 value + /// offsets — the shape FSST decompression yields, and the one whose bulk length walk + /// takes the branch-free I32 fast path. + private VarBinArray decodeProtoUtf8(String dictBytes, MemorySegment offsets, + MemorySegment codes, int valuesLen) { + MemorySegment bytes = MemorySegment.ofArray(dictBytes.getBytes(StandardCharsets.UTF_8)); + MemorySegment dictMeta = MemorySegment.ofArray( + new ProtoDictMetadata(valuesLen, protoPType(PType.U8), null, null).encode()); + MemorySegment varBinMeta = MemorySegment.ofArray( + new ProtoVarBinMetadata(protoPType(PType.I32)).encode()); + ArrayNode valuesNode = new ArrayNode(EncodingId.VORTEX_VARBIN, varBinMeta, + new ArrayNode[]{primitiveNode(2)}, new int[]{1}); + ArrayNode dictNode = new ArrayNode(EncodingId.VORTEX_DICT, dictMeta, + new ArrayNode[]{primitiveNode(0), valuesNode}, new int[]{}); + DecodeContext ctx = new DecodeContext(dictNode, DType.UTF8, 1, + new MemorySegment[]{codes, bytes, offsets}, REGISTRY, Arena.ofAuto()); + return (VarBinArray) SUT.decode(ctx); + } + + /// Builds the legacy (buffer-only) utf8 dict layout: dictionary bytes, I64 value + /// offsets and u8 codes, with the 1-byte code-ptype metadata. + private VarBinArray decodeLegacyUtf8(String dictBytes, MemorySegment offsets, MemorySegment codes) { + MemorySegment bytes = MemorySegment.ofArray(dictBytes.getBytes(StandardCharsets.UTF_8)); + MemorySegment meta = MemorySegment.ofArray(new byte[]{(byte) PType.U8.ordinal()}); + ArrayNode node = new ArrayNode(EncodingId.VORTEX_DICT, meta, new ArrayNode[0], new int[]{0, 1, 2}); + DecodeContext ctx = new DecodeContext(node, DType.UTF8, 1, + new MemorySegment[]{bytes, offsets, codes}, REGISTRY, Arena.ofAuto()); + return (VarBinArray) SUT.decode(ctx); + } + } + // ── parameter sources ────────────────────────────────────────────────────── static Stream codeAndValueTypes() { @@ -649,8 +882,8 @@ private static ArrayNode primitiveNode(int bufferIndex) { return new ArrayNode(EncodingId.VORTEX_PRIMITIVE, null, new ArrayNode[0], new int[]{bufferIndex}); } - private static io.github.dfa1.vortex.core.proto.ProtoPType protoPType(PType core) { - return io.github.dfa1.vortex.core.proto.ProtoPType.valueOf(core.name()); + private static ProtoPType protoPType(PType core) { + return ProtoPType.valueOf(core.name()); } // ── segment builders (little-endian) ─────────────────────────────────────── diff --git a/reader/src/test/java/io/github/dfa1/vortex/reader/decode/SparseEncodingDecoderTest.java b/reader/src/test/java/io/github/dfa1/vortex/reader/decode/SparseEncodingDecoderTest.java index 7a9b142cb..8b5a0e8d8 100644 --- a/reader/src/test/java/io/github/dfa1/vortex/reader/decode/SparseEncodingDecoderTest.java +++ b/reader/src/test/java/io/github/dfa1/vortex/reader/decode/SparseEncodingDecoderTest.java @@ -16,6 +16,7 @@ import io.github.dfa1.vortex.reader.array.DoubleArray; import io.github.dfa1.vortex.reader.array.MaskedArray; import io.github.dfa1.vortex.reader.array.VarBinArray; +import org.junit.jupiter.api.Nested; import org.junit.jupiter.api.Test; import java.lang.foreign.Arena; @@ -209,6 +210,176 @@ void decode_threeChildren_throws() { .hasMessageContaining("2"); } + /// Malformed-input cases for `vortex.sparse` (TODO.md §Security, ADR 0003): the patch count, + /// the patch children's buffers and the patch-value offsets all come from untrusted file + /// bytes and none of them are cross-validated at decode time. Every case here crashed with a + /// raw JDK exception before the guards landed — the class named in each comment is what the + /// reader used to leak. + /// + /// Unsorted patch indices are covered separately in `LazySparseArrayTest.AdversarialInput`: + /// the crash surfaces in the lazy array's `forEach`/`fold` walker, not in this decoder's + /// `decode()`, since the primitive/bool paths stay lazy by design. + @Nested + class AdversarialInput { + + @Test + void missingPatchesMetadata_throws() { + // Given — non-empty metadata that never sets field 1 (patches): an unrelated + // unknown field (tag = field 2, varint wire type 0, value 0) that the proto reader + // silently skips. This is different from *absent* metadata (already rejected by the + // "missing metadata" check above): the bytes are present and parse cleanly, but + // `patches` stays null, which used to NPE on `patches.len()`. + MemorySegment meta = MemorySegment.ofArray(new byte[]{0x10, 0x00}); + ArrayNode node = new ArrayNode(EncodingId.VORTEX_SPARSE, meta, + new ArrayNode[]{primitiveNode(1), primitiveNode(2)}, new int[]{0}); + MemorySegment[] segs = {f64Fill(0.0), TestSegments.leInts(0), TestSegments.leDoubles(1.0)}; + DecodeContext ctx = new DecodeContext(node, DType.F64, 1, segs, REGISTRY, Arena.ofAuto()); + + // When / Then + assertThatThrownBy(() -> SUT.decode(ctx)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("missing patches metadata"); + } + + /// `ClassCastException` — `decodeChild` dispatches on the *child node's own* encoding + /// id, not on the ptype this decoder expects. A crafted values child of the wrong + /// concrete array type (here `vortex.bool` under an `i64` sparse array) decodes without + /// error and used to blow up at the unchecked `(LongArray)` cast. + @Test + void patchValuesChildWrongType_throws() { + // Given — an i64 sparse array whose values child is `vortex.bool`, not primitive i64 + ArrayNode boolValuesNode = new ArrayNode(EncodingId.VORTEX_BOOL, null, new ArrayNode[0], new int[]{2}); + MemorySegment[] segs = {f64Fill(0.0), TestSegments.leInts(1), boolBitmap(true)}; + + // When / Then + assertThatThrownBy(() -> decode(DType.I64, 1, 0, PType.U32, 5, segs, + primitiveNode(1), boolValuesNode)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("patch values child decoded to unexpected type"); + } + + /// `ArithmeticException: / by zero` — the `Materialized*` accessors broadcast an + /// undersized buffer with `i % elementCount`, and a zero-byte patch-indices buffer makes + /// `elementCount` zero. + @Test + void emptyPatchIndicesChild_throws() { + // Given — metadata claims 2 patches but the patch-indices segment carries no bytes + MemorySegment[] segs = {f64Fill(0.0), empty(), TestSegments.leDoubles(5.0, 7.0)}; + + // When / Then + assertThatThrownBy(() -> decode(DType.F64, 2, 0, PType.U32, 5, segs, + primitiveNode(1), primitiveNode(2))) + .isInstanceOf(VortexException.class) + .hasMessageContaining("empty patch indices child"); + } + + /// Same divide-by-zero, reached through the values child instead of the indices child. + @Test + void emptyPatchValuesChild_throws() { + // Given — metadata claims 2 patches but the patch-values segment carries no bytes + MemorySegment[] segs = {f64Fill(0.0), TestSegments.leInts(1, 3), empty()}; + + // When / Then + assertThatThrownBy(() -> decode(DType.F64, 2, 0, PType.U32, 5, segs, + primitiveNode(1), primitiveNode(2))) + .isInstanceOf(VortexException.class) + .hasMessageContaining("empty patch values child"); + } + + /// `OutOfMemoryError` — a null fill makes the decoder build a per-patch validity bitmap + /// sized from the declared patch count, so an absurd count reserved exabytes of direct + /// memory before any buffer was even read. Patches sit at distinct row positions, so the + /// count can never exceed the row count (Rust `Patches::new`: `indices.len() <= array_len`). + @Test + void patchCountAboveRowCount_throws() { + // Given — 5 rows but a patch count of 2^61, with a null fill to reach the bitmap alloc + MemorySegment[] segs = {nullFill(), TestSegments.leInts(1, 3), TestSegments.leDoubles(5.0, 7.0)}; + + // When / Then + assertThatThrownBy(() -> decode(nullableF64(), 1L << 61, 0, PType.U32, 5, segs, + primitiveNode(1), primitiveNode(2))) + .isInstanceOf(VortexException.class) + .hasMessageContaining("patch count"); + } + + /// The other end of the same range check: proto `len` is a signed int64, so a crafted + /// file can declare a negative patch count. + @Test + void negativePatchCount_throws() { + // Given — a negative declared patch count + MemorySegment[] segs = {f64Fill(0.0), TestSegments.leInts(1, 3), TestSegments.leDoubles(5.0, 7.0)}; + + // When / Then + assertThatThrownBy(() -> decode(DType.F64, -1, 0, PType.U32, 5, segs, + primitiveNode(1), primitiveNode(2))) + .isInstanceOf(VortexException.class) + .hasMessageContaining("patch count"); + } + + /// `IndexOutOfBoundsException` — the utf8/binary path merges patches eagerly and reads + /// `numPatches + 1` entries from the patch-value offsets buffer. Declaring 4 patches over + /// a values child that only carries 2 walked past the end of that buffer. + @Test + void varBinPatchCountBeyondValueOffsets_throws() { + // Given — 4 declared patches but only 2 value offsets pairs ("b", "d") + MemorySegment[] segs = { + nullFill(), + TestSegments.leInts(0, 1, 2, 3), // 4 patch indices, so the count is plausible + utf8Bytes("bd"), + TestSegments.leInts(0, 1, 2) // only 3 offsets = 2 values + }; + + // When / Then + assertThatThrownBy(() -> decode(nullableUtf8(), 4, 0, PType.U32, 5, segs, + primitiveNode(1), varBinNode(2, 3))) + .isInstanceOf(VortexException.class) + .hasMessageContaining("patch value offsets out of range"); + } + + /// `IndexOutOfBoundsException` from `MemorySegment.copy` — non-monotonic patch-value + /// offsets make the per-patch lengths cancel out, so the output buffer is sized far + /// smaller than the first patch that actually gets copied into it. + @Test + void varBinNonMonotonicValueOffsets_throws() { + // Given — offsets 0, 2, 0: patch 0 is 2 bytes long, patch 1 is -2, total 0 bytes + MemorySegment[] segs = { + nullFill(), + TestSegments.leInts(0, 1), + utf8Bytes("bd"), + TestSegments.leInts(0, 2, 0) + }; + + // When / Then + assertThatThrownBy(() -> decode(nullableUtf8(), 2, 0, PType.U32, 5, segs, + primitiveNode(1), varBinNode(2, 3))) + .isInstanceOf(VortexException.class) + .hasMessageContaining("patch value offsets out of range"); + } + + /// The offsets-vs-payload cross-check: a patch-value offsets buffer claiming more bytes + /// than the value buffer holds must be rejected before the output buffer is sized from it. + @Test + void varBinValueOffsetsBeyondValueBuffer_throws() { + // Given — 2 bytes of value data but offsets claiming a 1 GiB final patch + MemorySegment[] segs = { + nullFill(), + TestSegments.leInts(0, 1), + utf8Bytes("bd"), + TestSegments.leInts(0, 1, 1 << 30) + }; + + // When / Then + assertThatThrownBy(() -> decode(nullableUtf8(), 2, 0, PType.U32, 5, segs, + primitiveNode(1), varBinNode(2, 3))) + .isInstanceOf(VortexException.class) + .hasMessageContaining("patch bytes"); + } + } + + private static MemorySegment empty() { + return MemorySegment.ofArray(new byte[0]); + } + private static Array decode(DType dtype, long numPatches, long offset, PType indicesPtype, long n, MemorySegment[] segs, ArrayNode idxNode, ArrayNode valNode) { ProtoPatchesMetadata patches = new ProtoPatchesMetadata( diff --git a/reader/src/test/java/io/github/dfa1/vortex/reader/decode/StructEncodingDecoderTest.java b/reader/src/test/java/io/github/dfa1/vortex/reader/decode/StructEncodingDecoderTest.java new file mode 100644 index 000000000..917b9fe69 --- /dev/null +++ b/reader/src/test/java/io/github/dfa1/vortex/reader/decode/StructEncodingDecoderTest.java @@ -0,0 +1,168 @@ +package io.github.dfa1.vortex.reader.decode; + +import io.github.dfa1.vortex.core.error.VortexException; +import io.github.dfa1.vortex.core.model.ColumnName; +import io.github.dfa1.vortex.core.model.DType; +import io.github.dfa1.vortex.core.model.EncodingId; +import io.github.dfa1.vortex.core.testing.TestSegments; +import io.github.dfa1.vortex.reader.ReadRegistry; +import io.github.dfa1.vortex.reader.array.Array; +import io.github.dfa1.vortex.reader.array.LongArray; +import io.github.dfa1.vortex.reader.array.MaterializedLongArray; +import io.github.dfa1.vortex.reader.array.StructArray; +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; + +import java.lang.foreign.Arena; +import java.lang.foreign.MemorySegment; +import java.util.List; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +class StructEncodingDecoderTest { + + private static final StructEncodingDecoder SUT = new StructEncodingDecoder(); + private static final ReadRegistry REGISTRY = TestRegistry.ofDecoders( + SUT, new PrimitiveEncodingDecoder(), new BoolEncodingDecoder(), new NotBoolDecoder()); + + @Test + void encodingId_isVortexStruct() { + // Given / When / Then + assertThat(SUT.encodingId()).isEqualTo(EncodingId.VORTEX_STRUCT); + } + + @Test + void decode_twoFields_returnsStructArrayWithBothColumns() { + // Given — a two-field struct whose children are one i64 buffer each + MemorySegment[] segs = {TestSegments.leLongs(1, 2), TestSegments.leLongs(30, 40)}; + + // When + Array result = decode(structOf("a", "b"), 2, segs, primitiveNode(0), primitiveNode(1)); + + // Then + assertThat(result).isInstanceOf(StructArray.class); + StructArray struct = (StructArray) result; + assertThat(((LongArray) struct.field(0)).getLong(0)).isEqualTo(1); + assertThat(((LongArray) struct.field(1)).getLong(1)).isEqualTo(40); + } + + /// Malformed-input cases for `vortex.struct` (TODO.md §Security, ADR 0003). + /// + /// The TODO's two Struct gotchas — `fieldNames.size() != children.size()` and an invalid + /// field name — are both a *dtype* concern rather than an encoding one, and are guarded and + /// tested one layer up in `PostscriptParser.convertDType` (see + /// `PostscriptParserDTypeGuardsTest`: names/dtypes arity mismatch, duplicate names, control + /// characters in a name). Malformed UTF-8 in a name cannot reach a guard at all: the + /// FlatBuffer string decode substitutes U+FFFD rather than failing, so there is nothing left + /// to reject by the time the name is a `String`. + /// + /// What genuinely belongs here is the encoding-level arity check: the `ArrayNode` child + /// count is independent file data and must agree with the dtype's field count (plus at most + /// one leading validity child). + @Nested + class AdversarialInput { + + @Test + void fewerChildrenThanFields_throws() { + // Given — a three-field struct dtype but only two children in the array node + MemorySegment[] segs = {TestSegments.leLongs(1, 2), TestSegments.leLongs(30, 40)}; + + // When / Then + assertThatThrownBy(() -> decode(structOf("a", "b", "c"), 2, segs, + primitiveNode(0), primitiveNode(1))) + .isInstanceOf(VortexException.class) + .hasMessageContaining("children for struct dtype"); + } + + /// One child over the validity-inclusive maximum: `nfields + 1` is legal (leading + /// validity), `nfields + 2` is not, so this pins the upper edge of the accepted range. + @Test + void moreChildrenThanFieldsPlusValidity_throws() { + // Given — a one-field struct dtype with three children + MemorySegment[] segs = {TestSegments.leLongs(1, 2)}; + + // When / Then + assertThatThrownBy(() -> decode(structOf("a"), 2, segs, + primitiveNode(0), primitiveNode(0), primitiveNode(0))) + .isInstanceOf(VortexException.class) + .hasMessageContaining("children for struct dtype"); + } + + /// The non-struct dtype branch is a scalar wrapper accepting exactly one child (values) + /// or two (validity + values); anything else is a corrupt node. + @Test + void scalarWrapperWithThreeChildren_throws() { + // Given — an i64 dtype (not a struct) under a struct-encoded node with three children + MemorySegment[] segs = {TestSegments.leLongs(1, 2)}; + + // When / Then + assertThatThrownBy(() -> decode(DType.I64, 2, segs, + primitiveNode(0), primitiveNode(0), primitiveNode(0))) + .isInstanceOf(VortexException.class) + .hasMessageContaining("unexpected child count 3"); + } + + /// A validity child is only meaningful as a bool array; a crafted node whose validity + /// subtree decodes to something else must fail loudly instead of being cast blind. + @Test + void scalarWrapperValidityChildNotBool_throws() { + // Given — the leading validity child decodes to an i64 array. Real files reach this + // through encodings that honor their own node's shape over the requested BOOL dtype; + // the stub decoder reproduces that without depending on any one of them. + MemorySegment[] segs = {TestSegments.leLongs(1, 2), TestSegments.leLongs(30, 40)}; + ArrayNode notBool = new ArrayNode(NotBoolDecoder.ID, null, new ArrayNode[0], new int[0]); + + // When / Then + assertThatThrownBy(() -> decode(DType.I64, 2, segs, notBool, primitiveNode(1))) + .isInstanceOf(VortexException.class) + .hasMessageContaining("validity decoded to unexpected type"); + } + + /// The same guard on the struct-dtype path, where the validity child is the extra + /// `nfields + 1`-th child rather than the first of a scalar wrapper pair. + @Test + void structValidityChildNotBool_throws() { + // Given — a one-field struct with a leading validity child that decodes to i64 + MemorySegment[] segs = {TestSegments.leLongs(1, 2)}; + ArrayNode notBool = new ArrayNode(NotBoolDecoder.ID, null, new ArrayNode[0], new int[0]); + + // When / Then + assertThatThrownBy(() -> decode(structOf("a"), 2, segs, notBool, primitiveNode(0))) + .isInstanceOf(VortexException.class) + .hasMessageContaining("validity decoded to unexpected type"); + } + } + + /// Decodes anything into an i64 array — the stand-in for a validity subtree that decodes + /// successfully but not into a [io.github.dfa1.vortex.reader.array.BoolArray]. + private static final class NotBoolDecoder implements EncodingDecoder { + + private static final EncodingId ID = EncodingId.parse("test.notbool"); + + @Override + public EncodingId encodingId() { + return ID; + } + + @Override + public Array decode(DecodeContext ctx) { + return new MaterializedLongArray(DType.I64, ctx.rowCount(), TestSegments.leLongs(1, 2)); + } + } + + private static Array decode(DType dtype, long rowCount, MemorySegment[] segs, ArrayNode... children) { + ArrayNode node = new ArrayNode(EncodingId.VORTEX_STRUCT, null, children, new int[0]); + return SUT.decode(new DecodeContext(node, dtype, rowCount, segs, REGISTRY, Arena.ofAuto())); + } + + private static DType.Struct structOf(String... names) { + List fieldNames = List.of(names).stream().map(ColumnName::of).toList(); + List fieldTypes = fieldNames.stream().map(n -> DType.I64).toList(); + return new DType.Struct(fieldNames, fieldTypes, false); + } + + private static ArrayNode primitiveNode(int bufferIndex) { + return new ArrayNode(EncodingId.VORTEX_PRIMITIVE, null, new ArrayNode[0], new int[]{bufferIndex}); + } +} diff --git a/reader/src/test/java/io/github/dfa1/vortex/reader/decode/VarBinEncodingDecoderTest.java b/reader/src/test/java/io/github/dfa1/vortex/reader/decode/VarBinEncodingDecoderTest.java index e4cb4b291..0079a8371 100644 --- a/reader/src/test/java/io/github/dfa1/vortex/reader/decode/VarBinEncodingDecoderTest.java +++ b/reader/src/test/java/io/github/dfa1/vortex/reader/decode/VarBinEncodingDecoderTest.java @@ -1,12 +1,15 @@ package io.github.dfa1.vortex.reader.decode; +import io.github.dfa1.vortex.core.error.VortexException; import io.github.dfa1.vortex.core.model.DType; import io.github.dfa1.vortex.core.model.EncodingId; +import io.github.dfa1.vortex.core.model.PType; import io.github.dfa1.vortex.core.testing.TestSegments; import io.github.dfa1.vortex.core.proto.ProtoVarBinMetadata; import io.github.dfa1.vortex.reader.ReadRegistry; import io.github.dfa1.vortex.reader.array.Array; import io.github.dfa1.vortex.reader.array.VarBinArray; +import org.junit.jupiter.api.Nested; import org.junit.jupiter.api.Test; import java.lang.foreign.Arena; @@ -14,6 +17,7 @@ import java.nio.charset.StandardCharsets; import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; class VarBinEncodingDecoderTest { @@ -50,6 +54,109 @@ void decode_i32Offsets_happyPath() { assertThat(arr.getBytes(2)).containsExactly('c'); } + /// Adversarial offsets from an untrusted file (TODO.md §Security, per-encoding + /// adversarial tests). `decode()` deliberately never scans the offsets — VarBin decode + /// is zero-copy and lazy — so every malformed offset has to be caught by the accessors, + /// and always as a [VortexException]: a non-monotonic pair used to reach + /// `new byte[end - start]` as a `NegativeArraySizeException`, and an offset past the + /// data buffer used to reach `MemorySegment.copy` as a raw `IndexOutOfBoundsException` + /// (or, in `getByteLength`, to be reported silently as a bogus length). + @Nested + class AdversarialOffsets { + + @Test + void nonMonotonicOffsets_getBytes_throws() { + // Given offsets [0, 5, 2] over "abcde": row 1 spans [5, 2), a negative length + MemorySegment data = MemorySegment.ofArray("abcde".getBytes(StandardCharsets.UTF_8)); + VarBinArray array = (VarBinArray) SUT.decode(ctx(i32OffsetsMeta(), data, TestSegments.leInts(0, 5, 2), 2)); + + // When / Then + assertThatThrownBy(() -> array.getBytes(1)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("out of range for a data buffer"); + } + + @Test + void nonMonotonicOffsets_getByteLength_throws() { + // Given the same descending pair — the length itself must be rejected, not + // handed back as a negative int + MemorySegment data = MemorySegment.ofArray("abcde".getBytes(StandardCharsets.UTF_8)); + VarBinArray array = (VarBinArray) SUT.decode(ctx(i32OffsetsMeta(), data, TestSegments.leInts(0, 5, 2), 2)); + + // When / Then + assertThatThrownBy(() -> array.getByteLength(1)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("out of range for a data buffer"); + } + + @Test + void negativeOffset_getBytes_throws() { + // Given a negative I32 offset (0xFFFFFFFF widens to -1), which would copy from + // before the start of the data buffer + MemorySegment data = MemorySegment.ofArray("abc".getBytes(StandardCharsets.UTF_8)); + VarBinArray array = (VarBinArray) SUT.decode(ctx(i32OffsetsMeta(), data, TestSegments.leInts(-1, 2), 1)); + + // When / Then + assertThatThrownBy(() -> array.getBytes(0)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("out of range for a data buffer"); + } + + @Test + void offsetPastDataBuffer_getBytes_throws() { + // Given an end offset of 100 over a 3-byte data buffer + MemorySegment data = MemorySegment.ofArray("abc".getBytes(StandardCharsets.UTF_8)); + VarBinArray array = (VarBinArray) SUT.decode(ctx(i32OffsetsMeta(), data, TestSegments.leInts(0, 100), 1)); + + // When / Then + assertThatThrownBy(() -> array.getBytes(0)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("out of range for a data buffer"); + } + + @Test + void offsetPastDataBuffer_getByteLength_throws() { + // Given the same overrun — getByteLength used to return 100 silently, letting a + // caller size a copy from data it never owned + MemorySegment data = MemorySegment.ofArray("abc".getBytes(StandardCharsets.UTF_8)); + VarBinArray array = (VarBinArray) SUT.decode(ctx(i32OffsetsMeta(), data, TestSegments.leInts(0, 100), 1)); + + // When / Then + assertThatThrownBy(() -> array.getByteLength(0)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("out of range for a data buffer"); + } + + @Test + void offsetPastDataBuffer_limited_throws() { + // Given a truncating slice whose cut point (row 1 -> offset 100) lies past the + // 3-byte data buffer: asSlice would have thrown IndexOutOfBoundsException + MemorySegment data = MemorySegment.ofArray("abc".getBytes(StandardCharsets.UTF_8)); + VarBinArray array = (VarBinArray) SUT.decode(ctx(i32OffsetsMeta(), data, TestSegments.leInts(0, 100, 200), 2)); + + // When / Then + assertThatThrownBy(() -> array.limited(1)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("out of range for a data buffer"); + } + + @Test + void truncatedOffsetsSegment_getBytes_throws() { + // Given an offsets child with n+1 declared but only n values present. The + // decoder broadcast-materializes it (offCap < n + 1), so reading past the end is + // only possible on a directly built array — this exercises the readOffset guard + // directly against a short offsets segment. + MemorySegment data = MemorySegment.ofArray("abc".getBytes(StandardCharsets.UTF_8)); + VarBinArray array = new VarBinArray.OffsetMode(DType.UTF8, 4, data, + TestSegments.leInts(0, 1, 2), PType.I32); + + // When / Then + assertThatThrownBy(() -> array.getBytes(3)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("out of range for an offsets segment"); + } + } + @Test void decode_broadcastOffsets_singleOffsetExpandsToAllRows() { // Given an offsets child holding a single value (as ConstantEncoding emits): diff --git a/reader/src/test/java/io/github/dfa1/vortex/reader/decode/VarBinViewEncodingDecoderTest.java b/reader/src/test/java/io/github/dfa1/vortex/reader/decode/VarBinViewEncodingDecoderTest.java index 4c7fe92a7..63eef6bf8 100644 --- a/reader/src/test/java/io/github/dfa1/vortex/reader/decode/VarBinViewEncodingDecoderTest.java +++ b/reader/src/test/java/io/github/dfa1/vortex/reader/decode/VarBinViewEncodingDecoderTest.java @@ -1,11 +1,13 @@ package io.github.dfa1.vortex.reader.decode; +import io.github.dfa1.vortex.core.error.VortexException; import io.github.dfa1.vortex.core.model.DType; import io.github.dfa1.vortex.core.model.EncodingId; import io.github.dfa1.vortex.core.io.VortexFormat; import io.github.dfa1.vortex.reader.ReadRegistry; import io.github.dfa1.vortex.reader.array.Array; import io.github.dfa1.vortex.reader.array.VarBinArray; +import org.junit.jupiter.api.Nested; import org.junit.jupiter.api.Test; import java.lang.foreign.Arena; @@ -74,6 +76,122 @@ void decode_noBuffers_throws() { .hasMessageContaining("at least 1 buffer"); } + /// Adversarial views from an untrusted file (TODO.md §Security, per-encoding adversarial + /// tests). The view headers are decoded lazily and were read with no validation at all: + /// a negative size reached `new byte[size]` as a `NegativeArraySizeException`, a bogus + /// buffer index indexed `dataBufs` as a raw `ArrayIndexOutOfBoundsException`, and a row + /// past the views segment (or a data offset past its buffer) escaped as a raw + /// `IndexOutOfBoundsException`. + @Nested + class AdversarialViews { + + @Test + void negativeViewSize_getBytes_throws() { + // Given a view whose size field is -1 + VarBinArray array = decodeViews(1, views -> views.set(VortexFormat.LE_INT, 0, -1)); + + // When / Then + assertThatThrownBy(() -> array.getBytes(0)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("negative varbin view size"); + } + + @Test + void negativeViewSize_getByteLength_throws() { + // Given the same header read through the length accessor + VarBinArray array = decodeViews(1, views -> views.set(VortexFormat.LE_INT, 0, -1)); + + // When / Then + assertThatThrownBy(() -> array.getByteLength(0)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("negative varbin view size"); + } + + @Test + void rowPastViewsSegment_throws() { + // Given 3 declared rows but only one 16-byte view on the wire + VarBinArray array = decodeViews(3, views -> views.set(VortexFormat.LE_INT, 0, 2)); + + // When / Then + assertThatThrownBy(() -> array.getBytes(2)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("out of range for a views segment"); + } + + @Test + void bufferIndexPastDataBuffers_throws() { + // Given a long (non-inlined) view pointing at data buffer 7 when the decoder + // handed the array a single buffer + VarBinArray array = decodeViews(1, views -> { + views.set(VortexFormat.LE_INT, 0, 20); // size > 12 -> long view + views.set(VortexFormat.LE_INT, 8, 7); // buffer index + views.set(VortexFormat.LE_INT, 12, 0); // offset within buffer + }); + + // When / Then + assertThatThrownBy(() -> array.getBytes(0)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("references data buffer 7"); + } + + @Test + void dataOffsetPastBuffer_throws() { + // Given a long view whose [offset, offset + size) window runs past its buffer + VarBinArray array = decodeViews(1, views -> { + views.set(VortexFormat.LE_INT, 0, 20); + views.set(VortexFormat.LE_INT, 8, 0); + views.set(VortexFormat.LE_INT, 12, 900); + }, MemorySegment.ofArray(new byte[32])); + + // When / Then + assertThatThrownBy(() -> array.getBytes(0)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("out of range for data buffer 0"); + } + + @Test + void truncatedViewsSegment_forEachByteLength_throws() { + // Given 3 declared rows over a single view: the bulk length walk sizes the + // segment once up front instead of running off it + VarBinArray array = decodeViews(3, views -> views.set(VortexFormat.LE_INT, 0, 2)); + + // When / Then + assertThatThrownBy(() -> array.forEachByteLength(len -> { + })) + .isInstanceOf(VortexException.class) + .hasMessageContaining("holds fewer than 3 views"); + } + + @Test + void truncatedViewsSegment_limited_throws() { + // Given the same shape truncated by a scan limit + VarBinArray array = decodeViews(3, views -> views.set(VortexFormat.LE_INT, 0, 2)); + + // When / Then + assertThatThrownBy(() -> array.limited(2)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("holds fewer than 2 views"); + } + + private VarBinArray decodeViews(long rowCount, java.util.function.Consumer writer, + MemorySegment... dataBufs) { + Arena arena = Arena.ofAuto(); + MemorySegment views = arena.allocate(16); + writer.accept(views); + // The decoder takes the views from the LAST buffer; data buffers come first. + MemorySegment[] segs = new MemorySegment[dataBufs.length + 1]; + System.arraycopy(dataBufs, 0, segs, 0, dataBufs.length); + segs[dataBufs.length] = views; + int[] bufferIndices = new int[segs.length]; + for (int i = 0; i < segs.length; i++) { + bufferIndices[i] = i; + } + ArrayNode node = new ArrayNode(EncodingId.VORTEX_VARBINVIEW, null, new ArrayNode[0], bufferIndices); + DecodeContext ctx = new DecodeContext(node, DType.UTF8, rowCount, segs, ReadRegistry.empty(), arena); + return (VarBinArray) SUT.decode(ctx); + } + } + /// Writes a ≤12-byte inline view: length prefix then the bytes packed in-place. private static void writeInlineView(MemorySegment views, int row, byte[] bytes) { long off = (long) row * 16; From 3e587d9e9d5bbaa92f9f5a82f4139219a3c8116f Mon Sep 17 00:00:00 2001 From: Davide Angelocola Date: Wed, 5 Aug 2026 21:21:59 +0200 Subject: [PATCH 2/2] docs(changelog): record adversarial-input hardening for reader decoders --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 77771624c..f49f98a51 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Fixed + +- Malformed files no longer crash the reader with a raw JDK exception when decoding VarBin, Dict, Bitpacked, ALP, Sparse, Chunked, or Struct columns — every case now fails as `VortexException`. ([ef982992](https://github.com/dfa1/vortex-java/commit/ef982992)) + ### Added - `MemorySize` domain primitive in `core.model` replaces raw byte-count `long`/`int` arithmetic: validated non-negative at construction, with `ofKiB`/`ofMiB`/`ofGiB` factories and a `toGiB()` display accessor. Migrated `WriteOptions#globalDictMaxRetainedBytes()`, `PostscriptParser`'s layout-metadata size cap, `DType.Extension#MAX_METADATA_SIZE`, and `VortexHttpReader`'s HTTP tail-fetch window. ([#321](https://github.com/dfa1/vortex-java/issues/321))