perf(fsst): close the vortex-jni gap on encode and decode hot paths

Compressor: single unaligned 8-byte word loads on the fast path (byte-by-byte
loadWord kept for the <8-byte tail), escape literal taken from the loaded word.
LossyPerfectHashTable: packed-int lookup over a two-longs-per-slot table (the
paper's C layout) — one cache line per lookup instead of three parallel-array
reads; empty slots self-answer no-match with no occupancy flag or sentinel.
ShortCodeTable/Matcher: pre-packed no-match slots, one array read per fallback.
FsstEncodingEncoder: all rows compress into one shared scratch (was two heap
allocations plus an extra copy per row), one-pass wire-code remap.
FsstEncodingDecoder: prefix-sum output offsets from the lengths child and
256-row batched decompression (per-row switch/modulo removed; whole-chunk
single call measured 1.8x slower via OSR-compiled mega-loop), plus new
malformed-input guards (empty children, invalid offsets, row-count and
decoded-length overflow, decoded-vs-claimed mismatch) with negative tests.

JavaVsJniFsstBenchmark -f 3: encode 1.848 -> 3.059 +- 0.430 ops/s (jni 2.854,
parity), decode 27.137 -> 32.924 +- 0.824 ops/s (jni 28.353, 1.16x faster).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: dfa1

CHANGELOG.md

@@ -18,6 +18,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
   Measured with `JavaVsJniFsstBenchmark` ([1003a673](https://github.com/dfa1/vortex-java/commit/1003a673)), unmodified before and after.
 
+- `vortex.fsst` hot paths close the remaining `vortex-jni` gap on both sides: single 8-byte word loads and a one-cache-line packed hash-slot layout in the compressor, one shared scratch buffer in the encoder (was two heap allocations per row), and prefix-sum output offsets plus 256-row batched decompression in the decoder (was a per-row offset `switch` and two broadcast modulos per row). ([#300](https://github.com/dfa1/vortex-java/pull/300))
+
+  | Benchmark | Before | After | `vortex-java` vs `vortex-jni` |
+  |---|---|---|---|
+  | `javaFsstEncode` | 1.848 ops/s | 3.059 ± 0.430 ops/s | 1.6x slower → parity (1.07x, overlapping error) |
+  | `javaFsstDecode` | 27.137 ops/s | 32.924 ± 0.824 ops/s | 1.3x slower → 1.16x faster |
+
+  Measured with `JavaVsJniFsstBenchmark` `-f 3` (jni: encode 2.854 ± 0.289, decode 28.353 ± 5.283 ops/s on the same machine and run).
+
 ### Fixed
 
 - `ParquetImporter` detects duplicate column names in the source Parquet schema and throws a clear message naming the duplicate(s) and the source file, instead of a confusing `VortexWriter` internal-invariant error several frames removed from the actual cause. ([f2c05e57](https://github.com/dfa1/vortex-java/commit/f2c05e57))

TODO.md

@@ -15,13 +15,10 @@
 
 ### FSST follow-ups
 
-Out of scope for the #287 rewrite (which closed most of the `vortex-jni` gap with a scalar,
-branch-free algorithm — see [ADR-0022](adr/0022-fsst-module-extraction.md)):
+Out of scope for the #287 rewrite (which, together with the follow-up hot-path pass, closed the
+`vortex-jni` gap with a scalar, branch-free algorithm — encode at parity, decode ~1.16x faster on
+`JavaVsJniFsstBenchmark` — see [ADR-0022](adr/0022-fsst-module-extraction.md)):
 
-- [ ] **AVX512/SIMD compression kernel** — the paper measures a SIMD kernel as the fastest known
-  string compressor at that tier, but this project's scalar rewrite already narrowed the encode gap
-  to `vortex-jni` from 36x to 1.6x. Needs a Vector-API/JDK-incubator decision and its own ADR (cf.
-  [ADR-0005](adr/0005-vector-api-adoption.md)).
 - [ ] **True per-row lazy/random-access decompression** exploiting FSST's headline random-access
   property — today `FsstEncodingDecoder.decode()` eagerly materializes the whole column up front
   regardless of what is queried. Connects to [ADR-0010](adr/0010-lazy-decode.md) (Lazy decode) but

fsst/src/main/java/io/github/dfa1/vortex/fsst/Compressor.java

@@ -2,6 +2,9 @@
 
 import java.lang.foreign.MemorySegment;
 import java.lang.foreign.ValueLayout;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.VarHandle;
+import java.nio.ByteOrder;
 import java.util.ArrayList;
 import java.util.List;
 
@@ -23,7 +26,13 @@ public final class Compressor {
     /// `core`'s `VortexFormat.LE_LONG` so this module keeps its zero dependency on `core`; the FSST
     /// wire format packs symbol bytes LSB-first, so all `MemorySegment` reads/writes are little-endian.
     static final ValueLayout.OfLong LE_LONG =
-            ValueLayout.JAVA_LONG_UNALIGNED.withOrder(java.nio.ByteOrder.LITTLE_ENDIAN);
+            ValueLayout.JAVA_LONG_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
+
+    /// Little-endian unaligned `long` view over a `byte[]`, the JIT-intrinsified way to load an
+    /// 8-byte input word in one instruction on the `byte[]` compress fast path (the byte-by-byte
+    /// [#loadWord(byte[], int, int)] assembly is 8 loads + shifts and is kept for the tail only).
+    private static final VarHandle LONG_LE_BYTES =
+            MethodHandles.byteArrayViewVarHandle(long[].class, ByteOrder.LITTLE_ENDIAN);
 
     private final List<Symbol> symbolsByGainDescending;
     private final Matcher matcher;
@@ -142,8 +151,27 @@ public Decompressor toDecompressor() {
     public long compress(byte[] input, int start, int end, byte[] out, long outPos) {
         int pos = start;
         int outIndex = (int) outPos;
+        // Fast path: while 8 real input bytes remain, load the word in one intrinsified read. No
+        // over-long-match guard is needed here — a match is at most 8 bytes, so it can never reach
+        // past end while pos <= end - 8.
+        int fastEnd = end - 8;
+        while (pos <= fastEnd) {
+            long word = (long) LONG_LE_BYTES.get(input, pos);
+            int packedMatch = matcher.longestMatch(word);
+            int length = Matcher.lengthOf(packedMatch);
+            if (length > 0) {
+                out[outIndex++] = (byte) Matcher.codeOf(packedMatch);
+                pos += length;
+            } else {
+                // The escaped literal is the word's low byte — already loaded, no re-read.
+                out[outIndex++] = (byte) ESCAPE;
+                out[outIndex++] = (byte) word;
+                pos++;
+            }
+        }
         while (pos < end) {
-            int packedMatch = matcher.longestMatch(loadWord(input, pos, end));
+            long word = loadWord(input, pos, end);
+            int packedMatch = matcher.longestMatch(word);
             int length = Matcher.lengthOf(packedMatch);
             // Reject an over-long match: loadWord zero-pads bytes past end, and the branch-free
             // Matcher has no notion of end, so a symbol whose trailing bytes are zero can spuriously
@@ -156,7 +184,7 @@ public long compress(byte[] input, int start, int end, byte[] out, long outPos)
                 pos += length;
             } else {
                 out[outIndex++] = (byte) ESCAPE;
-                out[outIndex++] = input[pos];
+                out[outIndex++] = (byte) word;
                 pos++;
             }
         }
@@ -187,8 +215,27 @@ public long compress(byte[] input, int start, int end, byte[] out, long outPos)
     public long compress(MemorySegment input, long start, long end, MemorySegment out, long outPos) {
         long pos = start;
         long outIndex = outPos;
+        // Fast path: while 8 readable bytes remain (bounded by both end and the segment's own
+        // size — the segment may end exactly at end with no trailing slack), load the word in one
+        // unaligned read. No over-long-match guard is needed here — a match is at most 8 bytes.
+        long fastEnd = Math.min(end, input.byteSize()) - 8;
+        while (pos <= fastEnd) {
+            long word = input.get(LE_LONG, pos);
+            int packedMatch = matcher.longestMatch(word);
+            int length = Matcher.lengthOf(packedMatch);
+            if (length > 0) {
+                out.set(ValueLayout.JAVA_BYTE, outIndex++, (byte) Matcher.codeOf(packedMatch));
+                pos += length;
+            } else {
+                // The escaped literal is the word's low byte — already loaded, no re-read.
+                out.set(ValueLayout.JAVA_BYTE, outIndex++, (byte) ESCAPE);
+                out.set(ValueLayout.JAVA_BYTE, outIndex++, (byte) word);
+                pos++;
+            }
+        }
         while (pos < end) {
-            int packedMatch = matcher.longestMatch(loadWord(input, pos, end));
+            long word = loadWord(input, pos, end);
+            int packedMatch = matcher.longestMatch(word);
             int length = Matcher.lengthOf(packedMatch);
             // Same boundary guard as the byte[] overload: reject a match that reaches past the real
             // input end, since loadWord zero-pads and the branch-free Matcher has no notion of end.
@@ -197,7 +244,7 @@ public long compress(MemorySegment input, long start, long end, MemorySegment ou
                 pos += length;
             } else {
                 out.set(ValueLayout.JAVA_BYTE, outIndex++, (byte) ESCAPE);
-                out.set(ValueLayout.JAVA_BYTE, outIndex++, input.get(ValueLayout.JAVA_BYTE, pos));
+                out.set(ValueLayout.JAVA_BYTE, outIndex++, (byte) word);
                 pos++;
             }
         }

fsst/src/main/java/io/github/dfa1/vortex/fsst/LossyPerfectHashTable.java

@@ -35,30 +35,21 @@ final class LossyPerfectHashTable {
     /// Low three bytes of the input word — the only bytes the hash keys on.
     private static final long PREFIX_MASK = 0x00FF_FFFFL;
 
-    /// Per-slot packed symbol bytes, LSB-first ([Symbol] convention). Meaningful only where the
-    /// corresponding [#occupied] entry is set.
-    private final long[] packedBytes;
-
-    /// Per-slot mask `~0L >>> ignoredBits` (`ignoredBits = 64 - 8 * length`) applied to an input
-    /// word before comparing, clearing the high bytes past the candidate's length.
-    private final long[] keepMask;
-
-    /// Per-slot symbol length in bytes, 3-8.
-    private final int[] lengths;
-
-    /// Per-slot code (the symbol's list index), meaningful only where [#occupied] is set.
-    private final int[] codes;
-
-    /// Whether each slot holds a candidate.
-    private final boolean[] occupied;
-
-    private LossyPerfectHashTable(long[] packedBytes, long[] keepMask, int[] lengths, int[] codes,
-                                  boolean[] occupied) {
-        this.packedBytes = packedBytes;
-        this.keepMask = keepMask;
-        this.lengths = lengths;
-        this.codes = codes;
-        this.occupied = occupied;
+    /// Slot table with two adjacent longs per slot (the FSST paper's C layout): `slots[2 * s]` is
+    /// the candidate's packed symbol bytes (LSB-first, [Symbol] convention) and `slots[2 * s + 1]`
+    /// is its metadata `ignoredBits << 16 | code << 8 | length` (`ignoredBits = 64 - 8 * length`).
+    /// A lookup derives the keep-mask from the ignored-bits with one shift instead of loading a
+    /// separate mask array, so the whole 16-byte slot lands in a single cache line — one memory
+    /// touch per lookup instead of three parallel-array reads across three lines.
+    ///
+    /// Empty slots are all-zero: metadata 0 makes the keep-mask `~0L` and the symbol 0, so an
+    /// input word of exactly 0 can "hit" an empty slot — harmlessly, because the returned low 16
+    /// bits (`code << 8 | length`) are then 0, which is precisely the "no match" answer. No
+    /// occupancy flag or sentinel is needed.
+    private final long[] slots;
+
+    private LossyPerfectHashTable(long[] slots) {
+        this.slots = slots;
     }
 
     /// Builds the table from the trained symbols in descending-gain order, keeping only those of
@@ -76,43 +67,41 @@ private LossyPerfectHashTable(long[] packedBytes, long[] keepMask, int[] lengths
     /// @param symbolsByGainDescending the trained symbols, code = list index, gain-descending
     /// @return a hash table resolving 3-8 byte matches with first-writer-wins on collision
     static LossyPerfectHashTable of(List<Symbol> symbolsByGainDescending) {
-        long[] packedBytes = new long[SLOTS];
-        long[] keepMask = new long[SLOTS];
-        int[] lengths = new int[SLOTS];
-        int[] codes = new int[SLOTS];
-        boolean[] occupied = new boolean[SLOTS];
+        long[] slots = new long[2 * SLOTS];
         for (int code = 0; code < symbolsByGainDescending.size(); code++) {
             Symbol symbol = symbolsByGainDescending.get(code);
             if (symbol.length() < 3) {
                 continue; // Length 1-2 belongs to ShortCodeTable.
             }
             int slot = slotFor(symbol.packedBytes());
-            if (occupied[slot]) {
+            if (slots[2 * slot + 1] != 0) {
                 continue; // First writer (higher gain) wins; skip the collision.
             }
-            occupied[slot] = true;
-            packedBytes[slot] = symbol.packedBytes();
-            keepMask[slot] = keepMaskFor(symbol.length());
-            lengths[slot] = symbol.length();
-            codes[slot] = code;
+            long ignoredBits = 64L - 8 * symbol.length();
+            slots[2 * slot] = symbol.packedBytes();
+            slots[2 * slot + 1] = ignoredBits << 16 | (long) code << 8 | symbol.length();
         }
-        return new LossyPerfectHashTable(packedBytes, keepMask, lengths, codes, occupied);
+        return new LossyPerfectHashTable(slots);
     }
 
-    /// Looks up `word` and reports whether a stored 3-8 byte symbol really matches it.
+    /// Looks up `word` and returns the matched symbol as `code << 8 | length`, or 0 when no stored
+    /// 3-8 byte symbol matches.
     ///
     /// The input word's first three bytes select one slot; the candidate there matches only if the
-    /// masked compare passes (`(word & keepMask) == candidate.packedBytes`), which rules out both
-    /// hash collisions with unrelated bytes and empty slots. On a miss the returned [HashMatch] has
-    /// `hit == false` and the caller falls back to [ShortCodeTable].
+    /// masked compare passes (`(word & (~0L >>> ignoredBits)) == candidate.packedBytes`), which
+    /// rules out hash collisions with unrelated bytes. An empty slot can only "hit" an all-zero
+    /// input word, and then still answers 0 (its metadata is 0), which is the no-match result. A
+    /// stored symbol's length is 3-8, so a real hit is never 0 and the caller can test the result
+    /// directly. On a miss the caller falls back to [ShortCodeTable].
     ///
     /// @param word an 8-byte little-endian input word starting at the current match position, with
     ///             any bytes past the remaining input already zero-padded by the caller
-    /// @return the match result: `hit`, and when hit the matched `code` and `length`
-    HashMatch lookup(long word) {
-        int slot = slotFor(word);
-        boolean hit = occupied[slot] && (word & keepMask[slot]) == packedBytes[slot];
-        return new HashMatch(hit, codes[slot], lengths[slot]);
+    /// @return the match as `code << 8 | length`, or 0 when there is no match
+    int lookup(long word) {
+        int slot = slotFor(word) << 1;
+        long symbol = slots[slot];
+        long meta = slots[slot + 1];
+        return (word & (~0L >>> (int) (meta >>> 16))) == symbol ? (int) (meta & 0xFFFF) : 0;
     }
 
     /// Computes the slot index a word hashes to, keyed on its first three bytes. Package-visible so
@@ -126,18 +115,4 @@ static int slotFor(long word) {
         return (int) (mixed >>> 32) & SLOT_MASK;
     }
 
-    private static long keepMaskFor(int length) {
-        int ignoredBits = 64 - 8 * length;
-        return ~0L >>> ignoredBits;
-    }
-
-    /// Result of a [LossyPerfectHashTable#lookup(long)]: whether a 3-8 byte symbol matched and, if
-    /// so, its code and length. When `hit` is false, `code` and `length` are unspecified and the
-    /// caller must ignore them.
-    ///
-    /// @param hit whether a stored symbol really matched the input word
-    /// @param code the matched symbol code, valid only when `hit`
-    /// @param length the matched symbol length in bytes (3-8), valid only when `hit`
-    record HashMatch(boolean hit, int code, int length) {
-    }
 }

fsst/src/main/java/io/github/dfa1/vortex/fsst/Matcher.java

@@ -55,11 +55,13 @@ public static Matcher of(List<Symbol> symbolsByGainDescending) {
     ///             any bytes past the remaining input already zero-padded by the caller
     /// @return the longest match as `code << 8 | length`; length 0 signals "no match, escape"
     public int longestMatch(long word) {
-        LossyPerfectHashTable.HashMatch hashMatch = hashTable.lookup(word);
-        if (hashMatch.hit()) {
-            return hashMatch.code() << 8 | hashMatch.length();
-        }
-        return shortCodes.codeFor(word) << 8 | shortCodes.lengthFor(word);
+        // Both tables are read unconditionally so the select below has no side to skip — the JIT
+        // can lower it to a conditional move instead of a data-dependent branch, which matters
+        // because hash hit/miss alternates unpredictably across input positions. The extra
+        // short-code read on a hash hit is one L1 load into a 256 KB table.
+        int hashMatch = hashTable.lookup(word);
+        int shortMatch = shortCodes.packedFor(word);
+        return hashMatch != 0 ? hashMatch : shortMatch;
     }
 
     /// Extracts the code from a packed [#longestMatch(long)] result.

fsst/src/main/java/io/github/dfa1/vortex/fsst/ShortCodeTable.java

@@ -1,5 +1,6 @@
 package io.github.dfa1.vortex.fsst;
 
+import java.util.Arrays;
 import java.util.List;
 
 /// Direct-indexed table resolving the shortest FSST matches — length 0 (no match), 1, or 2 —
@@ -25,7 +26,12 @@ final class ShortCodeTable {
     /// codes are `0..254` (`0xFF` is the escape), so `-1` can never collide with a real code.
     static final int NO_CODE = -1;
 
-    /// Packed `code << 8 | length` per 16-bit key. A zero length marks "no match".
+    /// Packed no-match value stored in unpopulated slots: `NO_CODE << 8 | 0`. Storing the sentinel
+    /// pre-packed lets [#packedFor(long)] return the slot verbatim — one array read, no branch —
+    /// and an arithmetic `>> 8` recovers [#NO_CODE] while `& 0xFF` recovers length 0.
+    private static final int NO_MATCH = NO_CODE << 8;
+
+    /// Packed `code << 8 | length` per 16-bit key. A zero length marks "no match" ([#NO_MATCH]).
     private final int[] slots;
 
     private ShortCodeTable(int[] slots) {
@@ -46,6 +52,7 @@ private ShortCodeTable(int[] slots) {
     /// @return a table resolving 0/1/2-byte matches for any two-byte input prefix
     static ShortCodeTable of(List<Symbol> symbolsByGainDescending) {
         int[] slots = new int[SLOTS];
+        Arrays.fill(slots, NO_MATCH);
         for (int code = 0; code < symbolsByGainDescending.size(); code++) {
             Symbol symbol = symbolsByGainDescending.get(code);
             if (symbol.length() == 1) {
@@ -69,23 +76,32 @@ static ShortCodeTable of(List<Symbol> symbolsByGainDescending) {
         return new ShortCodeTable(slots);
     }
 
-    /// Returns the code matched by the low two bytes of `word`, or [#NO_CODE] if none. One array
-    /// read, no branching.
+    /// Returns the match for the low two bytes of `word` as `code << 8 | length`, or
+    /// `NO_CODE << 8` (length 0) when there is no length-1 or length-2 match. One array read, no
+    /// branching — the slot value is returned verbatim, which is what makes this the hot path's
+    /// fallback of choice.
+    ///
+    /// @param word an input word; only its low 16 bits (first two input bytes) are consulted
+    /// @return the match as `code << 8 | length`; length 0 (and code [#NO_CODE]) means no match
+    int packedFor(long word) {
+        return slots[(int) (word & 0xFFFF)];
+    }
+
+    /// Returns the code matched by the low two bytes of `word`, or [#NO_CODE] if none.
     ///
     /// @param word an input word; only its low 16 bits (first two input bytes) are consulted
     /// @return the matched symbol code, or [#NO_CODE] when there is no length-1 or length-2 match
     int codeFor(long word) {
-        int packed = slots[(int) (word & 0xFFFF)];
-        return packed == 0 ? NO_CODE : packed >>> 8;
+        return packedFor(word) >> 8;
     }
 
     /// Returns the length of the symbol matched by the low two bytes of `word`: 2, 1, or 0 for no
-    /// match. One array read, no branching.
+    /// match.
     ///
     /// @param word an input word; only its low 16 bits (first two input bytes) are consulted
     /// @return the matched symbol length in bytes, or 0 when there is no match
     int lengthFor(long word) {
-        return length(slots[(int) (word & 0xFFFF)]);
+        return length(packedFor(word));
     }
 
     private static int length(int packed) {