Repository navigation
Speed up CSV parsing: windowed cell scans, batched sequences, compiled row builders - #27
Merged
Merged
Conversation
benchmarks/udsv/sweep.ts drives uDSV's bench/runone.cjs unmodified over spliterator and the parsers it ships adapters for, measures peak RSS through GNU time, and writes results.json; render.ts prints Markdown tables with throughput and memory columns. The README section is generated from it and kept honest, losses included. Claude-Session: https://claude.ai/code/session_01MonKZmMeL1Jvo7i6EHmbYx
…wer allocations - AsyncSequence.next() only awaits the opener on the first pull; the per-pull await was an async frame per row on every streamed format (20-40%). - Batched sources: an upstream marked BATCHED_SOURCE yields arrays walked without an await per element. - AsyncSpliterator.nextWindow() hands out every queued record as one contiguous view, and the streaming CSV path runs the cell kernel over each window, falling back per window on invalid UTF-8. The streaming branch awaits the scanner so a fresh process takes it on its first file. - A string source is the decoded text: no second copy of the source, cells reference the caller's string. - The kernel emits CELL_FLAG_HAS_ESCAPE when a quote opens directly after a quote, carried across windows; the cell loop slices a quoted cell once, inside the quotes, and only unescapes on the flag. The row-path walk unquotes as it splits. - Rows are filled into exact-width template copies instead of grown by push. With rows retained the path was scavenger-bound: 23MB allocated per 2MB parse against uDSV's 8.5MB. Measured: 300MB quoted stream 4.2s -> 2.1s, unquoted 3.2s -> 1.9s, 386MB in-memory string 4.2s -> 3.2s at 2237 -> 1321 MiB peak, litmus in-memory 103 -> 209 MiB/s on uDSV's harness. Claude-Session: https://claude.ai/code/session_01MonKZmMeL1Jvo7i6EHmbYx
Replace the zipSync iterator with an indexed loop when both arguments are arrays, then compile a per-header literal builder with new Function and cache it on the bound header array. Object mode 62ms -> 10ms on uDSV's 10k x 20 litmus fixture, the same as array mode; entries 38ms -> 17ms. A Function constructor that throws under CSP leaves the loop in place. bindTransformers reads own properties only; a header named __proto__ used to bind Object.prototype as its transformer. Claude-Session: https://claude.ai/code/session_01MonKZmMeL1Jvo7i6EHmbYx
CellRowScanner.nextBatch(out) pushes one kernel batch of rows into the caller's array. The in-memory parser yields a fresh array per batch and walks it in its own generator, so a row crosses one generator rather than two; the streaming path collects a whole engine window into one array with no flattening copy. litmus in-memory 9.7ms -> 8.3ms, 300MB quoted stream 2.1s -> 1.9s. Claude-Session: https://claude.ai/code/session_01MonKZmMeL1Jvo7i6EHmbYx
TextSpliterator.fromAsync and JSONSpliterator.fromAsync hand a streamed engine to windowedTextRows, which decodes each window once, splits the text on the delimiter (crlf and skipEmpty applied there), and yields the rows as a batched source. One decode and one await per high-water mark of rows instead of one of each per row. drop and take become sequence ops on these paths. A fatal decode failure falls back to per-row decoding so the error names the row; a text comment filter mirrors the byte one. 600k JSONL rows: 1289ms -> 707ms, level with readline + JSON.parse (702ms); a 500MB corpus file 1948ms against readline's 2092ms. Claude-Session: https://claude.ai/code/session_01MonKZmMeL1Jvo7i6EHmbYx
CharacterSequence.scanCells returns the result block as a window-relative view with a unitBase instead of copying it out and rebasing in a JS loop. CellRowScanner.nextBatch turns the whole batch into rows before returning and runs no user code, so nothing can scan in between. 386MB streamed into objects 1869ms -> 1685ms, litmus in-memory 8.3ms -> 7.4ms, 300MB quoted stream 1.9s -> 1.8s. Claude-Session: https://claude.ai/code/session_01MonKZmMeL1Jvo7i6EHmbYx
jiti, which Docusaurus loads its plugins through, transpiles an ESM dependency to CommonJS on the fly: it rewrites import.meta.url and leaves import.meta.resolve in place, and Node then refuses the file with "Cannot use 'import.meta' outside a module". 9.1.0 shipped that form in lib/internal/node-modules.js and broke a consumer's docs build. workerEntryUrl now resolves through createRequire(import.meta.url), which honours the # import map the same way, with the built-ins loaded through process.getBuiltinModule so the root keeps no static node: import. A test over the compiled output refuses any import.meta form but .url under out/lib and out/node. Claude-Session: https://claude.ai/code/session_01MonKZmMeL1Jvo7i6EHmbYx
…tor.from A string is already text, so encoding it, scanning the bytes and decoding each row back was pure overhead: 7.8us per call on a 40-line string against 1us for split. textRows splits the string on the delimiter text with the engine's crlf, skipEmpty, drop and take semantics, 2.5us per call. Gated by canSplitAsText: no byte position, no quote handling, UTF-8 only. The streaming window path shares the splitter. Parity with the byte route is pinned across delimiters and options. Claude-Session: https://claude.ai/code/session_01MonKZmMeL1Jvo7i6EHmbYx
The count adapters full-parse because uDSV's do; the library's scan-only path was never shown. Two new adapters, labeled as their own rows so the full-parse counts stay for apples-to-apples: - spliterator (stream, row count): CSVSpliterator.countAsync, counting engine rows without decoding columns. - spliterator (stream, row count, N workers): AsyncSpliterator.segments plus one persistent worker per core, each counting rows in its own range with countAsync's semantics and posting a single number back. Verified exact agreement with the sequential parser and expected.json on all three datasets. Segments are quote-blind, so embedded newlines in quoted fields would miscount in parallel; demonstrated on a synthetic file and documented in the adapter, README, and HANDOFF. 715/1091/393 MiB/s sequential and 669/2015/721 parallel against uDSV's 249/310/271 on litmus, openpayments, and gb-tuples. README tables and results.json regenerated from one fresh sweep; HANDOFF.md section 3.14 covers the work.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A performance arc over CSV (and JSONL/text) parsing, benchmarked against uDSV's own harness, plus the benchmark integration itself. Full narrative in HANDOFF.md (sections 3.1–3.14); all measurements reproducible via
benchmarks/udsv/sweep.ts.BATCHED_SOURCE): rows cross the sequence boundary in bulk.AsyncSpliterator.nextWindow()+ the WASM cell kernel on the streaming CSV path.new Function, CSP-safe fallback): object mode is now free relative to array mode (62.5 → 10.0 ms on litmus).readline+JSON.parse(707 vs 702 ms on 600k rows); 500MB corpus 1948 vs 2092 ms.TextSpliterator.from(string)7.8µs → 2.5µs on a 40-line string.Test plan
yarn compile && yarn test --run && yarn lintgreen (835 tests, 53 files)countAsyncand uDSVexpected.jsonon all datasets, plus 7 edge cases (empty file, CRLF, unterminated final row, escaped quotes)