Problem
Embedded ZIP import still creates a complete File for every selected ZIP entry before parsing starts.
For large support ZIPs, this creates avoidable memory pressure:
- fflate collects decompressed chunks for an entry
- Watson combines them into a
File
- the parser later reads that
File again through readFileLines
- all selected files can exist before parsing completes
The current incremental import work reduces base64 lifetime and starts parser work earlier, but it still materializes each selected entry as a File.
Goal
Allow embedded ZIP entry content to flow directly into the parser in bounded chunks.
Keep manual drag-and-drop upload support unchanged for users.
Scope
- Define a source-agnostic parser input contract:
- entry name
- entry size when known
- async byte or line stream
- Keep
File as the browser upload boundary type
- Adapt manual uploads from
File.stream() to the shared parser input contract
- Adapt fflate ZIP entry chunks to the same contract
- Update
readFileLines and the worker protocol to consume streamed entry content
- Preserve parser ordering between thread dumps and CPU usage files
- Apply backpressure so ZIP extraction does not outrun parser consumption
- Release entry buffers once parsing of that entry completes
- Keep unknown file totals explicit in progress reporting
Out of scope
- Supporting additional archive formats
- Changing the manual upload UI
- Replacing the existing parser or thread-dump model
- Adding a new benchmark framework
Success criteria
- Embedded ZIP import does not create a
File for every selected entry
- Memory use remains bounded while parsing large embedded archives
- Manual uploads continue to work through the shared input contract
- Thread-dump and CPU usage pairing remains correct
- Invalid ZIP input and parser errors remain user-visible
- Add unit tests for streamed entry boundaries and worker backpressure
- Compare manual-upload and embedded-import timing before and after the change
Problem
Embedded ZIP import still creates a complete
Filefor every selected ZIP entry before parsing starts.For large support ZIPs, this creates avoidable memory pressure:
FileFileagain throughreadFileLinesThe current incremental import work reduces base64 lifetime and starts parser work earlier, but it still materializes each selected entry as a
File.Goal
Allow embedded ZIP entry content to flow directly into the parser in bounded chunks.
Keep manual drag-and-drop upload support unchanged for users.
Scope
Fileas the browser upload boundary typeFile.stream()to the shared parser input contractreadFileLinesand the worker protocol to consume streamed entry contentOut of scope
Success criteria
Filefor every selected entry