Skip to content

Stream embedded ZIP entries directly into the parser #166

Description

@drauf

Problem

Embedded ZIP import still creates a complete File for every selected ZIP entry before parsing starts.

For large support ZIPs, this creates avoidable memory pressure:

  • fflate collects decompressed chunks for an entry
  • Watson combines them into a File
  • the parser later reads that File again through readFileLines
  • all selected files can exist before parsing completes

The current incremental import work reduces base64 lifetime and starts parser work earlier, but it still materializes each selected entry as a File.

Goal

Allow embedded ZIP entry content to flow directly into the parser in bounded chunks.

Keep manual drag-and-drop upload support unchanged for users.

Scope

  • Define a source-agnostic parser input contract:
    • entry name
    • entry size when known
    • async byte or line stream
  • Keep File as the browser upload boundary type
  • Adapt manual uploads from File.stream() to the shared parser input contract
  • Adapt fflate ZIP entry chunks to the same contract
  • Update readFileLines and the worker protocol to consume streamed entry content
  • Preserve parser ordering between thread dumps and CPU usage files
  • Apply backpressure so ZIP extraction does not outrun parser consumption
  • Release entry buffers once parsing of that entry completes
  • Keep unknown file totals explicit in progress reporting

Out of scope

  • Supporting additional archive formats
  • Changing the manual upload UI
  • Replacing the existing parser or thread-dump model
  • Adding a new benchmark framework

Success criteria

  • Embedded ZIP import does not create a File for every selected entry
  • Memory use remains bounded while parsing large embedded archives
  • Manual uploads continue to work through the shared input contract
  • Thread-dump and CPU usage pairing remains correct
  • Invalid ZIP input and parser errors remain user-visible
  • Add unit tests for streamed entry boundaries and worker backpressure
  • Compare manual-upload and embedded-import timing before and after the change

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions