Skip to content

Reuse manifests within each data source - #58

Merged
platypii merged 1 commit into
masterfrom
reuse-source-manifests
Oct 7, 2026
Merged

platypii merged 1 commit into
masterfrom
reuse-source-manifests

Conversation

@platypii

@platypii platypii commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Repeated queries on one Iceberg data source currently fetch and decode the same data manifests on every scan. Retain each manifest's promise within that source's lifetime, preserving lazy partition pruning and sharing in-flight reads across concurrent scans. Failed reads are removed so later scans can retry; separate sources retain independent snapshot state.

Add tests for lazy loading, repeated and concurrent scans, source isolation after appends, and retries. Also cover equality deletes across data/delete manifest merges, schema renames, manifest rewrites, and reads of the original snapshot.

Validation:

  • All 943 tests pass; lint and TypeScript checks pass.
  • On a local HypAware table, repeated queries now read 26 manifests on the first scan and zero on subsequent scans.
  • Release review compared eight local HypAware tables covering approximately 1.2 million rows and 128 query comparisons across unchanged, rewritten, and merged manifests, with no result differences. Original tables were not modified.

Loaded manifest entries remain in memory while their data source is reachable; there is no shared cache across source instances.

@platypii
platypii merged commit 474fa02 into master Oct 7, 2026
6 checks passed
@platypii
platypii deleted the reuse-source-manifests branch October 7, 2026 21:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant