Skip to content

Add raw Icechunk format inspection (object, chunk-ref, chunk-changes, web viewer) - #5

Merged
ianhi merged 9 commits into
mainfrom
raw-inspection
Oct 6, 2026
Merged

ianhi merged 9 commits into
mainfrom
raw-inspection

Conversation

@ianhi

@ianhi ianhi commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator

Adds a way to answer format-level questions about an Icechunk repo by reading its metadata files (repo info, snapshots, manifests, transaction logs) field by field, from the CLI, MCP, or a browser. Motivating questions from a user:

  1. Which chunks changed between two versions?
  2. How does a manifest record a chunk deletion?
  3. What checksum does a virtual ref store for its target object?

What's new

Command What it does
core-drill <repo> object <target> Decode one metadata file. IDs that name other files print as → manifests/<id> links; node IDs get their paths; compressed_location is decompressed; --at arrays/0/refs/100..150 selects a subtree or window.
core-drill <repo> chunk-ref <array> <coords> Find the manifest entry for one chunk: native chunk ID, inline bytes, or virtual location with its ETag / last-modified checksum.
core-drill <repo> chunk-changes <commit> Classify each coordinate in a commit's transaction log as added, overwritten, rewritten, deleted, or absent, by looking it up in the parent's and the commit's manifests.
core-drill <repo> web Serve the viewer from the binary on 127.0.0.1 and open it on this repo. Repo objects go through core-drill's storage connection, so S3, GCS, and Arraylake repos work without CORS.

All three lookups are also MCP tools (object, chunk_ref, chunk_changes) and support --output json.

$ core-drill ./repo chunk-changes MNB5ZD7ZTTZT01TTXV6G
## /native — 1 coordinate listed
- (1, 1) **deleted**
  - before: native chunk `P7TN6M0SRMNJMNXGXJ80`, 4318 bytes at offset 0 — in `manifests/NC0XXACBQRMJMMHKFRF0` at `arrays/0/refs/3`

Classify changes on a deletion commit

Find chunk on a virtual array, showing the stored ETag

How it works

  • Schema-driven decoding. schema/*.bfbs is compiled from the Icechunk .fbs files (scripts/gen-schema.sh). The Rust walker (src/raw/) and the JS viewer (web/decoder.js) both read every field by reflection, so a schema change only needs a regenerated .bfbs.
  • Untrusted input. Reads are bounds-checked, and verification goes through icechunk's own constructors. A corruption test truncates and flips every byte of each file type. flatbuffers-reflection's verifier is not used because it panics on out-of-range union tags. FlexBuffers decoding is bounded and panic-contained, and sanitize now also strips bidi control characters.
  • The viewer is self-contained: no build step, the zstd wasm decoder is vendored (web/vendor/zstd.js), and there are no CDN requests. web/test/decode.test.mjs checks it field for field against core-drill --output json.

Also in this PR

  • storage-size command and MCP tool: deduplicated on-disk bytes across history.
  • Output subcommands default to markdown instead of launching the TUI.
  • Upgrade to icechunk 2.2.2 and arraylake 0.14, which fixes opening Arraylake marketplace subscription repos ("Bucket info is missing"). arraylake 0.14 resolves a subscription to its parent repo's bucket.
  • cargo fmt over the repo (separate commit).
  • --locked in the from-source install command.

Tested with:

  • 37 Rust tests, including end-to-end lifecycle and corruption tests on an in-memory repo.
  • 133 parity checks between the web decoder and the CLI across four local test repos (node web/test/decode.test.mjs).
  • Manual runs against s3://icechunk-public-data/v1/era5_weatherbench2, al:iansorg/icechunk-format-examples (generated by scripts/create_chunk_lifecycle_repo.py --arraylake …), and a marketplace subscription repo.

Docs: DOCS/raw-inspection.md, web/README.md.

ianhi added 9 commits October 5, 2026 17:28
Lists chunks/, manifests/, and snapshots/ concurrently and reports bytes
and object counts per category. CLI-only (plus MCP) because a full listing
is too expensive to run on every TUI open.

Output subcommands (info, branches, tags, log, tree, ops-log,
storage-size) now default to markdown when --output is omitted instead
of launching the TUI.
`object` decodes repo info, snapshot, manifest, and transaction log files
field by field. The walker is driven by binary schemas compiled from the
Icechunk .fbs files (schema/*.bfbs, regenerated by scripts/gen-schema.sh),
reads are bounds-checked, and verification goes through icechunk's own
constructors. IDs are shown with the files they point at, node IDs with
their paths, and compressed virtual locations decompressed. `--at`
selects a subtree or a window of a long vector.

`chunk-ref` finds the manifest entry for one chunk (including virtual
ref checksums). `chunk-changes` classifies each coordinate a commit's
transaction log lists as added, overwritten, rewritten, deleted, or
absent by looking it up in the parent's and the commit's manifests.

All three are CLI commands with markdown and JSON output and MCP tools.
Also: sanitize strips bidi control characters, fetch.rs and
output/format.rs move into module directories, and
scripts/create_chunk_lifecycle_repo.py builds an example repo locally or
on Arraylake. See DOCS/raw-inspection.md.
web/index.html decodes the same files in the browser with no build
step: open a local folder, a URL or s3:// prefix, or a single file, and
click through repo -> snapshot -> manifest / transaction log. Includes
Find chunk and Classify changes. decoder.js mirrors src/raw and
web/test/decode.test.mjs checks it field for field against
`core-drill --output json`.
`core-drill <repo> web` serves the viewer from the binary on 127.0.0.1
and opens it with the repo loaded. Repo objects are fetched through
core-drill's own storage connection, so local, S3, GCS, and Arraylake
repos work without CORS setup. Requests with a foreign Host header are
rejected.

The zstd decoder is vendored as web/vendor/zstd.js (wasm inlined,
built by scripts/vendor-zstd.sh), so the page makes no CDN requests
and still works opened directly from disk.
Without it cargo resolves a newer icechunk that the arraylake crate
does not compile against.
Arraylake marketplace subscription repos have no bucket of their own;
arraylake 0.14 resolves them to the parent repo's storage
(`effective_storage`), which 0.2 did not, so opening one failed with
"Bucket info is missing". arraylake 0.14 requires icechunk >= 2.2.2.

Adapts to the new storage constructors (extra header and key-layout
arguments, S3Options builder) and shows the effective bucket in the
repo label.
The earth-mover org requires every action to be pinned to a full commit
SHA, so the dist plan job failed at setup. Pins are set through dist's
github-action-commits config so dist generate keeps them.
@ianhi
ianhi merged commit dde10ea into main Oct 6, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant