Repository navigation
Add raw Icechunk format inspection (object, chunk-ref, chunk-changes, web viewer) - #5
Merged
Merged
Conversation
Lists chunks/, manifests/, and snapshots/ concurrently and reports bytes and object counts per category. CLI-only (plus MCP) because a full listing is too expensive to run on every TUI open. Output subcommands (info, branches, tags, log, tree, ops-log, storage-size) now default to markdown when --output is omitted instead of launching the TUI.
`object` decodes repo info, snapshot, manifest, and transaction log files field by field. The walker is driven by binary schemas compiled from the Icechunk .fbs files (schema/*.bfbs, regenerated by scripts/gen-schema.sh), reads are bounds-checked, and verification goes through icechunk's own constructors. IDs are shown with the files they point at, node IDs with their paths, and compressed virtual locations decompressed. `--at` selects a subtree or a window of a long vector. `chunk-ref` finds the manifest entry for one chunk (including virtual ref checksums). `chunk-changes` classifies each coordinate a commit's transaction log lists as added, overwritten, rewritten, deleted, or absent by looking it up in the parent's and the commit's manifests. All three are CLI commands with markdown and JSON output and MCP tools. Also: sanitize strips bidi control characters, fetch.rs and output/format.rs move into module directories, and scripts/create_chunk_lifecycle_repo.py builds an example repo locally or on Arraylake. See DOCS/raw-inspection.md.
web/index.html decodes the same files in the browser with no build step: open a local folder, a URL or s3:// prefix, or a single file, and click through repo -> snapshot -> manifest / transaction log. Includes Find chunk and Classify changes. decoder.js mirrors src/raw and web/test/decode.test.mjs checks it field for field against `core-drill --output json`.
`core-drill <repo> web` serves the viewer from the binary on 127.0.0.1 and opens it with the repo loaded. Repo objects are fetched through core-drill's own storage connection, so local, S3, GCS, and Arraylake repos work without CORS setup. Requests with a foreign Host header are rejected. The zstd decoder is vendored as web/vendor/zstd.js (wasm inlined, built by scripts/vendor-zstd.sh), so the page makes no CDN requests and still works opened directly from disk.
Without it cargo resolves a newer icechunk that the arraylake crate does not compile against.
Arraylake marketplace subscription repos have no bucket of their own; arraylake 0.14 resolves them to the parent repo's storage (`effective_storage`), which 0.2 did not, so opening one failed with "Bucket info is missing". arraylake 0.14 requires icechunk >= 2.2.2. Adapts to the new storage constructors (extra header and key-layout arguments, S3Options builder) and shows the effective bucket in the repo label.
The earth-mover org requires every action to be pinned to a full commit SHA, so the dist plan job failed at setup. Pins are set through dist's github-action-commits config so dist generate keeps them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a way to answer format-level questions about an Icechunk repo by reading its metadata files (repo info, snapshots, manifests, transaction logs) field by field, from the CLI, MCP, or a browser. Motivating questions from a user:
What's new
core-drill <repo> object <target>→ manifests/<id>links; node IDs get their paths;compressed_locationis decompressed;--at arrays/0/refs/100..150selects a subtree or window.core-drill <repo> chunk-ref <array> <coords>core-drill <repo> chunk-changes <commit>core-drill <repo> webAll three lookups are also MCP tools (
object,chunk_ref,chunk_changes) and support--output json.How it works
schema/*.bfbsis compiled from the Icechunk.fbsfiles (scripts/gen-schema.sh). The Rust walker (src/raw/) and the JS viewer (web/decoder.js) both read every field by reflection, so a schema change only needs a regenerated.bfbs.flatbuffers-reflection's verifier is not used because it panics on out-of-range union tags. FlexBuffers decoding is bounded and panic-contained, andsanitizenow also strips bidi control characters.web/vendor/zstd.js), and there are no CDN requests.web/test/decode.test.mjschecks it field for field againstcore-drill --output json.Also in this PR
storage-sizecommand and MCP tool: deduplicated on-disk bytes across history.cargo fmtover the repo (separate commit).--lockedin the from-source install command.Tested with:
node web/test/decode.test.mjs).s3://icechunk-public-data/v1/era5_weatherbench2,al:iansorg/icechunk-format-examples(generated byscripts/create_chunk_lifecycle_repo.py --arraylake …), and a marketplace subscription repo.Docs:
DOCS/raw-inspection.md,web/README.md.