Compile mesh BVHs to a memory-mappable index file - #14
Open
lispandfound wants to merge 3 commits into
Open
lispandfound wants to merge 3 commits into
lispandfound wants to merge 3 commits into
Conversation
lispandfound
changed the title
lispandfound
force-pushed
the
index-format
branch
from
September 11, 2026 12:40
3a8db4f to
c085d43
Compare
Benchmark for ef12428Click to view benchmark
|
Benchmark for 1ab119fClick to view benchmark
|
lispandfound
force-pushed
the
index-format
branch
from
September 11, 2026 19:58
c085d43 to
35963ff
Compare
lispandfound
force-pushed
the
drop-numba
branch
from
September 11, 2026 19:58
ec3f1f6 to
20f9eb3
Compare
Benchmark for 524267bClick to view benchmark
|
Benchmark for 524267bClick to view benchmark
|
lispandfound
force-pushed
the
index-format
branch
from
September 20, 2026 21:32
35963ff to
7f2886e
Compare
lispandfound
force-pushed
the
drop-numba
branch
from
September 20, 2026 21:32
20f9eb3 to
1eee020
Compare
Benchmark for ca36781Click to view benchmark
|
Benchmark for ca36781Click to view benchmark
|
lispandfound
force-pushed
the
index-format
branch
from
September 20, 2026 21:36
7f2886e to
fbcdd13
Compare
lispandfound
force-pushed
the
drop-numba
branch
from
September 20, 2026 21:36
1eee020 to
d869355
Compare
Benchmark for f8b9a32Click to view benchmark
|
Benchmark for f8b9a32Click to view benchmark
|
lispandfound
force-pushed
the
index-format
branch
from
September 20, 2026 22:19
fbcdd13 to
f5bd379
Compare
lispandfound
force-pushed
the
drop-numba
branch
from
September 20, 2026 22:19
d869355 to
ac33c93
Compare
Benchmark for f97e9f4Click to view benchmark
|
Benchmark for f97e9f4Click to view benchmark
|
Every process that queried a mesh model read the mesh off disk and ran the
BVH build itself, then held the result on its heap. A country-scale mesh runs
to tens of millions of simplices, and each worker process paid the build and
kept its own copy. The tree also could not leave the process, since it had no
Python representation, so `--distributed` ran thread workers inside one
interpreter with a registry of live pointers in place of pickling.
`MeshModel` was already most of the way there. The `bvh` crate ran once in
`MeshModel::new`, `CompactBvh::from_bvh` packed the result into 56-byte nodes,
and queries walked plain slices. The work here is letting those slices come
from a file.
Every hot record (`CompactNode`, `Simplex`, `VertexRefs`, `Quality`) is now
`#[repr(C)]` and derives zerocopy's `FromBytes`, `IntoBytes`, `KnownLayout`
and `Immutable`, which pins the layout and lets a byte range become a `&[T]`
with size and alignment checked. `Simplex` and `ChildRef` swap their nalgebra
and `Aabb` fields for plain arrays so the derives can see them; `Matrix3::from`
on the array is a load the optimiser folds away. `Slab<T>` holds each array as
either `Owned(Vec<T>)` or `Mapped { mmap, offset, len }` and dereferences to
`&[T]`, so nothing downstream changed.
`src/index.rs` writes the four arrays behind a header page, each section on a
4 KiB boundary, and opens the file by mapping it. The header carries the
section offsets and counts, the `Real` width (a reader built with the other
width refuses the file), the model's metadata, and a 32-byte fingerprint of
the source mesh supplied by the caller. The writer renames a `.partial` file
into place so a crash leaves nothing that parses. `Mmap::map` is the one
`unsafe` call in the crate, and the file's immutability is what justifies it.
On the Python side, `MeshModel.from_path` maps `<mesh>.nzidx` when its
fingerprint matches the store and builds otherwise. The fingerprint hashes
the path, size and mtime of every file plus each `zarr.json`, so checking it
is a directory walk with no array reads. `nzcvm index build models/*.zarr`
compiles meshes, and the `models` and `synthetic` recipes run it.
`benchmarks/benchmark_index.py` at 2.8 million tetrahedra: build 1.1 s, open
7.5 ms, index 252 MB (91 B per tetrahedron), and a mapped tree answers
queries about 12% slower than a built one with the page cache warm, which is
the cost of file-backed pages missing transparent huge pages.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The design behind the compiled index, written down where the next person can find it: the records and why two of them changed shape, `Slab<T>`, the file layout, why the index is a flat file beside the zarr rather than inside it, the fingerprint and what it deliberately does not promise, and the benchmark that answers whether skipping the build is worth a second copy on disk. Cascade mounts NFS, which changes cold start and the failure modes. A page fault that misses the client cache is a round trip, and a file replaced under a live mapping ends the process with `SIGBUS`. The document sets out the `madvise` policy to add (populate on NFS, `WillNeed` elsewhere, never `Random`), the content-addressed naming that avoids `ESTALE`, and what to measure on Cascade before choosing between them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Findings from the review pass over the index-format diff: - Move the serialisation onto the types that own the records. `index.rs` keeps the header, a `SectionWriter` and a `SectionReader`; `CompactBvh`, `ModelMap` and `MeshModel` write and read their own sections, so the seven one-caller accessors and `from_parts` go. - Replace the positional offset/count arrays in the header with a tagged section table that records each section's stride. The reader looks sections up by tag and refuses one whose stride differs from its record type. - Fold the mixed model map's parallel `kinds` array into the refs: a constant simplex carries `u32::MAX` in its second slot. - Remove the dead `Queryable` trait and `enum_dispatch` on `Model`, the unused `ModelMap::is_mapped`, `Slab: From<Vec>`, and make `Simplex::c3` private. - `Slab::mapped` validates with one `ref_from_prefix_with_elems` and maps the cast error onto `SlabError`; the size assertion on `CompactNode` holds at both `Real` widths. - `PyMeshModel::model()` replaces seven copies of the consumed-model check. - `ChildRef::contains` goes through `Aabb::contains` again. - Python: `current_index()` is the one place that decides whether an index is usable, shared by `from_path`, `compile_index` and the CLI, which used a weaker check. The fingerprint walk uses `scandir` so a store costs one `stat` per file. Tests share one synthetic tomography writer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lispandfound
force-pushed
the
drop-numba
branch
from
September 20, 2026 22:28
ac33c93 to
bfb6c8f
Compare
lispandfound
force-pushed
the
index-format
branch
from
September 20, 2026 22:28
f5bd379 to
c75ddfc
Compare
Benchmark for 833baf2Click to view benchmark
|
Benchmark for 833baf2Click to view benchmark
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every process that queried a mesh model read the mesh off disk, ran the BVH
build itself, and kept the result on its heap. Two things about that stopped
scaling: the models are tens of millions of simplices and every worker held a
copy, and the tree had no Python representation so
--distributedcould onlyrun threads inside one interpreter with
registry.pystanding in forpickling.
This PR writes the build's output to a file whose layout is the in-memory
layout, and maps it back. A worker opens the file, and the pages a query
touches fault in and are shared through the page cache by every process on the
node. Nothing rebuilds and nothing copies.
What it took
MeshModelwas already most of the way there: thebvhcrate ran once, theresult was packed into 56-byte
CompactNodes, and queries walked plainslices. So:
CompactNode,Simplex,VertexRefsandQualityare#[repr(C)]with zerocopy derives.SimplexandChildRefswap nalgebra/
Aabbfields for plain arrays (the derives can't see throughforeign types;
Matrix3::fromon the array folds away).Real = f64gets anexplicit
_padsoChildRefhas no tail padding.Slab<T>:Owned(Vec<T>)orMapped { mmap, offset, len }, derefs to&[T].MeshModel's four arrays are slabs; nothing downstream changed.A mixed model map (constant and interpolating simplices in one mesh) marks a
constant simplex with
u32::MAXin its second ref slot rather than carryinga parallel
kindsarray.src/index.rs: header page + 4 KiB-aligned sections, written via a.partialrename. The header carries a tagged section table (offset, count,stride, tag); the reader finds sections by tag and refuses one whose stride
differs from its record type, so a layout skew is caught at open. It also
records the
Realwidth (mismatch is refused), model metadata, and acaller-supplied 32-byte fingerprint.
index.rsonly does bytes and offsets:SectionWriter/SectionReaderare the whole API, andCompactBvh,ModelMapandMeshModelwrite and read their own sections, so no fieldshad to be exposed for serialisation. The one
unsafein the crate isMmap::map, justified by the file's immutability.current_index(mesh)is the single decision point (indexexists, header parses, fingerprint matches);
MeshModel.from_pathmaps<mesh>.nzidxwhen it returns a path and builds otherwise, andcompile_indexand the CLI use the same check. The fingerprint is ascandirmetadata walk (paths, sizes, mtimes,zarr.jsonbytes), onestatper file and no array reads.nzcvm index build models/*.zarrcompiles;
modelsandsyntheticrecipes run it.Is the build worth skipping?
You asked.
benchmarks/benchmark_index.pyat 2.76 M tetrahedra:mmap, fingerprint walk)Build scales ~linearly, so a 20 M-tet mesh is ~10 s per loading process; eight
workers on a node spend 80 CPU-seconds and eight memory copies reaching the
same tree. Mapped queries run ~12% slower warm (file-backed pages can't use
THP, so the tree walk misses the TLB more), which is the price of sharing.
Not in this PR (documented in
docs/design/index-format.md)registry.pyso--distributedruns processes.A mapped
MeshModelreduces to its path; the rest is plumbing.Surface/Coastlineindexes (same format, 2-D sections).PopulateReadon NFS,WillNeedlocally, neverRandom; content-addressed filenames to dodgeESTALE→SIGBUS; 1 MiBsection alignment. The doc lists what to measure on Cascade first.
Testing
Rust tests: mapped ≡ built at random points (proptest), mixed model maps,
empty meshes, header/name/AABB round trip, refusal of foreign, truncated,
wrong-
Real-width and wrong-stride files, section alignment. Python: loader policy, stalefingerprint falls back to build, CLI,
mapped ≡ builtthroughModelTree.🤖 Generated with Claude Code
Stack created with GitHub Stacks CLI • Give Feedback 💬