Add flat table output formats: CSV and Parquet - #7
Open
lispandfound wants to merge 2 commits into
Open
lispandfound wants to merge 2 commits into
lispandfound wants to merge 2 commits into
Conversation
lispandfound
added this pull request to stack #6
September 10, 2026 09:39
Benchmark for 78ef729Click to view benchmark
|
Benchmark for cc7c656Click to view benchmark
|
lispandfound
changed the title
Benchmark for 09d7c3eClick to view benchmark
|
A borehole grid labels its columns with coordinates on the `i` axis, but
nothing carried those labels into a form a person or a spreadsheet reads.
Pulling a profile out of the Zarr output means opening a DataTree and knowing
which `i` a station sits at.
Writing to a `*.csv`, `*.parquet` or `*.pq` path now emits one row per grid
point, under a header that names every column. Any coordinate a grid carries
past the `(i, j, k)` index becomes a leading label column, so a borehole run
comes out as
grid,site,network,i,j,k,x,y,z,depth,rho,vp,vs,qp,qs,alpha
boreholes,GULL,NZ,0,0,0,1531509.5,5161095.5,-641.124146,0,1810,1800,500,...
which pandas groups straight back into per-station profiles. A `grid` column
separates the refinement levels of an SW4 domain in a single table.
Both encodings come off one `flatten` call, so the columns are identical
either way. CSV writes nine digits of precision, which round-trips a float32
exactly and keeps a seven-digit easting from printing as `1.5315095e+06`.
Parquet keeps the float32 columns typed, so there is no text rendering at all;
`pandas[parquet]` is already a dependency, because the borehole grid reads its
sites from Parquet. Per-column metadata makes a small Parquet file the larger
of the two, so the choice is about the reader rather than about size.
Both writers flatten through pandas and so hold the table in memory: they are
for the outputs a person reads, not for volumetric grids.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The label columns rest on an assumption the `Layer` contract never states: that a layer hands back the coordinates of the grid it received. A layer that assembles its output from raw NumPy does not, because the array has no coordinates to keep. `nzcvm.layers.dummy.constant` is exactly that, so the assumption is not idle, and `map_blocks` rejects a chunk whose coordinates differ from its template. Only five of the six layers appear in `examples/borehole.toml`, so `backus` had never seen a labelled grid at all. `tests/test_layer_coordinates.py` passes a two-site borehole grid to each registered layer on its own, then to the whole chain, then through `execute_model_pipeline`, and checks the labels come back on the result with the grid's shape intact. All six pass. Reverting one layer to drop its coordinates fails both the per-layer test and the `map_blocks` test, so the tests bite. The covered set is read off the config modules in `nzcvm.config.layers` rather than `Layer.registry`, which the rest of the suite adds dummy and sentinel layers to. A new shipped layer therefore fails here until someone covers it. The layers only accept concrete chunks, since `execute_model_pipeline` hoists the chunked dispatch into one `map_blocks` per grid, so the fixtures compute the grid before handing it over. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lispandfound
force-pushed
the
csv-format
branch
from
September 10, 2026 10:36
aebc618 to
63ef6ed
Compare
lispandfound
removed this pull request from stack #6
September 10, 2026 10:39
lispandfound
added this pull request to stack #9
September 10, 2026 10:39
Benchmark for 938db35Click to view benchmark
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The borehole grid in #5 labels its columns with coordinates on the
iaxis,but nothing carried those labels into a form a person or a spreadsheet reads.
Pulling a profile out of the Zarr output means opening a DataTree and knowing
which
ia station sits at.Flat table output
Writing to a
*.csv,*.parquetor*.pqpath now emits one row per gridpoint, under a header that names every column. Any coordinate a grid carries
past the
(i, j, k)index becomes a leading label column, so a borehole runcomes out as
which
pandasgroups straight back into per-station profiles. Agridcolumnseparates the refinement levels of an SW4 domain in a single table.
Both encodings come off one
flattencall innzcvm/formats/table.py, so thecolumns are identical either way:
and keeps a seven-digit easting from printing as
1.5315095e+06.at all.
pandas[parquet]is already a dependency, because the borehole gridin Add a borehole grid: vertical profiles at a set of sites #5 reads its sites from Parquet.
Per-column metadata makes a small Parquet file the larger of the two (10082
vs 9628 bytes for the four-site example), so the choice is about the reader
rather than about size. Both writers hold the table in memory, which suits the
outputs a person reads rather than volumetric grids.
Layers have to preserve coordinates
The label columns rest on an assumption the
Layercontract never states:that a layer hands back the coordinates of the grid it received. A layer that
assembles its output from raw NumPy does not, because the array has no
coordinates to keep —
nzcvm.layers.dummy.constantis exactly that, so theassumption is not idle, and
map_blocksrejects a chunk whose coordinatesdiffer from its template. Only five of the six layers appear in
examples/borehole.toml, sobackushad never seen a labelled grid.tests/test_layer_coordinates.pypasses a two-site borehole grid to eachregistered layer on its own, then to the whole chain, then through
execute_model_pipeline. All six preserve the labels and the grid shape. Thecovered set is read off the config modules in
nzcvm.config.layersratherthan
Layer.registry, which the rest of the suite adds dummy and sentinellayers to, so a new shipped layer fails there until someone covers it.
Testing
Verified against the real pipeline: both tables reproduce the Zarr output of
the same run element for element, and grouping by
sitegives the same fourprofiles as #5. Reverting one layer to drop its coordinates fails both the
per-layer test and the
map_blockstest, so those tests bite.What survives, and what doesn't
Probed each stage of a run for a grid carrying an extra coordinate:
GridSchema.new(...)constructorassign_coordsafter constructionexecute_model_pipeline/map_blocksQualitiesSchema.from_datasetGridSchema.from_datasetread backA coordinate survives because xarray keeps it on each data variable it
indexes, so
cls.new(**dataset.data_vars)reassembles it for free.Unrelated observation
BackusAveragedLayeremitsRuntimeWarning: divide by zero encountered in reciprocalwhenever a sample falls outside every mesh, because it takesnp.reciprocal(q.vs)before masking oncovered. The masked result is finiteand correct. It reproduces on a
regulargrid too, so it predates this stackand I left it alone.
Stack created with GitHub Stacks CLI • Give Feedback 💬