Skip to content

Treat any extra site key or column as a label, not just a name - #8

Closed
lispandfound wants to merge 1 commit into
csv-formatfrom
site-labels
Closed

lispandfound wants to merge 1 commit into
csv-formatfrom
site-labels

Conversation

@lispandfound

@lispandfound lispandfound commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

site was a privileged coordinate. Site declared a required name, the
builder wrote it to a fixed Coordinate.SITE, and a station's network code or
a driller's reference had nowhere to go. Nothing about a borehole grid earns
that: the grid needs a position, and everything else is the caller's business.

Longitude and latitude place a site; nothing else is reserved

Every other key in a [[grid.sites]] block, and every other column of a CSV
or Parquet site file, becomes a coordinate on the i axis under the name the
caller gave it, and so a column in table output.

[[grid.sites]]
longitude = 172.15
latitude = -43.70
site = "GULL"          # not a keyword
network = "NZ"
grid,site,network,i,j,k,x,y,z,depth,rho,vp,vs,qp,qs,alpha
boreholes,GULL,NZ,0,0,0,1531509.5,5161095.5,-641.124146,0,1810,1800.00012,500,...

A label keeps the type it was written with, so a numeric column arrives
numeric rather than stringified. Coordinate.SITE is gone, and
examples/borehole.toml calls its labels site and network while the code
privileges neither.

Mashumaro folds the extra keys in a __pre_deserialize__ hook, so the
longitude and latitude validators still run and a config file needn't spell
the labels mapping out.

keep_extra_columns = false drops the labels and keeps only the spatial
coordinates.

Two things this has to guard

A label can't shadow a grid name. A coordinate shadows a variable or
attribute of the same name, so a label called name would turn grid.name
from the grid's name into a DataArray and break {grid.name: grid} in the
builder. I confirmed that before relying on it:

>>> grid.name
'boreholes'
>>> grid.assign_coords(name=("i", ["A", "B"])).name
<xarray.DataArray 'name' (i: 2)>          # not the grid's name any more

RESERVED_COORDINATES in nzcvm.grids.grid is derived from GridSchema's
own fields, plus the logical index, the coastline coordinate and the component
names a writer merges alongside the grid, so it keeps up if the schema
changes. A label naming one of those raises:

Site label(s) name would shadow a grid variable or attribute of the same
name. Rename them, or set keep_extra_columns = false.

Sites have to agree on their labels. A missing key is nearly always a
typo, and the alternative is a column of nulls:

Site 1 carries labels ['network'], but site 0 carries ['network', 'site'].
Every site needs the same labels.

keep_extra_columns = false skips both checks, since there is nothing left to
collide or disagree.

Testing

just synthetic
uv run nzcvm generate examples/borehole.toml synthetic/boreholes.csv
uv run nzcvm generate examples/borehole.toml synthetic/boreholes.parquet
uv run nzcvm generate examples/borehole.toml synthetic/boreholes.zarr
uv run --dev pytest tests/          # 274 passed, 1 skipped
uv run --dev pytest --doctest-modules nzcvm/
uv run ruff check nzcvm/ tests/ && uv run ty check && uv run deptry .

All three writers carry both labels through: CSV and Parquet as site and
network columns, Zarr as coordinates on i. Verified for inline TOML keys,
CSV columns and Parquet columns (including a numeric label that stays
float64), for keep_extra_columns = false, and for both error paths.

Incidental finding, not fixed here

ConfigObject sets forbid_extra_keys = True on a nested class named Meta,
but mashumaro looks for one named Config, so the setting is inert across
every config in the package — an unknown key anywhere is silently dropped
rather than reported:

>>> Site.from_dict({"longitude": 172.0, "latitude": -43.5, "typo": 1})   # before this PR
Site(name=..., longitude=172.0, latitude=-43.5)     # "typo" vanished

That is why extra site keys could be accepted before this change; they just
went nowhere. Renaming Meta to Config would start rejecting typos in every
config section, which is a good change but a wider one than this PR, so I left
it alone.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com


Stack created with GitHub Stacks CLIGive Feedback 💬

`site` was a privileged coordinate: `Site` declared a required `name`, the
builder wrote it to a fixed `Coordinate.SITE`, and a station's network code or
a driller's reference had nowhere to go. Nothing about a borehole grid earns
that. The grid needs a position; everything else is the caller's business.

Longitude and latitude now place a site and the config reserves nothing else.
Every other key in a `[[grid.sites]]` block, and every other column of a CSV
or Parquet site file, becomes a coordinate on the `i` axis under the name the
caller gave it, and so a column in table output. A label keeps the type it was
written with, so a numeric column arrives numeric. `Coordinate.SITE` is gone;
`examples/borehole.toml` calls its labels `site` and `network`, and the code
privileges neither.

Two things the change has to guard.

A coordinate shadows a variable or attribute of the same name, so a label
called `name` would turn `grid.name` from the grid's name into a DataArray and
break `{grid.name: grid}`. `RESERVED_COORDINATES` in `nzcvm.grids.grid` is
derived from `GridSchema`'s own fields, plus the logical index, the coastline
coordinate and the component names a writer merges alongside the grid, so it
keeps up if the schema changes. A label naming one of those raises.

Sites also have to agree on which labels they carry. A missing key is nearly
always a typo, and the alternative is a column of nulls.

`keep_extra_columns = false` drops the labels and keeps only the spatial
coordinates.

Mashumaro folds the extra keys in a `__pre_deserialize__` hook, so the
longitude and latitude validators still run and a config file needn't spell
the `labels` mapping out.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@lispandfound
lispandfound added this pull request to stack #6 September 10, 2026 10:19
@github-actions

Copy link
Copy Markdown

Benchmark for 8868c0f

Click to view benchmark
Test Base PR %
Mesh_Point_Query/centre/4096000 303.3±2.17ns 314.0±2.94ns +3.53%
Mesh_Point_Query/far_corner/4096000 200.1±1.91ns 202.7±8.09ns +1.30%
Mesh_Point_Query/near_origin/4096000 346.5±2.27ns 345.5±2.95ns -0.29%

@lispandfound

Copy link
Copy Markdown
Contributor Author

Folding this into #5: the label design belongs in the borehole grid itself rather than arriving as a follow-up. #5 now ships Site with only longitude and latitude reserved, RESERVED_COORDINATES, and keep_extra_columns, and #7 stacks the CSV/Parquet output on top of that.

@lispandfound
lispandfound deleted the site-labels branch September 10, 2026 10:36
@lispandfound
lispandfound removed this pull request from stack #6 September 10, 2026 10:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant