Instance format: a portable, coordinate-keyed export of a built model

Summary

Add a writer that exports a built model as a directory of typed parquet tables keyed by declaration and coordinate, plus a small YAML metadata file. It is an export format for portability, a peer of the LP/MPS writers and to_pyomo / to_linopy. It is not part of the archive and never an input.

It should be built when a concrete consumer needs it. The likely first one is a JuMP adapter.

Scope

In scope

• A writer in relational/sinks/writers/, chosen by suffix (e.g. sps.write(spec, sources, 'dispatch.instance')), reading Handoff and Declared (feat(api): a built model can be handed to pyomo as a ConcreteModel indexed by its labels #1793).
• A documented, versioned on-disk layout that tools in other languages (Julia/JuMP, R, DuckDB) can read without specsolve installed.
• A diff over two instance directories, joined on declaration + coordinates.

Out of scope

• The archive. No keep_instance, no instance/ section, no LAYOUT bump (see "Why not the archive").
• Input. Nothing solves from an instance. Accepting one would be handing in a plan, which hard rule 5 refuses.
• Expressions, where: strings, symbols. Those live in the spec; the metadata links to it by hash.
• Mutable data / updates. An instance is a snapshot of one build.

How we got here

1. MOSDEX. We reviewed MOSDEX: A New Standard for Data Exchange with Optimization Solvers (Bloom, Saltzman, King, 2022/23). It proposes JSON + SQL for solver data exchange, with "instance-form" tables that carry each artifact's natural keys plus a solver row/column id. We agree with most of its diagnosis: model/data separation, relational data, indexing, an output standard. mathspec/specsolve already cover most of it with a closed language instead of SQL. One idea worth taking was a readable, coordinate-keyed instance.

2. Existing pieces. Most of the machinery is already open:

   • feat(api): a written LP or MPS file can name each variable and constraint by its declaration and coordinate #1795: LP/MPS files name variables and constraints by declaration and coordinate. Human-readable, but labels are flattened to strings and it scales poorly for machine consumption.
   • feat(api): a built model can be handed to pyomo as a ConcreteModel indexed by its labels #1793: Declared records which declaration owns each column, row and set, at which coordinate, with label types preserved. to_pyomo builds on it.
   • feat(api): a built model can be handed to linopy with its declared names and dimensions #1797: to_linopy builds on the same pair.
   • Build and change models via math-spec PyPSA/linopy#922: carries the spec through netcdf. Close to an instance, but in linopy's padded internal layout, not engine-neutral.

3. Archive or export? We first considered an opt-in instance/ archive section (mirroring keep_windows), so feat(cli): a manifest file runs several solves of a model with one command #1845's manifest would get it for free. On reflection, the archive doesn't need it:

   • The build is deterministic (dense row-major labels), so an instance is a pure function of spec + sources + specsolve/mathspec versions, all of which the archive already records. Rebuilding gives the same matrix.
   • Inspection is already covered by Model.row, the named LP files (feat(api): a written LP or MPS file can name each variable and constraint by its declaration and coordinate #1795), and the in-memory oracle.
   • Storing it would make the archive hold a version-dependent cache, and the coefficient matrix is usually much larger than the sources.

   What an instance provides that nothing else does is use where specsolve is not present to rebuild. So the scope is portability.

Why not the archive (short version)

Proposed layout

dispatch.instance/
  instance.yaml                  # format version, spec + sources hashes,
                                 # specsolve/mathspec versions, conventions, counts
  variables/<name>.parquet       # (typed dims…, col, lower, upper, vtype, obj)
  constraints/<name>.parquet     # (typed dims…, row, sense, rhs)
  A/constraint=<name>/*.parquet  # (row, col, coeff), hive-partitioned
  Q.parquet                      # (col, col2, coeff), if quadratic
  sos/<name>.parquet             # (dims without the ordering dim…, col, weight)

The metadata file is instance.yaml, not manifest.yaml, to avoid confusion with #1845's run manifest.

Notes:

• Identity is declaration + coordinates. col / row are specsolve's dense labels, kept for convenience; other engines number differently, so diff never joins on them.
• Types are preserved (int, str, datetime), as in to_pyomo.
• No namespace problem. A variable and a constraint can share a name (start_up, shut_down), since they live in different directories. No rename needed.
• Edge cases follow feat(api): a built model can be handed to pyomo as a ConcreteModel indexed by its labels #1793: a dimensionless declaration is one row with no dim columns; a dropped row is absent; an empty scalar is a zero-row file.
• Partitioning A by constraint gives per-declaration coefficient blocks without extra bookkeeping.

Conventions to fix in instance.yaml

• Sense spelling (<=, >=, ==) and objective sense.
• Sign convention for coefficients and rhs (all terms on the left, constant on the right).
• All-zero rows are kept as rows (as feat(api): a built model can be handed to pyomo as a ConcreteModel indexed by its labels #1793 does for pyomo).
• Format version, independent of mathspec's language version and the archive LAYOUT.

Use cases

1. Other-language engines/adapters. A JuMP (or R) adapter can load a specsolve build directly, before a native mathspec engine exists there.
2. External validation. Another engine's CI compares its build against a reference instance produced by specsolve, as a file rather than inside our test suite.
3. Audit snapshots. Freeze exactly what a solver was given, when pinning versions and rebuilding isn't an option.

Follow-ups (not part of this issue)

• specsolve write in the CLI (feat(cli): a manifest file runs several solves of a model with one command #1845 lists write as deliberately not done); the instance writer then comes through suffix dispatch like LP/MPS.
• specsolve diff A B on two instance directories.
• Optionally, linopy (A link with a delay moves power to a later snapshot #922) and the pyomo/JuMP adapters emit the same layout, so builds compare across engines as files.

Trigger

No work until a consumer asks for it. The JuMP adapter is the expected first one. Until then, the named LP/MPS files (#1795) cover "show me what was built."