Repository navigation
Extract the language and typeset into their own package so linopy can build on the same math #732
Description
Prompt: I want to pull the language part of lpspec out into a separate package. How much should we move out? Whats a good seam? linopy wants to use the same math. Any caveats?
Note
The following content was generated by AI.
Base commit: 8a5d18a28d51f050ac7b1581325f4d21a1d20bda (clean tree). The import graph, line counts and names below were derived at that commit; re-derive before acting on them.
How we got here
The lpspec ↔ PyPSA/linopy#871 comparison established that the two languages are siblings (both descend from Calliope's math language) and that the convergence worth pursuing is on the file surface a human reviews — sections, grammar, missing-data semantics, error timing — not on engines. linopy wanting to use the same math makes a second, out-of-repo consumer of the language concrete: the evidence YAGNI asks for before a package split exists now, not hypothetically.
This issue records the extraction analysis: what moves, where the seam is, what the package would export, and what has to be settled before the seam freezes. It is a decision: on yes it becomes a roadmap parent for the extraction work.
The seam already exists in the import graph
language/ imports only errors.py and itself — no polars, no linopy, no relational. Three consumers sit on top of it, all importing the same narrow set: lowering.py (targets the relational plan), linopy/ (the shim), and typeset/. The relational engine below the plan never touches language/ at all. Extraction is moving files, not untangling them.
What moves (~4.5k of 8.7k src/ lines)
language/— parser, where-parser, schema, validation, dim algebra, degree, macro expansion, piecewise, helpers.typeset/— it consumes only the AST, and docs-from-the-math is a language-level feature both sides ship (feat(language): a window's width is a column, not a run of shifts #871 has its own LaTeX builder; convergence means one).- The language half of
errors.py—SchemaError,LanguageError,DimensionError,PiecewiseExpansionError,did_you_mean, plus theLpspecErrorbase so both packages share a root. docs/SPEC.md— the language package owns the spec; the engine becomes a conformer.
Stays: api, lowering, relational/, sources, strategy. lowering.py looks language-adjacent but targets the relational plan — it is the engine's front door, not the language's back door. The linopy/ shim stays with the engine as differential oracle; its long-term home is linopy itself, and that migration — not the package split — is when the open convergence questions become blocking.
The export surface
Post-validation, pre-data: everything checkable without data (dim algebra, degree, name kinds, dtype guards) happens inside the package; everything needing data (binding, absence handling, omissions) is the consumer's. One door in (load_model, then expression_of/where_of), AST out. The in-repo linopy shim is the existence proof that a dense xarray consumer can build from this surface.
40 classes — but wide in types, narrow in machinery: 20 are AST nodes and 12 are schema blocks, passive data types whose count mirrors the language's construct count. Only Namespace, Model, SymbolTable and the errors carry behavior. Contract churn therefore tracks SPEC churn, not implementation churn.
Full export surface, derived from what the staying consumers import
| Group | Names | Classes |
|---|---|---|
Schema (language/model.py, language/validation.py) |
load_model; Model, DimensionBlock, CoordinateSpec, ParameterBlock, VariableBlock, BoundsBlock, ConstraintBlock, ObjectiveBlock, MacroBlock, PiecewiseBlock, PiecewiseLink, SosBlock — every block field is contract; consumers walk them |
12 |
Resolution (language/resolution.py) |
Namespace, expression_of, where_of |
1 |
Expression AST (language/expression_parser.py) |
NumberNode, NameNode, VariableNode, ParameterNode, DimensionNode, CoordinateNode, KeywordNode, EdgeNode, UnaryOperatorNode, BinaryOperatorNode, FunctionCallNode, ComparisonNode; aliases ExpressionNode, ArithmeticNode; children() |
12 |
Where AST (language/where_parser.py) |
AndNode, OrNode, NotNode, BooleanLiteralNode, ParameterDefinedNode, VariableDefinedNode, ParameterComparisonNode, DimensionComparisonNode; alias WhereNode |
8 |
Static analysis (language/dimensions.py, language/degree.py) |
dims_of; degree.carries_variable, degree.check_binary, degree.check_expression |
0 |
Expansion (language/piecewise.py) |
expand_piecewise (macros expand inside load_model, no export) |
0 |
Builtin table (language/helpers.py) |
BUILTINS, Builtin, BUILTIN_NAMES, EDGE_WRAP; message formatters call_shape_error, edge_error, unknown_helper_message (consumers raise these errors at their layer; wording stays single-homed) |
1 |
| Errors | LpspecError, SchemaError, LanguageError, DimensionError, PiecewiseExpansionError; did_you_mean |
5 |
| Typeset | to_latex, to_markdown, to_typst, SymbolTable, FORMATS, the typeset CLI entry |
1 |
Deliberately not exported: parse_expression/parse_where (both engines assert trees must come through resolution.expression_of()/where_of()), read_yaml, check_schema, validate_expressions, resolve_expression/resolve_where, _StrictBlock.
The AST group is where compatibility risk concentrates: consumers dispatch exhaustively over node types, so a new node or field breaks every builder — which is the aligned outcome, since the node set changes exactly when the language does.
Caveats
- "Breaking changes are free" dies at the seam. The AST node types and
Namespaceare private-shaped today. The moment linopy builds against them they are a contract discovered out-of-repo, later. Export the smallest surface the two real consumers need; keep everything else private. - A shared grammar is not a shared language. The deepest lpspec/feat(language): a window's width is a column, not a run of shifts #871 split — what missing data means (position-dependent absence vs per-parameter
defaultfill) — lives in the consumers, not the parser. If linopy takes parse+validate and keeps default-fill semantics, the same file means different models in the two stacks, which is worse than two syntaxes. The package must export semantics, not just an AST: SPEC plus a conformance suite — model files with expected build outcomes (rows built, rows dropped, refusals) any consumer runs against its own engine. The missing-data question should be settled, or explicitly scoped into the spec, before the seam freezes; it constrains everything else. - Dependency hygiene is the pitch — enforce it. The language package depends on pyparsing + PyYAML + pydantic and nothing heavier. Give it its own architecture guard (no polars, no pandas, no linopy, no xarray) so the invariant that made extraction cheap survives contributions from the consumer's side.
- Two version lines. The engine keeps git-tag versioning; the language package gets its own, and the
version:field in model files now gates against the language version. Right outcome, but a semantic change to the field — a SPEC §0 edit. - Naming. The language is lpspec's identity: the cleanest split gives the extracted package the
lpspecname and renames the engine, rather than inventinglpspec-lang.
Fix at the seam, not after
UnresolvedComparisonNode/UnresolvedNameNodeleak engine-side. A consumer of a "resolved" tree importing nodes named Unresolved means either resolution genuinely leaves them in (misnamed contract) or they are imported only defensively (should not be exported). Settle before the names freeze.Namespaceis hand-assembled by every consumer from the schema, widened with data-derived labels. That recipe would be copied into linopy — add aNamespace.of(schema, *, labels=...)constructor so the Model→Namespace mapping has one home.
Activity