One Model-level switch for the sparse path instead of three independent opt-ins #976

Describe the feature you'd like to see

Discussion issue deferred from #972 (part G) and #969 (part 3). Before PR 4 adds more switches, decide what the configuration surface of the sparse/CSR path should be.

Problem

A model only stays sparse end to end with three independent opt-ins, and any subset of them silently gives dense results:

1. linopy.options["semantics"] = "v1": the sparse path is v1-only (is_v1() gate at linopy/expressions.py:613-618 for groupby, :2836 for @).
2. linopy.options["sparse_groupby"] = True, or per call groupby(...).sum(sparse=True): turns on sparse groupby (expressions.py:613) and, as a side effect, decides whether @ returns a CSR-backed result (expressions.py:2893).
3. Model(freeze_constraints=True), or per call add_constraints(..., freeze=True): makes the constraint a CSRConstraint (model.py:1310-1311, :1335).

Under legacy semantics sparse_groupby=True is ignored without a message (only an explicit sparse=True raises, expressions.py:614). Without sparse_groupby, groupby().sum() and @ on a dense input return dense expressions. Without freezing, a sparse lhs becomes a dense Constraint. warn_on_densify would report the last case, but it is off by default (config.py:88).

import warnings

import numpy as np
import pandas as pd

import linopy

warnings.simplefilter("ignore", FutureWarning)

groups = pd.Series(np.arange(6) % 2, index=pd.RangeIndex(6, name="i"), name="g")
weights = pd.DataFrame(
    np.eye(6)[:, :2], index=groups.index, columns=pd.Index(["a", "b"], name="k")
)


def build(semantics: str, sparse_groupby: bool, freeze: bool) -> None:
    linopy.options.reset()
    linopy.options(semantics=semantics, sparse_groupby=sparse_groupby)
    m = linopy.Model(freeze_constraints=freeze)
    x = m.add_variables(lower=0, coords=[groups.index], name="x")
    grouped = x.to_linexpr().groupby(groups).sum()
    matmul = x.to_linexpr() @ weights
    con = m.add_constraints(grouped >= 1, name="c")
    print(
        f"{semantics=!s:6} {sparse_groupby=!s:5} {freeze=!s:5} -> "
        f"groupby sparse: {grouped.is_sparse!s:5}  @ sparse: {matmul.is_sparse!s:5}  "
        f"constraint: {type(con).__name__}"
    )


for args in [("legacy", True, True), ("v1", False, True), ("v1", True, False), ("v1", True, True)]:
    build(*args)

Output on master (b9a698b):

semantics=legacy sparse_groupby=True  freeze=True  -> groupby sparse: False  @ sparse: False  constraint: CSRConstraint
semantics=v1     sparse_groupby=False freeze=True  -> groupby sparse: False  @ sparse: False  constraint: CSRConstraint
semantics=v1     sparse_groupby=True  freeze=False -> groupby sparse: True   @ sparse: True   constraint: Constraint
semantics=v1     sparse_groupby=True  freeze=True  -> groupby sparse: True   @ sparse: True   constraint: CSRConstraint

Only the last row is sparse end to end. In the first two rows the CSRConstraint is built from a dense lhs (Constraint.freeze() → CSRConstraint.from_dense), so the build itself was dense and nothing says so.

Proposal

As little exposed configuration as possible. The rule: the user picks the representation of expressions once per model, and never per operation. Mutability of a single constraint stays a per-constraint choice.

• One read-only model key, Model(sparse: bool = False). For that model it turns on sparse groupby().sum(), @/dot and merge results, and frozen constraints by default. No setter: freeze_constraints has one today (model.py:535-537), and flipping it mid-build makes earlier and later objects differ. Model(dtypes=...) is the precedent for a constructor-only setting that expression operations read. A bool, not a mode (backend=...): there is no third state.
• v1 only, checked where it matters. Model(sparse=True) raises under legacy semantics. Because semantics is a global option that can change after creation, the sparse gates (expressions.py:613, :2836, :3686) also raise when the model is sparse and semantics is legacy, instead of falling back to dense without a message. The model does not pin its semantics.
• No per-operation arguments. groupby(...).sum(sparse=...) and options["sparse_groupby"] go. @ no longer reads sparse_groupby (expressions.py:2893).
• Per-constraint override stays. add_constraints(freeze=...) overrides the model default, like mask= overrides auto_mask. Constraint.freeze() / CSRConstraint.mutable() stay. Immutability is a property of one registered object (loc, update, from_rule need it), not of how an intermediate expression is stored.
• Softening. Frozen constraints soften natively since feat(csr): native soften for frozen constraints #979, so penalty= needs no special case in a sparse model. Do not add a softened constraint unfrozen automatically: a dense constraint inside a sparse model is the divergence this issue removes.
• Model.chunk. Model(sparse=True, chunk=...) and setting chunk on a sparse model raise, instead of dropping freeze (model.py:1312-1313).
• warn_on_densify stays global. It configures reporting, not results, and applies across models.
• Not the v1 default. CSRConstraint has fewer methods than Constraint (constraints.py:937-956); a default that removes methods is more than a storage change. Revisit after the update/from_rule gaps close (soften is done in feat(csr): native soften for frozen constraints #979), independently of the v1 switch.
• Persistence. The key is written to netcdf and read back, and Model.copy keeps it. Today Model.copy keeps freeze_constraints but rebuilds a CSRConstraint as a dense Constraint; a copy of a sparse model stays sparse. Old files that store freeze_constraints map onto the key (see Deprecation).
• Deprecation. sparse_groupby, sum(sparse=...) and Model(freeze_constraints=...) map onto the new key with a FutureWarning and are removed with legacy semantics; all are released API since 0.8.0. Netcdf files written by 0.8.x store freeze_constraints (io.py:1349); reading that attribute stays supported indefinitely.
• Definition of done. For the reproduction above, the four configurations collapse to Model(sparse=True), which is sparse end to end (groupby, @, constraint). Each partial setup either raises (legacy + sparse) or is dense by the user's explicit choice (freeze=False), never silently.

Long-term target: a Constraint with an optional CSR backing, like LinearExpression has today. Then "frozen" disappears as a concept and sparse only describes storage. Until then the key bundles storage and immutability, because a plain Constraint cannot hold a CSR backing.

Combining expressions of two models is out of scope: it is accepted today and silently wrong (the second model's labels alias the first's), which is a separate bug.

Open questions

• warn_on_densify default. With the key, a CSR backing can only come from a model that opted in, so warning by default no longer affects existing users. If the default flips to True, a deliberate freeze=False must not warn (model.py:1350-1357 does today); only unrequested densifying should.
• Scope of the key. add_indicator_constraints already follows freeze_constraints (model.py:1507-1508). Do the objective (sparse since Sparse path for the objective #967) and add_expressions follow the key too?
• PyPSA migration. Under the key every constraint in n.model.constraints is a CSRConstraint. Which PyPSA code paths use loc, update or soften on constraints and need freeze=False? (soften works on frozen constraints since feat(csr): native soften for frozen constraints #979.)
• Removal date. The deprecated switches are "removed with legacy semantics", which Remove legacy semantics at linopy 1.0 (Stage 3) #846 tracks for 1.0, but 1.0 has no target release. Add them to the Remove legacy semantics at linopy 1.0 (Stage 3) #846 checklist, or give them their own removal release.
• Constraints.add(freeze=...) (constraints.py:2462). Keep it as the low-level twin of add_constraints(freeze=...), or drop it if nothing outside linopy uses it.

Consequence for the roadmap

• PR 4 part G (Tracking: sparse/CSR path follow-ups after #961 #972, Make the sparse path observable and controllable (.is_sparse, densify warning, per-call sparse=) #969 part 3) changes from adding per-call sparse= on @/dot/merge to wiring the single read-only model key into groupby, @ and merge, turning the silent v1 gates into errors, removing the @ coupling to options["sparse_groupby"] and deprecating the per-call arguments.
• feat(csr): native soften for frozen constraints #979 (PR 3, closes Native sparse soften for frozen constraints (penalty= with freeze=True) #975) is merged, so softening needs no special case in the key.
• The docs (Document the sparse expression path (CSRLinearExpression, sparse=True, sparse_groupby) #971) then describe one switch, the freeze= override, and is_sparse and warn_on_densify as diagnostics.