feat(csr): keep CSR backing through metadata, constant arithmetic, sum/groupby and selection by FabianHofmann · Pull Request #973 · PyPSA/linopy

FabianHofmann · 2026-09-23T15:04:22Z

Closes #962, closes #964, closes #965, closes #966. Part of #972 (PR 1 of 3). Partially addresses #969 and #970.

Note

The following content was generated by AI.

Changes proposed in this Pull Request

A CSR-backed LinearExpression (v1 only, from groupby(...).sum(sparse=True) or options["sparse_groupby"]) now keeps its sparse backing through the common model-building operations instead of silently expanding to the dense _term rectangle. Sparse results equal the dense ones cell by cell (values, coordinates incl. auxiliary coordinates, absent cells); only the term layout may differ.

Observability (#962, #969 parts 1+2)

Inspection no longer densifies: repr, dims, shape, sizes, coords, indexes, const, isnull() are served from the CSR store. The repr prints only the displayed rows and shows LinearExpression (sparse).
Public LinearExpression.is_sparse.
Opt-in options["warn_on_densify"] (default off): one PerformanceWarning per densification, naming the reason and pointing at user code. A new warn_outside_linopy helper is shared with warn_legacy.

Operations that keep the backing

Elementwise arithmetic with non-scalar constants: *, /, +, - with DataArray / Series / ndarray, aligned through the same _broadcast_and_align as the dense path (Keep CSR backing for elementwise ops with non-scalar constants #965).
sum(dim) and chained groupby().sum() via direct row aggregation (Sparse sum(dim=...) and chained groupby on CSR-backed expressions #964).
where, sel, isel, loc, [] via one selection path on grid row numbers, so semantics and errors match xarray (Keep CSR backing through where/mask/sel/isel/loc #966).
add_constraints(..., mask=, freeze=True) with an expression lhs stays sparse (mask part of Widen the CSRConstraint fast path in Model.add_constraints #970).
merge of differing grids with auxiliary coordinates.

Operations that still densify (operands adding new dims, MultiIndex, callables, expression other in where, …) do so explicitly and report it under warn_on_densify.

Zero policy. All operations keep explicit zeros, so sparse and dense give the same terms up to order; only @/dot prunes them (unchanged from #961). Documented in the linopy/csr.py module docstring.

Bug fixes (dense and sparse)

merge with join="outer"/"left", and arithmetic with a constant on an outer join, raised xarray's MergeError (or, on the sparse path, wrote 0) when only one operand carried an auxiliary coordinate. A scalar reindex/concat fill was also applied to auxiliary coordinates; the fill is now restricted to the data values. Affects legacy semantics too.
Objective.sel wrapped LinearExpression.sel and failed on a quadratic objective.

Internal

Auxiliary coordinates of a CSR expression live on Grid.aux (scalar coordinates included).
LinearExpressionGroupby stores its source as obj (was data), so it can group a CSR-backed expression without densifying.

Not in this PR (see #972): aux coords on CSRConstraint (#941), CSRConstraint API errors (#963), expression rhs and mask= on a pre-built CSRConstraint (#970), sparse objective / flat / to_polars (#967, #968), per-call sparse= (#969 part 3), docs (#971).

Testing

test/test_csr.py extended with sparse-vs-dense parity tests for every operation above, a sparse_and_dense helper, and a "backing kept" assertion; fallbacks assert exactly one densify notice with its reason.
Dense regression tests for the aux-coord fill bug in test/test_legacy_violations.py, over join and operand order.
Full suite (Python 3.12, excluding test/remote and Xpress, which fails locally on a licence limit only): 7002 passed, 1018 skipped. test_csr.py + test_legacy_violations.py on Python 3.11: 848 passed.
ruff and mypy clean.
Two Opus review passes (correctness ~150 differential sparse/dense cases; quality/tests); all confirmed findings fixed in ededa153, bb22c8a2, 63ebfa97.

Checklist

AI-generated content is marked (see AGENTS.md).
Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
Unit tests for new features were added (if applicable).
A note for the release notes doc/release_notes.rst of the upcoming release is included.
I consent to the release of this PR's code under the MIT license.

…and warn_on_densify Closes the densify-on-read of #962 and parts 1+2 of #969.

…ants (#965)

…ons (#964)

Selection runs on the grid's row numbers with xarray semantics, rows are gathered by CSRLinearExpression.taken; Grid.aux carries scalar coords. Cross-grid merges with aux coords stay sparse.

…l on joins - warn_outside_linopy: shared frame-skipping warning, dynamic on <3.12 - each densification emits exactly one notice carrying its reason - aggregated keeps a float constant on empty grids - joins with a constant fill only values; aux coords of new labels are NaN

…ed constraint once Keyword-only flags on _combined_with_constant; sparse_and_dense test helper and one parametrized absent-cells test.

…roupby obj argument and constant-join aux fix

github-actions · 2026-09-23T15:07:56Z

Build cost — v1 vs legacy

v1 build peak & time relative to legacy, on this commit — not a comparison against master (that is CodSpeed).

peak — v1 / legacy	time — v1 / legacy

Full table (time + peak, mean)

benchmarks/drivers/test_build.py::test_build[basic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.08911 (1.09)   │   15.03 (1.00) 
 (v1)        0.08204 (1.0)   │    15.00 (1.0) 

benchmarks/drivers/test_build.py::test_build[basic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.09494 (1.08)   │   12.04 (1.00) 
 (v1)        0.08799 (1.0)   │    12.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=0]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03828 (1.08)   │   15.20 (1.0) 
 (v1)        0.03551 (1.0)   │   15.20 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=100]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.05512 (1.10)   │   44.93 (1.0) 
 (v1)        0.04998 (1.0)   │   44.93 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04191 (1.07)   │   11.51 (1.0) 
 (v1)        0.03911 (1.0)   │   11.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.09865 (1.04)   │   24.34 (1.06) 
 (v1)        0.09482 (1.0)   │    23.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=250]
                 time (s)         peak (MiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1096 (1.06)   │   16.12 (1.00) 
 (v1)        0.1032 (1.0)   │    16.12 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=10000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.02395 (1.07)   │   752.18 (1.10) 
 (v1)        0.02228 (1.0)   │    685.15 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=100]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02338 (1.06)   │   3.12 (1.33) 
 (v1)        0.02201 (1.0)   │    2.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=0]
                 time (s)          peak (MiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.0641 (1.22)   │   126.16 (1.44) 
 (v1)       0.05236 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=100]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.06274 (1.22)   │   126.16 (1.44) 
 (v1)        0.05156 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=50]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.06416 (1.21)   │   126.16 (1.44) 
 (v1)        0.05284 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[masked-n=100]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.05523 (1.02)   │    715.12 (1.0) 
 (v1)        0.05399 (1.0)   │   787.73 (1.10) 

benchmarks/drivers/test_build.py::test_build[masked-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.05344 (1.10)   │   4.54 (1.27) 
 (v1)        0.04873 (1.0)   │    3.57 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=0]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.3723 (1.04)   │   704.12 (1.09) 
 (v1)        0.3567 (1.0)   │    643.85 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=100]
                 time (s)        peak (MiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.3917 (1.04)   │   18.34 (1.0) 
 (v1)        0.3755 (1.0)   │   18.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=50]
                time (s)       peak (MiB) 
 name               mean   │         mean 
──────────────────────────────────────────
 (legacy)   0.389 (1.04)   │   9.54 (1.0) 
 (v1)       0.3735 (1.0)   │   9.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.07737 (1.12)   │   3.77 (1.12) 
 (v1)        0.06935 (1.0)   │    3.37 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=50]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.07528 (1.08)   │   216.59 (1.10) 
 (v1)        0.06994 (1.0)   │    196.23 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=0]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.03803 (1.08)   │   938.49 (1.0) 
 (v1)        0.03527 (1.0)   │   938.49 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.03978 (1.09)   │   9.66 (1.0) 
 (v1)        0.03644 (1.0)   │   9.66 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.03928 (1.09)   │   5.32 (1.0) 
 (v1)        0.03598 (1.0)   │   5.32 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=0]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.0214 (1.03)   │   1.47 (1.0) 
 (v1)       0.02087 (1.0)   │   1.47 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.02118 (1.02)   │   1.47 (1.0) 
 (v1)        0.02081 (1.0)   │   1.47 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.02152 (1.03)   │   1.47 (1.0) 
 (v1)        0.02096 (1.0)   │   1.47 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=1000]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.1873 (1.04)   │   946.85 (1.06) 
 (v1)        0.1803 (1.0)   │    891.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=10]
                 time (s)         peak (KiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1849 (1.04)   │   12.01 (1.00) 
 (v1)        0.1772 (1.0)   │    11.99 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04746 (1.06)   │   147.70 (1.06) 
 (v1)        0.04476 (1.0)   │    139.87 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04735 (1.06)   │   2.60 (1.09) 
 (v1)        0.04462 (1.0)   │    2.38 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=0]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03888 (1.08)   │   696.75 (1.03) 
 (v1)        0.03608 (1.0)   │    673.70 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=100]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)      0.083 (1.0)   │   137.97 (1.0) 
 (v1)       0.08343 (1.01)   │   137.97 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.05998 (1.09)   │   69.22 (1.0) 
 (v1)        0.05478 (1.0)   │   69.22 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04554 (1.10)   │   402.33 (1.00) 
 (v1)        0.04151 (1.0)   │    402.30 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04492 (1.08)   │   3.19 (1.19) 
 (v1)        0.04146 (1.0)   │    2.69 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.04917 (1.06)   │   29.00 (1.54) 
 (v1)        0.04618 (1.0)   │    18.84 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.05934 (1.10)   │   37.95 (1.43) 
 (v1)        0.05385 (1.0)   │    26.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[storage-n=10]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.09484 (1.0)   │    410.93 (1.0) 
 (v1)       0.0965 (1.02)   │   427.84 (1.04) 

benchmarks/drivers/test_build.py::test_build[storage-n=250]
                 time (s)         peak (MiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)    0.1011 (1.0)   │     9.94 (1.0) 
 (v1)       0.1043 (1.03)   │   10.22 (1.03)

📊 Interactive plots + CSV: download the semantics-report-v1-vs-legacy artifact from this run.

_{Report-only · not a gate · refreshed on every push · obsolete once legacy is dropped.}

codspeed · 2026-09-23T15:08:25Z

Merging this PR will degrade performance by 7.86%

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
❌ 1 regressed benchmark
✅ 178 untouched benchmarks
⏩ 181 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Benchmark	`BASE`	`HEAD`	Efficiency
❌	`test_to_lp[sparse_network-n=10]`	718.3 KB	1,343.2 KB	-46.52%
⚡	`test_to_lp[qp-n=1000]`	2.6 MB	2 MB	+29.81%
⚡	`test_to_lp[nodal_balance-severity=50]`	3.7 MB	3.3 MB	+12.69%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing feat/csr-expression-kernel (799d514) with master (82e7993)}

181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

…draticExpression in test op table

FabianHofmann added 11 commits September 23, 2026 13:05

refactor(csr): move auxiliary coordinates into Grid

bd955bf

feat(csr): serve metadata and repr from the CSR store, add is_sparse …

8fcb2b7

…and warn_on_densify Closes the densify-on-read of #962 and parts 1+2 of #969.

fix(csr): no densify notice for dense-in dense-out @

3bcfc7e

feat(csr): keep CSR backing for elementwise ops with non-scalar const…

bbabfbc

…ants (#965)

feat(csr): sparse sum(dim) and chained groupby on CSR-backed expressi…

5a67060

…ons (#964)

feat(csr): keep CSR backing through where/sel/isel/loc (#966)

c0364fc

Selection runs on the grid's row numbers with xarray semantics, rows are gathered by CSRLinearExpression.taken; Grid.aux carries scalar coords. Cross-grid merges with aux coords stay sparse.

feat(csr): sparse mask= in add_constraints (#970)

8446895

fix(merge): keep an aux coord carried by one operand on outer/left joins

b160b36

refactor(csr): drop ndim override, share cell-row offsets, build mask…

bb22c8a

…ed constraint once Keyword-only flags on _combined_with_constant; sparse_and_dense test helper and one parametrized absent-cells test.

docs: user-facing wording for sparse metadata, note LinearExpressionG…

63ebfa9

…roupby obj argument and constant-join aux fix

FabianHofmann added performance This improves performance while not (meaningfully) altering behaviour for users sparse Sparse / CSR-backed expressions and constraints labels Sep 23, 2026

FabianHofmann mentioned this pull request Sep 23, 2026

feat(csr): constraint and export boundaries of the sparse path (PR 2 of #972) #974

Merged

fix(types): widen LinearExpression.where cond/other typing, allow Qua…

799d514

…draticExpression in test op table

This was referenced Sep 24, 2026

One Model-level switch for the sparse path instead of three independent opt-ins #976

Closed

Tracking: sparse/CSR path follow-ups after #961 #972

Open

FabianHofmann merged commit 38fbd4a into master Sep 24, 2026
21 of 23 checks passed

FabianHofmann deleted the feat/csr-expression-kernel branch September 24, 2026 11:12

Navigation Menu

Repository navigation

feat(csr): keep CSR backing through metadata, constant arithmetic, sum/groupby and selection - #973

feat(csr): keep CSR backing through metadata, constant arithmetic, sum/groupby and selection#973
FabianHofmann merged 12 commits into
masterfrom
feat/csr-expression-kernel

FabianHofmann commented Sep 23, 2026

Uh oh!

github-actions Bot commented Sep 23, 2026 •

edited

Loading

Uh oh!

codspeed Bot commented Sep 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

FabianHofmann commented Sep 23, 2026

Changes proposed in this Pull Request

Checklist

Uh oh!

github-actions Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build cost — v1 vs legacy

Uh oh!

codspeed Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will degrade performance by 7.86%

Performance Changes

Footnotes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Sep 23, 2026 •

edited

Loading

codspeed Bot commented Sep 23, 2026 •

edited

Loading