Skip to content

Cache Model.matrices and trim the matrix export path #1008

Description

@FabianHofmann

Note

The following content was generated by AI.

Describe the feature you'd like to see

Child of #972 (phase 2, PR 5). This helps dense models too, not only sparse models.

Export now takes about 65% of the build time of a sparse model. Most of that time is in HiGHS itself, but three costs come from linopy:

  1. Model.matrices builds a new MatrixAccessor on every access (linopy/model.py:361-363). Every caller that reads m.matrices pays for a full rebuild.
  2. Every access runs variable_scaling_lookup and constraint_scaling_lookup, even when the model sets no scaling. This costs about 30 ms and 140 MB of temporary memory for 1.44M variables.
  3. _stack (linopy/matrices.py:25-38) calls eliminate_zeros on the stacked matrix. Frozen blocks are already pruned, so this call does nothing useful (9 to 15 ms).
import numpy as np, pandas as pd, linopy
linopy.options["semantics"] = "v1"
i, t = pd.RangeIndex(2000, name="i"), pd.RangeIndex(720, name="t")
m = linopy.Model(sparse=True)
x = m.add_variables(0, 1, coords=[i, t], name="x")
m.add_constraints(x <= 1, name="ub")
m.add_objective(x.sum())
assert m.matrices is not m.matrices  # rebuilt on each access, 55 ms each here

Proposed direction

  • Cache the accessor only when every constraint is frozen. This is always the case in a sparse model, and it avoids staleness after in-place edits of an unfrozen Constraint. Invalidate the cache wherever label_index.invalidate() already runs (linopy/common.py:947) and when the objective changes.
  • Skip the scaling lookups when the model sets no scaling.
  • Skip eliminate_zeros when all stacked blocks are CSRConstraint.
  • Add the PyPSA-like benchmark from Tracking: sparse/CSR path follow-ups after #961 #972 as a tracked script under benchmark/, so this PR and the next ones can show numbers before and after.
Benchmark (200 buses x 720 snapshots, 2.09M vars, 1.73M cons, 4.2M nnz)
phase sparse dense dense, frozen
m.matrices 106-117 ms / 254 MB 1337 ms / 1832 MB 110 ms / 254 MB
to_highspy 436-481 ms 1245 ms 445 ms
to_file (LP) 743-820 ms 1795 ms 649 ms

Micro measurements per matrices access: variable_scaling_lookup 23 ms / 99 MB, constraint_scaling_lookup 7 ms / 42 MB, label_to_pos 22 ms / 121 MB, vstack of 2 blocks with a peak of 299 MB, eliminate_zeros 9 ms on a stack without zeros.

Activity

  1. added
    performanceThis improves performance while not (meaningfully) altering behaviour for users
    on Oct 6, 2026
  2. coroa commented on Oct 6, 2026

    @coroa
    Member

    Wait, why?

    In my head the selling point for CSR was. CSR is easy to stack. Why would we want to keep a matrices copy of something that is easy to rebuilt around? What am I missing?

    I think we need to be cognizant of the memory effects this introduces. Could we only keep what is expensive to re-build instead?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performanceThis improves performance while not (meaningfully) altering behaviour for users

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions