You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Cache Model.matrices and trim the matrix export path #1008
Child of #972 (phase 2, PR 5). This helps dense models too, not only sparse models.
Export now takes about 65% of the build time of a sparse model. Most of that time is in HiGHS itself, but three costs come from linopy:
Model.matrices builds a new MatrixAccessor on every access (linopy/model.py:361-363). Every caller that reads m.matrices pays for a full rebuild.
Every access runs variable_scaling_lookup and constraint_scaling_lookup, even when the model sets no scaling. This costs about 30 ms and 140 MB of temporary memory for 1.44M variables.
_stack (linopy/matrices.py:25-38) calls eliminate_zeros on the stacked matrix. Frozen blocks are already pruned, so this call does nothing useful (9 to 15 ms).
importnumpyasnp, pandasaspd, linopylinopy.options["semantics"] ="v1"i, t=pd.RangeIndex(2000, name="i"), pd.RangeIndex(720, name="t")
m=linopy.Model(sparse=True)
x=m.add_variables(0, 1, coords=[i, t], name="x")
m.add_constraints(x<=1, name="ub")
m.add_objective(x.sum())
assertm.matricesisnotm.matrices# rebuilt on each access, 55 ms each here
Proposed direction
Cache the accessor only when every constraint is frozen. This is always the case in a sparse model, and it avoids staleness after in-place edits of an unfrozen Constraint. Invalidate the cache wherever label_index.invalidate() already runs (linopy/common.py:947) and when the objective changes.
Skip the scaling lookups when the model sets no scaling.
Skip eliminate_zeros when all stacked blocks are CSRConstraint.
Micro measurements per matrices access: variable_scaling_lookup 23 ms / 99 MB, constraint_scaling_lookup 7 ms / 42 MB, label_to_pos 22 ms / 121 MB, vstack of 2 blocks with a peak of 299 MB, eliminate_zeros 9 ms on a stack without zeros.
In my head the selling point for CSR was. CSR is easy to stack. Why would we want to keep a matrices copy of something that is easy to rebuilt around? What am I missing?
I think we need to be cognizant of the memory effects this introduces. Could we only keep what is expensive to re-build instead?
Note
The following content was generated by AI.
Describe the feature you'd like to see
Child of #972 (phase 2, PR 5). This helps dense models too, not only sparse models.
Export now takes about 65% of the build time of a sparse model. Most of that time is in HiGHS itself, but three costs come from linopy:
Model.matricesbuilds a newMatrixAccessoron every access (linopy/model.py:361-363). Every caller that readsm.matricespays for a full rebuild.variable_scaling_lookupandconstraint_scaling_lookup, even when the model sets no scaling. This costs about 30 ms and 140 MB of temporary memory for 1.44M variables._stack(linopy/matrices.py:25-38) callseliminate_zeroson the stacked matrix. Frozen blocks are already pruned, so this call does nothing useful (9 to 15 ms).Proposed direction
Constraint. Invalidate the cache whereverlabel_index.invalidate()already runs (linopy/common.py:947) and when the objective changes.eliminate_zeroswhen all stacked blocks areCSRConstraint.benchmark/, so this PR and the next ones can show numbers before and after.Benchmark (200 buses x 720 snapshots, 2.09M vars, 1.73M cons, 4.2M nnz)
m.matricesto_highspyto_file(LP)Micro measurements per
matricesaccess:variable_scaling_lookup23 ms / 99 MB,constraint_scaling_lookup7 ms / 42 MB,label_to_pos22 ms / 121 MB,vstackof 2 blocks with a peak of 299 MB,eliminate_zeros9 ms on a stack without zeros.