Skip to content

docs(benchmarks): the published benchmarks are re-measured at the new model sizes, five times each tenfold step - #2039

Merged
FBumann merged 6 commits into
mainfrom
bench-results/pr-2032
Oct 14, 2026
Merged

FBumann merged 6 commits into
mainfrom
bench-results/pr-2032

Conversation

@github-actions

@github-actions github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

The published benchmark run on fbc735a, which is main at #2032 plus #2038's workflow fix. The comment on #2032 compares these numbers with the committed ones; the docs preview of this pull request shows the page.

Prompt: I want the benchmark to get main up to date first

Note

The following content was generated by AI.

This publishes #2032's new sizes, measured on the box, to main. It also carries #2038's two commits, because the run was dispatched from that branch: on main, published-benchmark.yml does not parse. Merging this makes #2038 redundant.

Merge this with a merge commit, not a squash. bench/reproduce.py now pins fbc735a5b, the commit that was measured. A merge commit puts fbc735a into main's history. After a squash it would be reachable only through claude/specsolve-2032-review-7hrcv0.

What is in the diff, and what was checked

Run on a cloud container at afff79b: pixi run check (lint, format, typecheck; 5613 passed, 566 skipped, 4 xfailed), pixi run test-bench (475 passed, 15 skipped), pixi run -e docs docs-build (strict, no issues), tests/test_docs_site.py and tests/test_doc_examples.py (123 passed, 4 skipped).

Read from the run's artifacts. All 8 cases finished. No cell was killed by the memory watchdog, and no library stops at a smaller size than in #2031. pyomo on network reaches s5, one size further. linopy and gurobipy-loop on network still stop at m, by the 16 GB and 30 s budgets. The matrix floor's row leaves the "launch to exit" table, because #2034 made test_fresh skip both matrix arms.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NTByC5UJJWvcFj3mvmv83v

claude and others added 2 commits October 14, 2026 06:00
…e refuses a workflow that does not

An apostrophe inside a single-quoted description made the file
unreadable, so GitHub saw no workflow_dispatch trigger and the
trigger:published-bench label's dispatch failed with HTTP 422.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NTByC5UJJWvcFj3mvmv83v
@read-the-docs-community

read-the-docs-community Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Documentation build overview

📚 specsolve | 🛠️ Build #35146496 | 📁 Comparing afff79b against latest (2588883)

  🔍 Preview build  

2 files changed
± about/benchmarks/index.html
± about/changelog/index.html

@FBumann
FBumann changed the base branch from claude/specsolve-2032-review-7hrcv0 to main October 14, 2026 12:04
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NTByC5UJJWvcFj3mvmv83v
@FBumann FBumann changed the title docs(benchmarks): the published benchmarks are re-measured on #2032 docs(benchmarks): the published benchmarks are re-measured at the new model sizes, five times each tenfold step Oct 14, 2026
@codspeed

codspeed Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 104 untouched benchmarks
⏩ 307 skipped benchmarks1


Comparing bench-results/pr-2032 (afff79b) with main (2588883)2

Open in CodSpeed

Footnotes

  1. 307 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on main (48c9414) during the generation of this report, so 2588883 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

… numbers were taken on

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NTByC5UJJWvcFj3mvmv83v
FBumann added a commit that referenced this pull request Oct 14, 2026
…every model, and a box dragged on any chart zooms all of them (#2040)

> **Prompt:** Can we add more brushes for seconds/MB ? And align all
axes?

> **Prompt:** I think such a box to zoom should replace all sliders!!! …
Keep the box. We need to keep al plots in sync!!

> [!NOTE]
> The following content was generated by AI.

Every panel of the benchmark chart has the same x and y axes. Dragging a
box on any panel zooms all four to the box's size range and value range,
and the table to its sizes. The size slider is gone, and a **Reset
zoom** button sits above the charts.

<details><summary>What changed</summary>

`docs/javascripts/benchmarks.js`:

- **Shared axes.** `draw()` collects every visible panel's points in
view, and `sharedScales` builds one x and one y axis from all of them.
Unzoomed, they look as on `main` but span every panel.
- **Zoom box.** `state.zoom` holds the last box in data units: a size
range `x` and a value range `y`, each null where the box left that axis
alone. A drag on any panel draws the box in all four at once. The panels
share axes and size, so the same pixels are the same range in each.
Tooltips are hidden during the drag so they do not cover it. On release
the box's corners are inverted through the panel's resolved scales,
which a control that draws nothing (`reader`) keeps. A side under 8 px
leaves that axis as it was, so a horizontal drag zooms only sizes and a
vertical drag only values. A zoomed axis ends exactly at the box. Its
log ticks are 1–2–5 inside the box, or d3's own log ticks where a narrow
range leaves fewer than three. The table keeps only the sizes inside the
box.
- **Zooming out.** **Reset zoom** sits above the charts at all times and
is disabled while nothing is zoomed. A double-click on a chart also
zooms out. Switching measure or view zooms out too, because the units
change. Switching the axis type keeps the zoom. The box also works under
÷ specsolve, where it replaces the fixed window, and the edge labels
follow it.
- **No text selection.** A press on a chart cancels `mousedown`, so
neither a drag nor a double-click selects text on the page. The charts
are also `user-select: none` with the `-webkit-` prefix, which Safari
still needs.
- **specsolve's name** sits right of its last point inside the y axis,
and is left out where none is. It is drawn as a control, outside the
marks' clip, so a zoomed axis had put it above the panel.
- **The size slider is removed**, with its decade stops. Those stops
counted a 5× size toward the decade above: dispatch `xs5` (50k) as 100k.
- **x ticks on powers of ten.** One axis over four models would
otherwise show every model's sizes as ticks. Where fewer than two powers
fit, the ticks fall back to 1–2–5.
- **y ticks evenly spaced.** `logTicks` always kept the top tick, so it
could sit next to the tick before it: "50 s" printed over "20 s". That
was already the case on `main` in the network and sector panels, and
shared axes would put it in every panel. The domain now runs on to the
next tick at the same step.

`docs/about/benchmarks.md`: the slider is removed. A bar above the
charts holds the hint "drag on a chart to zoom every chart" and the
reset button. The reading guide says the panels share their axes and how
the box works.

`docs/stylesheets/benchmarks.css`: the slider rules are removed. The
box, the reset bar, the crosshair cursor, and hiding tooltips during a
drag are added. `touch-action: pan-y` keeps a vertical swipe scrolling
the page on a phone, so there a drag zooms along x.

**Why not the library's own gestures.** TanStack Charts 1.0 has `brushX`
and `zoomX`, both along x only. Its docs say a different gesture should
be built by the application, with `d3-brush` or directly, and that the
y-domain policy belongs to the application too. This box is about 50
lines and adds no dependency, where `d3-brush` would add two.

</details>

<details><summary>Verified, and not done</summary>

`pixi run check` on `208b84c`: lint, format and typecheck clean; 5601
passed, 566 skipped, 4 xfailed. `tests/test_docs_site.py`: 10 passed.
`pixi run -e docs docs-build`: strict, no issues.

The page was then loaded in Chromium with #2039's `benchmarks.json`
dropped into the built site only. This session's egress policy blocks
`cdn.jsdelivr.net`, so the scratch copies of the page loaded the same
pinned packages (`@tanstack/charts@1.0.0`, `d3-scale@4.0.2`,
`d3-shape@3.2.0`) from npm instead. The tree's import map is unchanged.
Real mouse drags, each step checking that all four panels carry
identical tick labels:

| step | axes, all four panels | table sizes |
| --- | --- | --- |
| start | 1k–10M · 0.2 ms–200 s | all 21 |
| box on network | 20k–2M · 1 ms–1 s; box shown in 4 panels mid-drag |
10k … 1.5M |
| horizontal drag on nodal | 50k–200k · y kept | 50k, 100k, 150k |
| double-click | back to start | all 21 |
| vertical drag on dispatch | x kept · 50 ms–5 s | all 21 |
| Reset zoom | back to start | all 21 |
| box, then peak RSS | zoom cleared · 100 MB–5 GB | all 21 |
| box under ÷ specsolve | 50k–1M · 0.5×–5× | narrowed |
| then linear axes | zoom kept | narrowed |

The browser selection stays empty during and after a drag inside a plot,
past the panel's edge, and from the axis labels, and after a
double-click. A text selection during zooming was reported in a browser
this session does not have; Chromium did not reproduce it before or
after the fix.

Two console errors appear on both pages, `main`'s included: another
blocked host, and a `TypeError` reading `output`. They are not from this
change.

Not done, by decision: the zoom has no keyboard path. The size slider
had one, and the library's `zoomX` would have one for x only. On a phone
a drag zooms along x only, because `pan-y` keeps the page scrollable.

</details>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01NTByC5UJJWvcFj3mvmv83v

---------

Co-authored-by: Claude <noreply@anthropic.com>
FBumann added a commit that referenced this pull request Oct 14, 2026
…es as the columns and rows that reach the solver, and tables what each model declares (#2042)

> **Prompt:** And should we make the model overview at the top a table
with variables, constraints, dims, sparsity,... columns?

> **Prompt:** Should we also add information about complex groupings?
Thats what kills linopy memory...

> **Prompt:** columns and rows across the page. But it needs to be
clearly defined and understandable that its the count of variabels that
reach the solver

> **Prompt:** THis is quite some text to read before one sees results.
Lets make it more compact. … It seems like this could be much more
punchy too

> [!NOTE]
> The following content was generated by AI.

The benchmarks page opens on two sentences and its charts, and its
reading guide is half as long. Sizes are counted as **columns and
rows**, the variables and constraints that reach the solver. Below the
reading guide, a generated table says what each model is for, what it
declares, and what one snapshot sends to the solver.

Page order now: title, two sentences, charts, "Reading the charts", "The
models" (definition, table, memory finding), "A script that solves one
model", "Not measured yet", "Method".

| | dispatch | network | nodal | sector |
| --- | --- | --- | --- | --- |
| for | bounds and one balance: the shape an array library suits best |
a small PyPSA model: lines, stores, solar and wind, and pooled caps |
dispatch at nodes, where a technology exists only where it is installed
| each technology delivers one of five carriers, such as heat or
hydrogen |
| variables | p[generator] | p[bus, tech], f[line], charge[store],
discharge[store], soc[store] | p[node, tech] | p[node, tech] |
| constraints | power_balance | balance[bus], soc_balance[store],
pool_cap[pool] | balance[node] | balance[node, carrier] |
| grouped sums | — | line → bus, store → bus, bus × tech → pool | — | —
|
| dimensions | generator 100 | bus 50, tech 12, line 100, store 50 |
node 50, tech 12 | node 50, tech 12, carrier 5 |
| density | 100% | 25% | 25% | 25% |
| snapshots | 2–100k | 2–25k | 2–20k | 2–20k |
| columns per snapshot | 100 | 400 | 150 | 150 |
| rows per snapshot | 1 | 150 | 50 | 125 |
| nonzeros per snapshot | 100 | 800 | 150 | 150 |

> From 100k columns up, linopy's peak memory stays within 1.3× of
specsolve's on HiGHS, except on `network`, the one model that sums
through a grouping. There it is 2.8× at 1M columns, 1.76 GB against 0.63
GB, and the memory budget stops linopy at that size (#2031).

<details><summary>What changed</summary>

`docs/about/benchmarks.md`:

- The intro is two sentences, and the charts follow it.
- "Reading the charts" says each rule in one or two short sentences,
with a bold lead-in that carries the claim. It says the x axis counts
columns, the variables that reach the solver. The round count and the
projection's two edges move to "Method". The guide went from 424 to 229
words and from 25 to 17 sentences; the median sentence went from 14 to
11 words, and the sentences over 25 words from 5 to 0.
- The chart's axis and tooltips say "columns", through the chart spec's
`label`; the data field stays `variables`.
- "The models" holds a one-sentence definition of columns and rows, the
generated table, and the memory finding. The four description paragraphs
are gone, replaced by the table's "for" row.
- The `nodal` wording and the projection note say columns.

`bench/cases.py`: `Case.about`, the one line on what each published
model is for. It is set on `dispatch`, `nodal`, `network` and `sector`.

`bench/report.py`:

- `models()` writes the table into a new `models` fence, with one column
per model and one row per property. The "for" row is `Case.about`. The
declarations come from the case's `spec.yaml` (`declared()`), shown
without `snapshot`, since every one runs over it. The grouped sums come
from each distinct `sum(x, by=…, over=…, into=…)` (`grouped_sums()`).
The dimensions and the density come from the case's ladder. The
per-snapshot counts are `BASELINE`'s (specsolve's) largest measurement
divided by its snapshots, which is the model as built. `_span` prints to
three figures.
- `fresh()` heads its columns "N columns" rather than "N variables".

`bench/test_harness.py`: how a size prints, which declarations and
grouped sums `dispatch` and `network` give, the table's columns and rows
from the committed results (every model has a "for" line), and the fresh
table's new heading.

**What stays as it is.** The `bench.report` markdown tables that are not
on the page (`results`, `sweeps`) and the long CSV (`bench.tidy`) still
say `variables`; they are tooling output, not this page.

</details>

<details><summary>The memory sentence: numbers, and what they do not
show</summary>

linopy ÷ specsolve peak RSS, HiGHS, from #2031's committed results, at
every size of 100k columns or more:

| model | ratios |
| --- | --- |
| dispatch | 1.07× (100k), 1.00× (1M), 1.19× (10M) |
| nodal | 1.14× (300k), 1.26× (3M) |
| sector | 0.93× (300k), 1.17× (3M) |
| network | 1.62× (100k), **2.78×** (1M: 1.76 vs 0.63 GB) |

`network` is the only published model whose linopy port sums through
`groupby`: lines and stores into buses, and plants into pools. `sector`
sums along a dimension with a coefficient and does not group. On Gurobi
the gap also grows on `dispatch` and `nodal` (2.3–2.6× at their largest
size in the run behind #2039), so the sentence keeps to HiGHS.

What the data does not separate is the line and store sums from the pool
sums. #2024's results, before the pools (#2025), had linopy at 1.6× on
`network` at 1M (1.01 vs 0.65 GB). But #2023 also landed in between,
switching linopy's line and store sums from a dense incidence matrix to
`groupby`. So the sentence says "a grouping", not "the pools". A
`network` case without pool caps would separate the two.

</details>

<details><summary>Verified</summary>

On `51aa16b`, with `main` (#2040) merged in: `pixi run check` (lint,
format, typecheck; 5601 passed, 566 skipped, 4 xfailed), `pixi run -e
docs docs-build` (strict, no issues), `tests/test_docs_site.py` and
`tests/test_doc_examples.py` (123 passed, 4 skipped). `pixi run
test-bench` on `ea38637` (475 passed, 15 skipped); the last commit
changes only the page. `pixi run -e bench report` on the committed
results writes the `models` fence and the fresh table's new heading, and
nothing else.

Not checked in a browser: the chart's axis now reads "columns". The
label comes straight from the chart spec.

</details>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01NTByC5UJJWvcFj3mvmv83v

---------

Co-authored-by: Claude <noreply@anthropic.com>
…ow shows

#2039's results: network 2.70x at 1M columns (1.75 vs 0.65 GB); every
other model at or under 1.26x from 100k columns up, on HiGHS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NTByC5UJJWvcFj3mvmv83v
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants