chore(bench): a rebuild is skipped above a rung where its library ran out of memory, so it cannot take the box down

[FBumann]

Prompt: Also, is the projected memory and time consistent with how the box projects it and kills jobs?

Note

The following content was generated by AI.

test_rebuild builds in the pytest process, and before this PR no memory budget applied to it. Now Ceiling.reached refuses a sink-free cell above a rung where the same library hit a memory ceiling on any sink. In published run 38393347540 that rebuild was the cell closest to a kill.

Evidence: the box's own watchdog log

Job 115282561263 of run 38393347540:

• The box has 31,330 MB. memory-watchdog.sh kills a case under 7,832 MB available, which is about 23.5 GB in use.
• The run's high-water was 16,386 MB in use, during the highs / network step. Before that cell the box sat at about 1.9 GB.
• The largest isolated child in that file was specsolve at network/l, at 2.5 GB. So the 14.5 GB rise came from the pytest process itself.
• The progress dots place the rise on test_rebuild[network-l-linopy]. That cell has five builds of 3.67 s each, and the rise lasted 12:28:59 to 12:29:12.
• In the same file, test_emit had refused linopy above m: "linopy took 1.76 GB on network/m, so the next rung projects to 35.2 GB — over the 16 GB budget".

So the memory budget refused the rung in the isolated pass, where it can do no harm. Then the rebuild built the same rung, unbudgeted, in the process whose death loses the whole results file. That left 7 GB of headroom.

What the next published run loses

On the committed ceilings, two rebuild rows go: linopy at network/l, and the matching gurobi-file row if rebuilds run there. Rebuilds run only with --builds 5 on the highs file, so in practice that is linopy network/l. Pyomo at dispatch/l already stops on its time ceiling. The page's first-vs-steady table then shows that cell as over budget.

Mutation table, and gates

guard	result
if sink: return None: a sink keeps its own ceiling (conftest.py:371-372)	caught (tools.mutate)
the loop over memory ceilings (conftest.py:374-377)	caught (tools.mutate)
and above: the rung the ceiling was recorded at still runs	caught (by hand: the-rung-that-was-measured failed)

tools.mutate ran with CI=1, because under xdist every worker contends for the machine lock in pytest_sessionstart and nothing runs.

• Gates: ruff check ., ruff format --check ., pyrefly check, all of bench/test_harness.py (451 passed, 44 skipped), and tests/test_assertions.py and tests/test_architecture.py. These ran through uv rather than pixi, because pixi is not installed here. The diff does not touch src/.
• Not done, named here only: COPIES_HELD = 2 says the timed rounds and the isolated child hold the model at the same time. pytest-benchmem joins the child before the timed rounds start, so they never do. On network/m the box sat about 3.5 GB above idle against a 1.76 GB child, so the factor of two held in practice, and I left it.
• Also not done: the README says the published run needs 64 GB, but the box is a ccx33 with 31 GB.

🤖 Generated with Claude Code

https://claude.ai/code/session_013mNcjLhrkQM1bzzgSKxYyh

---

Generated by Claude Code

[codspeed]

Merging this PR will not alter performance

✅ 104 untouched benchmarks
⏩ 307 skipped benchmarks¹

---

_{Comparing chore/rebuild-memory-gate (3a8e134) with main (2588883)²}

Footnotes

1. 307 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. No successful run was found on main (7f30e95) during the generation of this report, so 2588883 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩