Fail the job when an evaluation produces no result

[haideraltahan]

The job script ignores the exit status of every evaluation it runs, so anything that goes wrong inside an array element still ends with Evaluation finished and a zero exit code. The element shows up as COMPLETED 0:0 with no result file, and the loss only surfaces later as a gap in collect --check.

I hit three variants of this on LUMI while testing the README quickstart, all of which looked like successful runs:

FATAL: container creation failed: ... mount source ... doesn't exist
/var/spool/slurmd/job.../slurm_script: line 174: .../bin/activate: Permission denied
OSError: [Errno 122] Disk quota exceeded: '/users/<me>/.cache/huggingface/lighteval/...'

The last one cost one of four evaluations in a run I would otherwise have recorded as complete.

The change

Each branch records the harness exit status and where it was told to write. An evaluation counts as done only if the harness exited 0 and produced a file — a container that fails to start and a missing venv both exit 0 while writing nothing, so the status alone is not enough. Failures are counted and become the job's exit status:

[error] belebele_deu_Latn (lm-eval-harness) exited 0 but wrote no result to .../020d75d501*.json
Job 1 finished with 1 failed evaluation(s).

Two knock-on changes:

• The CSV is redirected into the read loop rather than piped, because a pipeline runs the loop body in a subshell and the counter would be discarded at done.
• The evalchemy branch's cd is now in a subshell. It relied on the loop being a subshell to contain it, which is no longer true.

--output_path for lm_eval is now built from a variable so the result check can look for the file it actually writes (lm_eval appends its own timestamp).

Also raise SystemExit(1) when sbatch rejects the submission — it currently logs an error and returns normally, so a scheduling loop cannot tell the submission failed.

Tests

tests/test_job_failure_reporting.py renders a real script through schedule_evals and runs it against a stub harness on PATH, so it exercises the template rather than a copy of its logic:

• a crashing harness fails the job
• a harness that exits 0 without writing fails the job
• a working harness still exits 0 and reports the evaluation as finished
• both rows of a two-evaluation job are counted (the subshell case)
• a rejected sbatch exits non-zero

Four of the five fail on main; the happy-path one passes before and after. Rest of the suite passes (122, excluding the HuggingFace-backed ones), ruff check and ruff format clean.

Overlap

#107 makes the same loop change for the same reason. That branch is a much larger merge, so this is the standalone version; happy to close it if #107 is landing soon.