Skip to content

Align names with task identifiers in eval harnesses - #7

Merged
jonabur merged 3 commits into
OpenEuroLLM:mainfrom
spyysalo:main
Oct 6, 2026
Merged

jonabur merged 3 commits into
OpenEuroLLM:mainfrom
spyysalo:main

Conversation

@spyysalo

@spyysalo spyysalo commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Align catalogue eval names with harness task identifiers, and update the shipped eval sets, browser selectors, regression expectations, and YAML examples to use those names. Matching rules, metrics, normalization, language assignments, and weights are unchanged.

Contributors can run python3 -m tests.check before opening a PR. CI uses the same command: Python/JavaScript contract checks followed by the public browser suite in an automatically managed Chrome session. Missing browser prerequisites fail explicitly. Shared-engine coverage also validates the documented eval-set examples against the shipped catalogue.

Validation:

  • Full local public command passed: 60 Python/shared-engine tests, 103 Node tests, and the public browser suite.
  • Compared all 46 catalogue rules and both shipped sets: only eval names/references differ.
  • Optional private-export regressions have pre-existing failures in test_analysis, test_app, and test_english, reproduced on unchanged base commit 8ec9540; this PR does not resolve those failures.

Documentation: updated the configuration examples, README development command, contributor guidance, and development guide. PR builds validate and upload an artifact on success; only the separate main-branch deployment job publishes GitHub Pages.

@jonabur
jonabur merged commit c2be27a into OpenEuroLLM:main Oct 6, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants