feat(cli): add geniex eval - compare models on eval packs - #1260
feat(cli): add geniex eval - compare models on eval packs#1260MAN$I VERMA (mansiverma897993) wants to merge 1 commit into
Conversation
8b103ce to
d9b061b
Compare
|
Test report (go1.26.5, windows/amd64 - logic tests; no Snapdragon hardware on my side) 5 tests, all passing: pack parsing (valid + 9 rejection cases), pack resolution (file > built-in > error), prompt rendering, scoring (letter extraction, stripping, choice-text match, normalization), reports (accuracy/tok-s math, table, JSON round-trip). Not covered here: |
d9b061b to
f517120
Compare
|
Alan Zhu (@alanzhuly) Could you take a look? |
Run one or more models over an eval pack (built-in or custom JSON; task types multiple_choice / exact_match / contains) and compare accuracy, per-category breakdown, and decode tok/s, with optional JSON export. Greedy decoding, a fixed seed, thinking disabled, and a context reset per task keep runs repeatable. LLM models only. Closes qualcomm#1070 Signed-off-by: mansiverma897993 <vmansi756@gmail.com>
f517120 to
244e0e0
Compare
|
Alan Zhu (@alanzhuly) RemiliaForever (@RemiliaForever) CI Fails resolved: rebased onto the latest main and fixed a compile break (LlmCreateInput dropped ModelName on main). go test ./internal/eval/... passes, gofmt clean, commit 244e0e0. All CI runs (incl. zizmor) are re-triggered please Approve and run and let me know further changes !! |
2 participants
Closes #1070.
Adds
geniex eval: run one or more models over an eval pack and compare accuracy.Custom packs are JSON (task types:
multiple_choice,exact_match,contains); one built-in smoke pack ships embedded. Runs are repeatable: greedy decoding, fixed seed, thinking off, context reset per task. LLM only for now; wiring reuses the infer plumbing.Alan Zhu (@alanzhuly) happy to rework schema/scoring/UX to fit your direction. Test report in the comment below.