Skip to content

MT-Bench-101 (1/2): Declare MT-Bench-101 task and dataset adapter - #119

Merged
ErlisLushtaku merged 3 commits into
mainfrom
mt-bench-101/01-task-declaration
Sep 29, 2026
Merged

ErlisLushtaku merged 3 commits into
mainfrom
mt-bench-101/01-task-declaration

Conversation

@ErlisLushtaku

@ErlisLushtaku ErlisLushtaku commented Sep 9, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This is the first PR in the MT-Bench-101 stack. It declares the task and adds the dataset adapter; generation and judging are in #120.

Changes

  • Add a packaged mt-bench-101 task YAML with the upstream dataset pinned to commit bc18b3e2c18c99164e11528f1a79c92083db5953.
  • Expand the source dialogues into one row per evaluated turn while retaining the golden context. Tasks that only become meaningful after the first turn start at turn 2, following the reference implementation.
  • Add the typed protocol schema and register the dataset adapter and runner route.
  • Package the task-specific judge prompts used by the OpenCompass fork, with the typo and formatting corrections carried over from closed feat: add support for mt-bench-101 #22.

References

The task structure and prompts come from the MT-Bench-101 repository, the OpenCompass fork, and Bai et al. (2024).

@ErlisLushtaku
ErlisLushtaku marked this pull request as ready for review September 9, 2026 13:50
@kargibora
kargibora force-pushed the rebase/inference-usage-on-meta-eval branch from 58d6791 to d1ca639 Compare September 10, 2026 09:55
kargibora added a commit that referenced this pull request Sep 14, 2026
@ErlisLushtaku ErlisLushtaku changed the title WIP (1/2): Declare MT-Bench-101 task and dataset adapter MT-Bench-101 (1/2): Declare MT-Bench-101 task and dataset adapter Sep 15, 2026
@ErlisLushtaku
ErlisLushtaku changed the base branch from rebase/inference-usage-on-meta-eval to main September 15, 2026 08:02
MultiTurnGeneration,
)
from judgearena.tasks.schema.mt_bench_101 import (
GoldenContextGeneration,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is GoldenContext?

@ErlisLushtaku ErlisLushtaku Sep 18, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It means each turn is generated using the preceding reference dialogue as fixed context, rather than using the previously generated turns from the LLM. I added this to the schema.

@kargibora
kargibora force-pushed the mt-bench-101/01-task-declaration branch from e5d0f3d to 7b6c69a Compare September 15, 2026 10:36
@kargibora

Copy link
Copy Markdown
Member

Looks good to me as it is just the definitions. IT is a clean implementation!



class GoldenContextGeneration(StrictFrozenModel):
"""Generate each turn using the preceding reference dialogue as context."""

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for clarifiyng!

@ErlisLushtaku
ErlisLushtaku merged commit 0b82e77 into main Sep 29, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants