Hi — while analyzing the published ReAct baseline submissions I noticed the gemini-2.5-flash baseline is dominated by empty answers, and wanted to flag it in case it's unintended (or in case leaderboard readers should be aware).
Finding: in submissions/react_gemini-2.5-flash.json, 1,617 of 2,700 runs (59.9%) have an empty answer field (""). For comparison, across the other ReAct baselines:
┌────────────────────────┬─────────┬───────┬─────────┐
│ submission │ records │ empty │ empty % │
├────────────────────────┼─────────┼───────┼─────────┤
│ react_gemini-2.5-flash │ 2700 │ 1617 │ 59.9% │
├────────────────────────┼─────────┼───────┼─────────┤
│ react_kimi-k2-thinking │ 2600 │ 220 │ 8.5% │
├────────────────────────┼─────────┼───────┼─────────┤
│ react_gemini-3-pro │ 2700 │ 64 │ 2.4% │
├────────────────────────┼─────────┼───────┼─────────┤
│ react_gpt-5-mini │ 2700 │ 25 │ 0.9% │
├────────────────────────┼─────────┼───────┼─────────┤
│ react_gpt-5.2 │ 2700 │ 4 │ 0.1% │
└────────────────────────┴─────────┴───────┴─────────┘
The Flash empties are spread across all 12 datasets (per-dataset empty rate ranges 31%–87%: agnews 87%, bookreview 80%, stockindex 77%, GITHUB_REPOS 76%, stockmarket 74%, PATENTS 73%, PANCANCER_ATLAS 71%, DEPS_DEV_V1 64%, yelp 53%, googlelocal 50%, crmarenapro 40%, music_brainz_20k 31%) and roughly uniform across the run index 0–49 (~32 ± 8 empties per run, range 24–40). That pattern looks systematic (pipeline-wide) rather than question-specific.
Why it may matter: the official validator returns is_valid: False for llm_answer == "", and accuracy.py counts that as 0 (failure) toward Pass@1, so this baseline's Pass@1 largely reflects non-response rather than incorrect answers — i.e., "the agent produced no final answer" is being counted the same as "the agent answered wrongly." From the submission alone I can't tell whether the empties are from rate-limits/timeouts or the ReAct harness not extracting a final answer from the smaller model (the submission records carry only dataset/query/run/answer — no error or termination reason).
Repro:
import json
d = json.load(open("submissions/react_gemini-2.5-flash.json"))
empty = [r for r in d if not (r.get("answer") or "").strip()]
print(f"{len(empty)}/{len(d)} = {100*len(empty)/len(d):.1f}% empty")Questions:
- Is the high empty rate for this baseline expected/known?
- Should non-responses be retried, scored as a distinct state, or excluded from Pass@1 so they aren't conflated with wrong answers?
- Minor:
react_kimi-k2-thinking shows 8.5% empty (also across all datasets) — same underlying cause?
Happy to share the per-(dataset, run) breakdown if useful.
Hi — while analyzing the published ReAct baseline submissions I noticed the gemini-2.5-flash baseline is dominated by empty answers, and wanted to flag it in case it's unintended (or in case leaderboard readers should be aware).
Finding: in
submissions/react_gemini-2.5-flash.json, 1,617 of 2,700 runs (59.9%) have an empty answer field (""). For comparison, across the other ReAct baselines:The Flash empties are spread across all 12 datasets (per-dataset empty rate ranges 31%–87%: agnews 87%, bookreview 80%, stockindex 77%, GITHUB_REPOS 76%, stockmarket 74%, PATENTS 73%, PANCANCER_ATLAS 71%, DEPS_DEV_V1 64%, yelp 53%, googlelocal 50%, crmarenapro 40%, music_brainz_20k 31%) and roughly uniform across the run index 0–49 (~32 ± 8 empties per run, range 24–40). That pattern looks systematic (pipeline-wide) rather than question-specific.
Why it may matter: the official validator returns
is_valid: Falseforllm_answer == "", andaccuracy.pycounts that as0(failure) toward Pass@1, so this baseline's Pass@1 largely reflects non-response rather than incorrect answers — i.e., "the agent produced no final answer" is being counted the same as "the agent answered wrongly." From the submission alone I can't tell whether the empties are from rate-limits/timeouts or the ReAct harness not extracting a final answer from the smaller model (the submission records carry onlydataset/query/run/answer— no error or termination reason).Repro:
Questions:
react_kimi-k2-thinkingshows 8.5% empty (also across all datasets) — same underlying cause?Happy to share the per-(dataset, run) breakdown if useful.