Agent Learning Track baseline run (GPT-5.1, OpenAI API judge+sim) — results + gap vs leaderboard

[vishal-dys]

Summary

We ran the Agent Learning Track no-memory baseline using GPT-5.1 as the agent and GPT-5.4 for both the locked judge and user simulator, following the Agent Learning Track protocol. The only deviation from the documented setup is that we used the OpenAI API directly rather than Azure OpenAI for the judge and simulator client. Our results are ~6pp below the leaderboard's verified GPT-5.1 (no-memory) entry, and we wanted to share the data and ask whether the gap is expected.

Run Configuration

Setting	Value
Track	Agent Learning Track
Agent model	GPT-5.1 (OpenAI API)
Judge model	GPT-5.4 (OpenAI API)
Simulator model	GPT-5.4 (OpenAI API)
Judge reasoning effort	high
Protocol	gpt54.json (locked)
--num-runs	5
--retrieve-learnings-top-k	3
Arm	Baseline (no memory / no retrieve_learnings)

Baseline Results

Domain	Pass@1	Pass^5	UX Score
customer_support	50.4%	36.0%	3.582
shopping_assistant	44.8%	30.0%	3.532
travel	46.4%	20.0%	3.302
Macro Average	47.2%	28.7%	3.472
Leaderboard (GPT-5.1, no-memory, Verified)	53.3%	32.7%	3.87

Question

Is the ~6pp Pass@1 gap and ~0.4 UX gap expected when running the judge and simulator via the OpenAI API rather than Azure OpenAI? We want to confirm whether Azure vs OpenAI API routing for the locked eval client is a known source of variance before submitting our memory-augmented results.

We note our UX scores are now real per-dimension values (not defaulting to 3.0), so the judge is functioning correctly via OpenAI API. The gap appears to be consistent across all three domains rather than isolated to one, which makes us wonder if there is a systematic difference in judge/simulator behavior between Azure and OpenAI API endpoints for GPT-5.4.

Any guidance would be appreciated. Happy to share more details or trajectories if useful.

[nisyad-ms]

Thank you for sharing the detailed summary. The numbers reported in the Agent Learning Track are from an older benchmark version which had a different evaluation protocol. We have now added a benchmark version tag to clarify where each set of results are coming from. We will re-run those numbers on the latest version in the coming days (and retire older versions).

I also ran GPT 5.1 (reasoning effort=None) with the latest version and the results align with your numbers closely so we can be confident there is no difference between Azure and OpenAI endpoints. There is also an expected standard deviation of < 4% in the results. While sharing your results, please also share the std dev numbers.

[vishal-dys]

Thank you for the clarification and noted on the std dev numbers to be added.