Repository navigation
Test the Gym and ATIF loaders on real data and fix what it exposed - #119
Merged
aleckhoury merged 5 commits intoNov 2, 2026
Merged
Conversation
aleckhoury
force-pushed
the
trace-sessions/08-gym-atif
branch
2 times, most recently
from
November 2, 2026 21:38
3565a9d to
f626875
Compare
aleckhoury
force-pushed
the
trace-sessions/08-gym-atif
branch
2 times, most recently
from
November 2, 2026 21:57
f767586 to
8281c69
Compare
Published NeMo Gym example rollouts (NVIDIA-NeMo/Gym 399e678) and Harbor v0.24.0 ATIF trajectories (Apache-2.0): published Claude Code runs and Qwen Code output with nested subagents. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
Gym's gymnasium agent appends each environment observation to response.output as a role=user message before calling the model again (responses_api_agents/gymnasium_agent/app.py, documented in its README), and the tau2, simulated-user and stirrup agents do the same. The loader rejected every such rollout: 10 of the example_rollouts.jsonl files published at Gym 399e6783 failed. Prompt-role output messages now load in order with unknown authorship. Output that starts with a prompt-role message when the request has input stays rejected as ambiguous history. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
Gym's aviary agent (responses_api_agents/aviary_agent) sets response.contains_transitions and records response.output as a list of transitions, each the full model input after one environment step. The loader expected output items and failed every published aviary file (aviary, aviary_gsm8k, aviary_hotpotqa, aviary_bixbench at Gym 399e6783). The last transition now loads as the conversation when each transition extends the previous one; a transition that rewrites history (aviary's collapse_old_env_states) is rejected. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
A record byte-identical to an earlier one repeats its response id (or ng_trajectory rollout ID), so it is the same rollout recorded twice. Gym publishes such files: graphwalks at 399e6783 has lines 4 and 5 identical (same response id, task and rollout index), and the gymnasium example repeats one row 5 times. The content-hash identity made the whole file fail as a duplicate trace id. The repeat now loads once and GymTraceDescription.duplicate_record_count counts it, rather than becoming a second trace that would double the rollout's evidence. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
ATIF-v1.7 (RFC 0001) makes session_id run-scoped: a trajectory, its continuations and its siblings may share it, and trajectory_id is optional on standalone trajectories. No Harbor root trajectory sets trajectory_id, so the trace id fell back to session_id and two spec-valid documents of one run failed the whole file as a duplicate trace id (two real Harbor Qwen Code outputs sharing session-1, and Harbor's golden linear-history continuation pair). A trajectory without trajectory_id is now identified as "atif-" plus a SHA-256 prefix of its line, like Gym records; session_id stays on Trace.session_id. A document with neither id now loads instead of being rejected. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
aleckhoury
force-pushed
the
trace-sessions/08-gym-atif
branch
from
November 2, 2026 23:02
8281c69 to
26faa39
Compare
nicot
approved these changes
Nov 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gym and ATIF have no session key, so they get no provider PR earlier in the stack; this PR adds their real-data tests and the loader fixes that data exposed.
Real-data tests
tests/test_real_gym.py: publishedexample_rollouts.jsonlfiles from NVIDIA-NeMo/Gym399e678(tests/fixtures/real/gym_responses/): single tool call and multi-reward rollouts, plus the multi-turn gymnasium, aviary and graphwalks files used by the fixes below.tests/test_real_atif.py: Harbor v0.24.0 ATIF trajectories (Apache-2.0,tests/fixtures/real/atif/): published Claude Code runs and Qwen Code output with embedded and nested subagents. The NOTICE Harbor entry now covers this directory too.Loader fixes found on that data (one commit each; the bugs predate this stack)
response.outputbecome user/system turns in order. Output that starts with a prompt-role message after non-empty input is still rejected as ambiguous.response.contains_transitions) load from their last transition; a transition that rewrites history is rejected.describe()["duplicate_record_count"]reports the repeats. The same rollout id with different bytes is still rejected.trajectory_idis identified asatif-<sha256 of the line, 16 hex>instead of by itssession_id, which ATIF v1.7+ allows to be shared by continuations and sibling runs.session_idstays onTrace.session_id.Verified
ruff check,ruff format --check,ty check, the copyright-header check andpytestpass at every tip of this stack (#108–this PR);tests/test_codebase.pywas skipped locally (needs ripgrep).Stack position 8/8