Repository navigation
commonroad: regenerate the IDM demo fixtures with real checker verdicts - #129
Merged
Merged
Conversation
The straight_* demo fixtures are the example planner's output plus the official checker's verdict on it. The checker ships manylinux_x86_64 wheels only, so a real verdict needs a Linux x86_64 runner. This job runs the example in both modes, judges both solutions and uploads the six files, so the fixtures can be regenerated without a local checker.
The six straight_* fixtures predate the fix that starts the example's
solution at the planning problem's initial state, so both of them failed
the official starts_at_correct_state. They are now the current example's
output, judged by the official checker on an ubuntu runner with
commonroad-io 2024.3, commonroad-drivability-checker 2025.4.0 and
triangle 20250106.
idm 7/7 PASS
naive 6/7, obstacle_collision FAIL at t=37..44 (obstacle 2), which is
the point of the mode: it ignores the car ahead
Both solutions now start at time step 0 with 181 states, and both traces
carry the fingerprints of the committed solution and scenario.
The fixtures are now a plain rerun of the example, so the comparison is state for state instead of "the fresh solution's tail from step 1", and starts_at_correct_state is expected to PASS for both modes. naive is pinned to exactly one FAIL, obstacle_collision. The rule test that proved the official check rejects a 1-based solution used the committed fixture as its rejected shape. It now builds that shape itself, by dropping the first state of a fresh solution, so the check keeps being proved against something it must reject.
The demo tables still described the old fixtures. Both READMEs now read idm PASS 7/7 and naive FAIL 1/7 (obstacle collision at 3.7 s, intended), and the example README replaces its four-check summary with the real seven-check outcome on the straight road. ATTRIBUTION records the workflow run that produced the files and the checker versions it used, and drops the note saying the fixtures are deliberately kept as the older 1-based output.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
straight_*demo fixtures predated the fix that starts the example'ssolution at the planning problem's initial state, so both of them failed the
official
starts_at_correct_stateand the demo tables advertised that failure.They are now the current example's output with the official checker's verdict
on it.
How. The checker (
commonroad-drivability-checker) publishesmanylinux_x86_64 wheels only, so a real verdict cannot be produced on macOS or
ARM. This PR adds
.github/workflows/commonroad-regenerate-fixtures.yml, aworkflow_dispatch-only job that runs the example in both modes on an ubunturunner, judges both solutions and uploads the six files. The committed fixtures
are that run's artifact:
https://github.com/kosuke55/drawtonomy/actions/runs/35613419629
commonroad-io 2024.3, commonroad-drivability-checker 2025.4.0, triangle
20250106 - the same versions the previous verdicts were produced with.
The new statuses.
idmnaiveobstacle_collisionat t=37..44 (obstacle 2), the intended collision demoBoth solutions now start at time step 0 with 181 states, and each trace carries
the
solutionFingerprint/scenarioFingerprintthe writer computed during therun.
Tests.
test_committed_straight_fixtures_are_what_the_example_writesnowcompares a fresh run state for state instead of "the fresh solution's tail from
step 1", and expects
starts_at_correct_stateto PASS for both modes, withnaivepinned to exactly one FAIL. The test proving the official rule rejects a1-based solution used the committed fixture as its rejected shape; it now builds
that shape itself by dropping the first state of a fresh solution, so the rule
keeps being proved against something it must reject.
213 passed, 1 skipped locally (the skip is the checker-only verdict-shape test,
which the runner covers).