Repository navigation
Add evaluation tests and run them in CI - #27
nushakrishnan wants to merge 9 commits into
Conversation
|
Done, when fewer than three control points with height triangulate, the evaluation now saves a result with no alignment and reports CP score and recall of 0 with exit code 0. |
Thanks. For the evaluation wrt pGT, when the CP alignment fails, should we still try pGT alignment instead of directly assigning zero? |
Our standard alignment comes from the control points, so if an alignment from that isn't found then I don't think it makes sense to do separate pGT alignment. The pGT alignment is only for sequences without CPs. |
|
Assuming that one has an output that covers a span of only two control points (say out of ten), the pGT recall should probably not be zero but somewhere around 20%. The point of CP alignment for pGT evaluation is just to find a good transformation for it. |
Partially evaluating a trajectory in that case does make sense even though the CP alignment fails. Let me update it accordingly. It goes against our current eval strategy but it makes sense to give a partial pose recall and 0 for CP score. |
The tests use a small synthetic scene (a stereo rig moving past a set of control points, with the estimate in a different Sim(3) frame), so no dataset download is needed.
Covered:
.npyround trip (relies on Fix loading saved evaluation results and script exit codes #24) and the saved alignment consumed byevaluate_wrt_pgt.CI: new
tests.ymlworkflow runspython -m pytest testson Python 3.10 and 3.12, installing only the evaluation dependencies. Also adds atestpathsentry topyproject.tomland a short "Running the tests" section to the README.