Repository navigation
Fix inflated pose recall for incomplete trajectories - #25
Conversation
|
Thanks! Can you share which method you ran to get incomplete trajectories? I didn't face this with the methods I ran. |
|
We created the shorter trajectories for regression testing by subsampling public GT and removing a pose. We didn’t obtain them from a SLAM method we ran. The provided OpenVINS demo does expose the denominator issue through five unmatched GT timestamps, but we haven’t reproduced the shorter-trajectory bug on an unmodified method output yet. |
|
Thanks for checking, @nushakrishnan! I should have been clearer: we created the shorter trajectories for regression testing, and we haven't shown that your method outputs trigger this case. The test reproduces the association returning GT twice when the estimate is shorter, which gives zero errors. The denominator is a separate question: is recall intended to cover all GT keyframes, or only those with matching estimates? That would help us confirm whether that part should change. Thanks again for taking a look! |
|
Recall is intended to cover all pGT keyframes so the current implementation is correct, thanks for the fix! |
|
Thanks for confirming and merging, @nushakrishnan! Will the existing leaderboard submissions be re-evaluated with the updated evaluator and any affected numbers refreshed, or will this apply only to future evaluations? |
[verified] We found two issues that can inflate pose recall. When an estimate is shorter than GT,
associate_trajectories()can return GT twice, so the evaluator compares GT against itself. Unmatched GT poses also disappear from the recall denominator.This patch keeps the input order and counts all original GT poses in recall. In a controlled
sequence_1_19test, deleting one bad pose previously raised R@5m from 35.08% to 100%; it now stays at 35.08%.[verified] Added regression tests and checked 21 public scenes (105 pose cases), 33 CP cases, and 13 ATE cases. Complete-trajectory results and CP/ATE scores stayed unchanged. Tests and Ruff pass locally. Private test GT wasn't available.
Joint work by @Ace3Z and @rzninvo.