Skip to content

Fix inflated pose recall for incomplete trajectories - #25

Merged
nushakrishnan merged 1 commit into
cvg:mainfrom
Ace3Z:fix/pose-recall-coverage
Oct 7, 2026
Merged

nushakrishnan merged 1 commit into
cvg:mainfrom
Ace3Z:fix/pose-recall-coverage

Conversation

@Ace3Z

@Ace3Z Ace3Z commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

[verified] We found two issues that can inflate pose recall. When an estimate is shorter than GT, associate_trajectories() can return GT twice, so the evaluator compares GT against itself. Unmatched GT poses also disappear from the recall denominator.

This patch keeps the input order and counts all original GT poses in recall. In a controlled sequence_1_19 test, deleting one bad pose previously raised R@5m from 35.08% to 100%; it now stays at 35.08%.

[verified] Added regression tests and checked 21 public scenes (105 pose cases), 33 CP cases, and 13 ATE cases. Complete-trajectory results and CP/ATE scores stayed unchanged. Tests and Ruff pass locally. Private test GT wasn't available.

Joint work by @Ace3Z and @rzninvo.

@Ace3Z Ace3Z changed the title Fix pose recall for incomplete trajectories Fix inflated pose recall for incomplete trajectories Oct 7, 2026
@nushakrishnan

Copy link
Copy Markdown
Collaborator

Thanks! Can you share which method you ran to get incomplete trajectories? I didn't face this with the methods I ran.

@Ace3Z

Ace3Z commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

We created the shorter trajectories for regression testing by subsampling public GT and removing a pose. We didn’t obtain them from a SLAM method we ran. The provided OpenVINS demo does expose the denominator issue through five unmatched GT timestamps, but we haven’t reproduced the shorter-trajectory bug on an unmodified method output yet.

@Ace3Z

Ace3Z commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for checking, @nushakrishnan! I should have been clearer: we created the shorter trajectories for regression testing, and we haven't shown that your method outputs trigger this case. The test reproduces the association returning GT twice when the estimate is shorter, which gives zero errors.

The denominator is a separate question: is recall intended to cover all GT keyframes, or only those with matching estimates? That would help us confirm whether that part should change. Thanks again for taking a look!

@nushakrishnan

Copy link
Copy Markdown
Collaborator

Recall is intended to cover all pGT keyframes so the current implementation is correct, thanks for the fix!

@nushakrishnan
nushakrishnan merged commit 2334108 into cvg:main Oct 7, 2026
1 check passed
@Ace3Z

Ace3Z commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for confirming and merging, @nushakrishnan! Will the existing leaderboard submissions be re-evaluated with the updated evaluator and any affected numbers refreshed, or will this apply only to future evaluations?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants