Repository navigation
Did the groundtruth leak? #28
Description
Dear authors,
Some time ago I saw a "test1" submissions as shown also here: #21
It has now disappeared, it had few submitted sequences (thus a low total score), but the sequences that were submitted were very close to a perfect score with many being 100.0 when not even the top scoring Aria's SLAM achieves 100.0 in any test sequence.
I took a screenshot at the time of the "short" test sequences:
I should've opened the issue earlier, the scores look a bit too good to be true and made me doubt whether there may have been unintended access to ground-truth information. Additionally, the test1 submission was not marked "(r)" meaning it didn't come from your side.
Could you clarify whether this was a test submission from your side using the groundtruth but a different account?
P.S., the contact form does not seem to be working.
Activity
nushakrishnan commentedon Oct 8, 2026
Hello! No the ground-truth is not leaked. To identify the fix from this PR #22, the users submitted test trajectories to the evaluation server to validate their claim. Following the fix, they requested to remove these test submissions since it was used for debugging the erroneous evaluation. The server side evaluation script has been consequently fixed. We also maintain a changelog on the leaderboard page now for documenting issues and changes to the website code.
Regarding the contact form, what error are you facing? It works for me.
mateosss commentedon Oct 8, 2026
Very nice! Thank you for the clarification.
Regarding the contact form
I cannot get the captcha to be accepted by the form, maybe I'm just a robot :)
nushakrishnan commentedon Oct 8, 2026
Hope that helps :)