Course
llm-zoomcamp
Question
Why can MRR improve while Hit Rate stays exactly the same?
Answer
Hit Rate only measures whether at least one relevant document appears in the top-k retrieved results. It does not care about the position of that document.
MRR (Mean Reciprocal Rank), however, rewards systems that rank relevant documents higher. For example:
System A retrieves the correct document at position 5.
System B retrieves the same document at position 1.
Both systems have the same Hit Rate because they successfully retrieved the relevant document. However, System B has a higher MRR because the relevant document appears earlier in the ranking.
This situation often happens after improving reranking, chunking, or embedding quality: the system does not retrieve more relevant documents, but it places them in better positions.
Checklist
Course
llm-zoomcamp
Question
Why can MRR improve while Hit Rate stays exactly the same?
Answer
Hit Rate only measures whether at least one relevant document appears in the top-k retrieved results. It does not care about the position of that document.
MRR (Mean Reciprocal Rank), however, rewards systems that rank relevant documents higher. For example:
System A retrieves the correct document at position 5.
System B retrieves the same document at position 1.
Both systems have the same Hit Rate because they successfully retrieved the relevant document. However, System B has a higher MRR because the relevant document appears earlier in the ranking.
This situation often happens after improving reranking, chunking, or embedding quality: the system does not retrieve more relevant documents, but it places them in better positions.
Checklist