Sheet 2 ·How well does it work?
The passage holding the answer was in the top 5 for 84% to 99% of 105 questions, depending on the settings; weakest: BM25 · small.
Does the search find the answer?
recall@5: the share of 105 answerable questions whose answer-holding passage was among the top 5 retrieved. The model answers only from what it is given, so this caps everything after.
MRR rewards finding it early: 1 for the top passage, ½ for second, 0 outside the top 10.
Measure
| Search | Reranker off | Reranker on |
|---|---|---|
| BM25 | recall@5 84%, over 105 questions | recall@5 97%, over 105 questions |
| Vector | recall@5 89%, over 105 questions | recall@5 98%, over 105 questions |
| Hybrid | recall@5 92%, over 105 questions | recall@5 99%, over 105 questions |
| Search | Reranker off | Reranker on |
|---|---|---|
| BM25 | recall@5 93%, over 105 questions | recall@5 99%, over 105 questions |
| Vector | recall@5 90%, over 105 questions | recall@5 99%, over 105 questions |
| Hybrid | recall@5 96%, over 105 questions | recall@5 99%, over 105 questions |
| Search | Reranker off | Reranker on |
|---|---|---|
| BM25 | recall@5 94%, over 105 questions | recall@5 99%, over 105 questions |
| Vector | recall@5 91%, over 105 questions | recall@5 98%, over 105 questions |
| Hybrid | recall@5 97%, over 105 questions | recall@5 99%, over 105 questions |