Search-by-Answer Bias in Retrieval Models
Organizations: Bar-Ilan University · NYU Center for Data Science
Abstract
Dense retrievers are meant to select passages that match a query's semantic specification. We show they suffer from search-by-answer bias: they favor passages that mention answer objects the model knows (scores as likely), over passages that support valid alternative answers. We test this with controlled factual contrasts from LAMA/TREx. For each retriever, we measure how strongly the model scores the canonical object over an alternative object using a token-prediction head, and ask whether this score predicts passage rankings. Across eight retrievers, including DPR, Spider variants, Contriever, E5, and Qwen3-Embedding, larger token-prediction scores consistently predict higher retrieval scores for passages containing the corresponding object. This holds when passages differ only in the answer object, and even when the passage that mentions the high-scoring object expresses a wrong relation. Dense retrieval thus exhibits search-by-answer bias during passage selection, shaping which evidence is retrieved.
Figures & tables
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | object-contrast | relation–object conflict |
|---|---|---|
| DPR | 0.21 | 0.19 |
| Spider | 0.29 | 0.22 |
| Spider-NQ | 0.41 | 0.30 |
| Spider-Trivia | 0.48 | 0.26 |
| Contriever | 0.29 | 0.13 |
| E5-large | 0.45 | 0.30 |
| Model | Candidate pool | Selected | |
|---|---|---|---|
| DPR | 20,000 | 15,929 | 1,000 |
| Spider | 20,000 | 15,888 | 1,000 |
| Spider-NQ | 20,000 | 15,918 | 1,000 |
| Spider-Trivia | 20,000 | 15,684 | 1,000 |
| Contriever | 20,000 | 11,734 | 1,000 |
| E5-large | 20,000 | 14,477 | 1,000 |
| Setting | Model | All | All pref. | Bottom | Bottom pref. | Top | Top pref. |
|---|---|---|---|---|---|---|---|
| object-contrast | DPR | 0.21 | 62.5% | 0.11 | 57.0% | 0.14 | 68.0% |
| object-contrast | Spider | 0.29 | 66.9% | 0.14 | 56.8% | 0.18 | 77.0% |
| object-contrast | Spider-NQ | 0.41 | 75.0% | 0.19 | 65.2% | 0.32 | 84.8% |
| object-contrast | Spider-Trivia | 0.48 | 75.5% | 0.16 | 61.4% | 0.33 | 89.6% |
| object-contrast | Contriever | 0.29 | 72.3% | 0.04 | 63.6% | 0.22 | 81.0% |
| object-contrast | E5-large | 0.45 | 91.5% | 0.23 | 87.6% | 0.32 | 95.4% |
| Original relation | Alternative relation statement forms kept |
|---|---|
| applies to jurisdiction | SUBJ is located in OBJ; The headquarter of SUBJ is in OBJ; SUBJ was created in OBJ; SUBJ maintains diplomatic relations with OBJ |
| field of work | SUBJ is a OBJ by profession; SUBJ has the position of OBJ; SUBJ is a subclass of OBJ; SUBJ is a OBJ |
| native language | SUBJ wrote in OBJ; SUBJ worked in OBJ; SUBJ published material in OBJ |
| occupation | SUBJ has the position of OBJ; SUBJ works in the field of OBJ; SUBJ plays in OBJ position; SUBJ is a member of OBJ |
| employer | SUBJ is a member of OBJ; SUBJ is owned by OBJ; SUBJ works in the field of OBJ; SUBJ is developed by OBJ |
| owned by | SUBJ is a member of OBJ; SUBJ works for OBJ |
| Relation | DPR | Spider | Spider-NQ | Spider-Trivia | Contriever | E5 | Qwen3-4B | Qwen3-8B |
|---|---|---|---|---|---|---|---|---|
| applies to jurisdiction | 0.280 / 78.5% | 0.361 / 68.1% | 0.481 / 85.3% | 0.404 / 80.4% | 0.198 / 43.3% | – | 0.414 / 100.0% | -0.052 / 97.6% |
| native language | 0.029 / 0.9% | 0.347 / 0.1% | 0.451 / 0.5% | 0.421 / 1.7% | – | 0.103 / 79.2% | 0.095 / 56.8% | 0.148 / 88.1% |
| occupation | 0.402 / 2.5% | 0.399 / 0.0% | 0.365 / 0.0% | 0.458 / 1.3% | – | – | -0.087 / 40.2% | -0.221 / 72.4% |
| employer | 0.148 / 48.9% | – | – | – | – | – | 0.279 / 71.4% | 0.440 / 97.5% |
| owned by | 0.445 / 96.4% | 0.265 / 20.1% | 0.199 / 54.9% | 0.323 / 23.3% | – | – | 0.266 / 56.7% | 0.181 / 82.9% |
| located in administrative territorial entity | -0.135 / 58.7% | 0.249 / 91.9% | 0.161 / 58.8% | 0.251 / 97.8% | – | – | 0.161 / 92.3% | 0.410 / 98.4% |
| object-contrast | relation–object conflict | |||
|---|---|---|---|---|
| Model | Retriever repr. | Original backbone repr. | Retriever repr. | Original backbone repr. |
| DPR | 0.21 | 0.22 | 0.19 | 0.19 |
| Spider | 0.29 | 0.24 | 0.22 | 0.24 |
| Spider-NQ | 0.41 | 0.38 | 0.30 | 0.31 |
| Spider-Trivia | 0.48 | 0.46 | 0.26 | 0.30 |
| Contriever | 0.29 | 0.13 | 0.13 | 0.20 |
Dense Retrieval
Dense retrieval uses learned vector representations to find relevant documents, passages, products, or tools by semantic similarity.
See all 160 papers