Dense retrievers are meant to select passages that match a query's semantic specification. We show they suffer from search-by-answer bias: they favor passages that mention answer objects the model knows (scores as likely), over passages that support valid alternative answers. We test this with controlled factual contrasts from LAMA/TREx. For each retriever, we measure how strongly the model scores the canonical object over an alternative object using a token-prediction head, and ask whether this score predicts passage rankings. Across eight retrievers, including DPR, Spider variants, Contriever, E5, and Qwen3-Embedding, larger token-prediction scores consistently predict higher retrieval scores for passages containing the corresponding object. This holds when passages differ only in the answer object, and even when the passage that mentions the high-scoring object expresses a wrong relation. Dense retrieval thus exhibits search-by-answer bias during passage selection, shaping which evidence is retrieved.
Figures & tables
Figure 1: Illustration of search-by-answer bias in dense retrieval.
Figure 2: Representative object-contrast trends. The x-axis shows empirical deciles of the object-knowledge score xiR , from D1 (lowest) to D10 (highest). The y-axis shows the retrieval-score difference yiobj,R ; positive values indicate preference for the canonical-object passage. Each violin shows the distribution within one decile. Additional plots appear in Appendix D .
Figure 3: Controlled passage contrasts used to test search-by-answer bias.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Model
object-contrast r
relation–object conflict r
DPR
0.21
0.19
Spider
0.29
0.22
Spider-NQ
0.41
0.30
Spider-Trivia
0.48
0.26
Contriever
0.29
0.13
E5-large
0.45
0.30
Appendix
Table 1: Pearson correlations between the object-knowledge diagnostic xiR and the corresponding retrieval-score difference, computed over the random sample of 1,000 eligible examples for each retriever.
Figure 4: Preference rates across retrievers. In (a), we report the percentage of examples in which the passage containing the model-known object is ranked higher in the object-contrast setting. In (b), we report the percentage of examples in which the wrong-relation passage containing the model-known object is ranked higher than the relation-matched passage containing the alternative object in the relation–object conflict setting. Blue bars correspond to the upper five empirical model-knowledge deciles ( D6 – D10 , the 500 examples with larger xiR ); orange bars correspond to all ten deciles ( D1 – D10 , all 1,000 examples). Dashed black lines indicate the no-bias expectation: 50% in (a) and 0% in (b).
Model
Candidate pool
xiR>0
Selected
DPR
20,000
15,929
1,000
Spider
20,000
15,888
1,000
Spider-NQ
20,000
15,918
1,000
Spider-Trivia
20,000
15,684
1,000
Contriever
20,000
11,734
1,000
E5-large
20,000
14,477
1,000
Appendix
Table 2: Candidate-pool and sampling counts for the main LAMA/TREx experiment. For the MLM-based retrievers, the candidate pool is a fixed seed-13 random subset of 20,000 source examples drawn without replacement from the full 34,039-example pool. Qwen3-Embedding uses the full pool. All candidate examples satisfy the single-token requirement. From the examples with xiR>0 , we randomly sample 1,000 examples per model using seed 13.
Figure 5: Additional object-contrast trends across empirical deciles of model knowledge of the answer. Each violin shows the distribution of retrieval-score differences within one decile. Positive values mean that the retriever assigns a higher score to the canonical-object passage than to the alternative-object passage.
Figure 6: relation–object conflict setting, grouped by empirical deciles of model knowledge of the answer. Examples are sorted by xiR and divided into ten equal-sized groups of 100 examples, from D1 (lowest model preference for the canonical object) to D10 (highest model preference). Each violin shows the distribution of retrieval-score differences within one decile. Positive values mean that the retriever assigns a higher score to the wrong-relation passage containing the canonical object.
Setting
Model
All r
All pref.
Bottom r
Bottom pref.
Top r
Top pref.
object-contrast
DPR
0.21
62.5%
0.11
57.0%
0.14
68.0%
object-contrast
Spider
0.29
66.9%
0.14
56.8%
0.18
77.0%
object-contrast
Spider-NQ
0.41
75.0%
0.19
65.2%
0.32
84.8%
object-contrast
Spider-Trivia
0.48
75.5%
0.16
61.4%
0.33
89.6%
object-contrast
Contriever
0.29
72.3%
0.04
63.6%
0.22
81.0%
object-contrast
E5-large
0.45
91.5%
0.23
87.6%
0.32
95.4%
Appendix
Table 3: Results on the 1,000 examples randomly sampled from the eligible single-token pool with xiR>0 for each retriever. “Bottom” and “Top” denote the lower and upper five empirical deciles, respectively, defined only after sampling. Pearson correlations are computed over individual examples.
Original relation
Alternative relation statement forms kept
applies to jurisdiction
SUBJ is located in OBJ; The headquarter of SUBJ is in OBJ; SUBJ was created in OBJ; SUBJ maintains diplomatic relations with OBJ
field of work
SUBJ is a OBJ by profession; SUBJ has the position of OBJ; SUBJ is a subclass of OBJ; SUBJ is a OBJ
native language
SUBJ wrote in OBJ; SUBJ worked in OBJ; SUBJ published material in OBJ
occupation
SUBJ has the position of OBJ; SUBJ works in the field of OBJ; SUBJ plays in OBJ position; SUBJ is a member of OBJ
employer
SUBJ is a member of OBJ; SUBJ is owned by OBJ; SUBJ works in the field of OBJ; SUBJ is developed by OBJ
owned by
SUBJ is a member of OBJ; SUBJ works for OBJ
Appendix
Table 4: Manually approved relation mappings used to construct the relation–object conflict setting after removing mappings judged too close or invalid. The implementation stores these mappings as templates; here SUBJ denotes the original subject and OBJ denotes the canonical object.
Relation
DPR
Spider
Spider-NQ
Spider-Trivia
Contriever
E5
Qwen3-4B
Qwen3-8B
applies to jurisdiction
0.280 / 78.5%
0.361 / 68.1%
0.481 / 85.3%
0.404 / 80.4%
0.198 / 43.3%
–
0.414 / 100.0%
-0.052 / 97.6%
native language
0.029 / 0.9%
0.347 / 0.1%
0.451 / 0.5%
0.421 / 1.7%
–
0.103 / 79.2%
0.095 / 56.8%
0.148 / 88.1%
occupation
0.402 / 2.5%
0.399 / 0.0%
0.365 / 0.0%
0.458 / 1.3%
–
–
-0.087 / 40.2%
-0.221 / 72.4%
employer
0.148 / 48.9%
–
–
–
–
–
0.279 / 71.4%
0.440 / 97.5%
owned by
0.445 / 96.4%
0.265 / 20.1%
0.199 / 54.9%
0.323 / 23.3%
–
–
0.266 / 56.7%
0.181 / 82.9%
located in administrative territorial entity
-0.135 / 58.7%
0.249 / 91.9%
0.161 / 58.8%
0.251 / 97.8%
–
–
0.161 / 92.3%
0.410 / 98.4%
Appendix
Table 5: Relation-level relation–object conflict results for relation–model pairs with at least 30 examples, restricted to relations used in Table 4 . Each cell reports Pearson correlation / wrong-relation preference rate. Dashes indicate insufficient examples for that relation–model pair.
Figure 7: Shared fixed-example model-size analysis. All models are evaluated on the same 200 examples, selected using E5-large as the reference model with 20 examples from each of its ten object-knowledge bins. The plot shows approximate backbone parameter count against the Pearson correlation between the model-specific object-knowledge diagnostic and the retrieval-score shift in the object-contrast and relation–object conflict settings.
object-contrast
relation–object conflict
Model
Retriever repr.
Original backbone repr.
Retriever repr.
Original backbone repr.
DPR
0.21
0.22
0.19
0.19
Spider
0.29
0.24
0.22
0.24
Spider-NQ
0.41
0.38
0.30
0.31
Spider-Trivia
0.48
0.46
0.26
0.30
Contriever
0.29
0.13
0.13
0.20
Appendix
Table 6: Pearson correlations between the object-knowledge diagnostic and the final retriever’s score difference. “Retriever repr.” computes the diagnostic from the representation of the retrieval-trained model, whereas “Original backbone repr.” computes it from the representation of the corresponding pretrained backbone before retrieval training. Both correlations use the same 1,000 examples and the same final-retriever score differences.
Figure 8: Answer-level recall for known and unknown QAMPARI answers. Unknown answers are counted individually, rather than using an any-unknown-per-question metric.
Figure 9: Capped rank@100 for known and unknown QAMPARI answers. Answers not retrieved in the top-100 are counted as one position below the cutoff, rather than as their true corpus-level rank.
While language models have become dramatically more capable, retrieval still largely relies on lexical matching or similarity in fixed embedding spaces. This creates a capability asymmetry: what the retriever fails to find, even the strongest generator cannot recover. We propose DecisionRAG, which brings the capabilities of pretrained language models to retrieval using decision models that jointly evaluate candidate branches in a single forward pass, without generating any tokens, and show that judging candidates in a shared context makes their relevance estimates comparable. Like a person browsing a library, DecisionRAG navigates the corpus's inherent structure from broad topics down to individual passages, deciding at each level which branches are worth exploring. Without task-specific training or a first-stage retriever, DecisionRAG outperforms dense retrieval with an 8B reranker by 13.8 R@1 points on MultiDoc2Dial and 17.4 Hit@1 points on Natural Questions. Across four benchmarks, including a corpus of 2.68 million Wikipedia paragraphs, retrieval takes just 0.7-1.8 seconds per query.
Chao Wan, Paul Jünger, Beixi Du +3
Department of Computer Science, Cornell University, New York, USA · ASAPP, New York, USA
Dense retrieval embedding models are a fundamental component of modern retrieval-based AI systems. Most dense retrievers are trained with contrastive objectives, which require labeled positive and negative document pairs that are often costly and difficult to obtain. In this work, we investigate whether the autoregressive next-token prediction objective of a large language model (LLM) can provide supervision for dense retrieval. The intuition is simple: if a document contains information relevant to a query, conditioning on that document should make the target output easier for the LLM to predict. A key challenge is that the next-token prediction loss is computed inside the LLM, while the retriever is a separate embedding model. To address this challenge, we propose DREAM (Dense Retrieval Embeddings via Autoregressive Modeling), which injects retriever-generated query-document similarity scores into selected attention heads of a frozen LLM. During training, these scores determine how much attention each candidate document receives while the LLM predicts the target output. The resulting prediction loss provides gradients for retriever training through the attention mechanism. We evaluate DREAM on retrieval benchmarks BEIR and RTEB using embedding backbones ranging from 0.5B to 3B parameters. DREAM consistently outperforms existing baselines across different model scales. These results demonstrate that DREAM provides a promising approach for training dense retrievers through autoregressive modeling.
Yixuan Tang, Yi Yang
The Hong Kong University of Science and Technology
Dense retrieval enables effective semantic matching, yet the semantic factors underlying its relevance scores remain opaque. Sparse autoencoders (SAEs) support feature-level explanations by decomposing dense embeddings into sparse semantic features. However, the document semantics underlying a retriever's relevance scores may not be sufficiently explicit in the embeddings to be recovered as interpretable SAE features. We introduce Xetrieval, a framework for explaining dense retrieval through feature elicitation. Feature Elicitors make retrieval-relevant aspects of document meaning more explicit through auxiliary views, facilitating their identification as interpretable SAE features. A shared sparse dictionary learns from these varied representations to express query--document relations through describable features. We assess their influence on relevance scores through interventions on the original document embeddings. Experiments show that Xetrieval yields more interpretable features than an SAE baseline and identifies feature sets with a stronger influence on the retriever's scores. Task-level feature modulation further demonstrates more effective retrieval steering. These findings highlight the value of feature elicitation for relevance attribution and retrieval steering.
Zhixin Cai, Jun Bai, Yang Liu +7
School of Computer Science and Engineering, Beihang University · State Key Laboratory of General Artificial Intelligence, BIGAI