Dense retrievers play a vital role in accessing external and specialized knowledge to augment language models (LMs). Training dense retrievers typically requires annotated query-document pairs, which are costly to create and scarce in specialized domains (e.g., code) or in complex settings (e.g., requiring reasoning). These practical challenges have sparked growing interest in self-supervised retriever learning. Since LMs are trained to capture token-level dependencies through a self-supervised learning objective (i.e., next token prediction), we can analogously cast retrieval as learning dependencies among chunks of tokens. This analogy naturally leads to the question: How can we adapt self-supervised learning objectives in the spirit of language modeling to train retrievers? To answer this question, we introduce Revela, a unified and scalable training framework for self-supervised retriever learning via language modeling. Revela models semantic dependencies among documents by conditioning next token prediction on local and cross-document context through an in-batch attention mechanism. This attention is weighted by retriever-computed similarity scores, enabling the retriever to be optimized as part of language modeling. We evaluate Revela on domain-specific (CoIR), reasoning-intensive (BRIGHT), and general-domain (BEIR) benchmarks across various retriever backbones. Without annotated or synthetic query-document pairs, Revela surpasses larger supervised models and proprietary APIs on both CoIR and BRIGHT. It achieves BEIR's unsupervised SoTA with ~1000x less training data and 10x less compute. Performance increases with batch size and model size, highlighting Revela's scalability and its promise for self-supervised retriever learning.
Figures & tables
Figure 1: The framework of . The retriever’s in-batch similarity scores are used as in-batch attention weights inside transformer blocks. The retriever is trained by optimizing the language modeling objective, i.e., NTP. The related patterns in red and purple sequences are highlighted in bold and underline . An example of training dynamics is illustrated at App. A .
Figure 2: ’s architecture. With an attention map, the embeddings of in-batch attention {hil}i=1B can attend to the self-attention {eil}i=1B .
Dataset
BM25
UniX
E5-PT †
BGE †
REPLUG
REPLUG
E5-Mistral †
Ada-2
Voyage-2
Model Size
–
0.1B
0.1B
0.3B
0.5B
0.6B
1B
1B
3B
3B
7B
–
–
Apps
4.8
1.4
8.2
10.6
20.5
14.7
13.9
19.4
17.7
26.6
23.5
8.7
26.5
CosQA
15.6
25.1
26.2
27.1
27.5
26.4
20.1
30.2
25.2
29.0
33.2
28.9
29.8
ST2SQL
25.0
50.5
45.7
48.9
53.7
46.9
53.9
55.0
56.8
55.9
68.0
58.3
69.3
SN
40.9
60.2
49.9
35.2
57.9
58.3
50.8
64.0
53.7
62.6
67.4
74.2
81.8
SNCC
54.0
58.4
63.4
50.5
68.0
53.7
62.8
70.0
62.8
69.1
64.8
53.3
72.8
Table 1: Performance on CoIR (nDCG@10, %). Gray indicates supervised models. Bold marks the highest score among non-API models in each row. Columns marked † used code-related pairs during pre-training. The results of APIs are collected from Li et al. (2025) . Without query-document pairs, 3B surpasses larger supervised models and proprietary APIs, averaged across 10 tasks.
Figure 3: Performance on BRIGHT (left) and BEIR (right) (nDCG@10, %). Results for are shown in opaque bars, while all other models are represented by transparent bars. On BRIGHT, 3B surpasses E5-Mistral, a supervised retriever with more parameters, and properties APIs. On BEIR, achieves similar performance with E5-PT with much less data and compute. Please refer to Tab. 7 and Tab. 8 in § B.6 for the per-task results.
Model
BEIR
CoIR
AVG
-wiki 1B
42.7
53.2
48.0
Contriever-wiki 1B
42.4
50.3
46.4
-code 1B
39.6
58.6
49.1
Contriever-code 1B
32.3
52.1
42.2
Table 2: vs. Contriever Performance.
Figure 4: Performance comparison on CoIR and BEIR with different batch sizes . For both benchmarks, performance generally scales with batch size.
Figure 5: Performance comparison on CoIR and BEIR using various combinations of retrievers and LMs . For code retrieval tasks, larger LMs can yield greater gains in retriever performance.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Example of training dynamics of . The related patterns in red and purple sequences are highlighted in bold , underline , and italic .
Sub-Tasks (Abbr.)
Descriptions
AppsRetrieval (Apps) a
Retrieve code snippets based on natural language queries.
CosQA (CosQA) a
Find code snippets that answer web search queries.
SyntheticText2SQL (ST2SQL) a
Retrieve SQL queries based on natural language questions.
COIRCodeSearchNetRetrieval (SN) b
Retrieve explanations or summaries for code snippets.
CodeSearchNetCCRetrieval (SNCC) c
Identify code snippets that are similar to a given one.
CodeTransOceanContest (TransC) c
Retrieve code solutions based on contest problems.
Appendix
Table 3: CoIR Benchmark Tasks. The superscripts present the type of the tasks. The abbreviation of the tasks is noted in the parentheses, presented in Table 1 .
Abbrev.
Description
Bio.
StackExchange Biology questions with external web evidence
Earth.
StackExchange Earth Science questions with linked resources
Econ.
StackExchange Economics questions requiring theoretical support
Psy.
StackExchange Psychology questions with cited references
StackOverflow programming questions with linked web pages
Appendix
Table 4: BRIGHT Benchmark Tasks. Abbreviations and descriptions.
Figure 7: Example of one batch containing chunks split from Wikipedia: This batch contains chunks from two topics: Alcohol ( blue ) and Achill Island topic ( green ).
Name
URL
License
UniXcoder (UniX)
microsoft/unixcoder-base
Apache-2.0
Contriever
facebook/contriever
Not specified
RetroMAE
Shitao/RetroMAE
Not specified
GraphCodeBERT
microsoft/graphcodebert-base
Not specified
LaPraDoR
canwenxu/laprador
Apache-2.0
E5-large-unsupervised (E5-PT large )
intfloat/e5-large-unsupervised
MIT
Appendix
Table 5: Baseline retrievers, LMs (’s backbone), CodeRAG-Bench datasets, and evaluation benchmarks with their HuggingFace URLs and licenses.
Table 6: Embedding models and their reference URLs.
Dataset
BM25
Contriever
LaPraDor
E5
REPLUG
REPLUG
REPLUG
Revela-code
Model Size
–
0.1B
0.1B
0.1B
0.3B
0.5B
0.5B
1B
1B
3B
3B
3B
ArguAna
48.7
44.3
44.6
39.0
44.4
36.4
41.1
39.4
44.6
38.0
45.3
44.3
ClimateFEVER
13.6
7.2
12.2
15.3
15.7
16.2
13.8
15.0
15.8
13.0
16.6
18.6
DBPedia
29.9
27.0
25.0
18.3
37.1
18.0
21.3
19.5
27.6
23.4
27.1
30.9
FEVER
48.1
27.2
33.6
33.7
68.6
50.4
51.1
51.6
61.7
54.3
62.9
66.3
FiQA2018
25.1
12.4
19.8
19.2
43.2
19.9
27.2
22.7
30.4
24.8
34.3
33.2
Appendix
Table 7: Performance of unsupervised/self-supervised retriever models on BEIR datasets (nDCG@10, %). Bold marks the best score per dataset among unsupervised methods.
Dataset
BM25
E5-PT
REPLUG
E5-Mistral
Cohere
OpenAI
Voyage
Model Size
–
0.3B
0.5B
1B
3B
3B
7B
–
–
–
Biology
18.9
18.5
16.9
16.7
6.8
24.9
18.6
18.7
23.3
23.1
Earth Science
27.2
29.5
29.0
29.4
13.0
40.3
26.0
28.4
26.7
25.4
Economics
14.9
10.2
13.9
13.2
12.9
17.2
15.5
20.4
19.5
19.9
Psychology
12.5
17.6
13.3
15.0
15.1
21.4
15.8
21.6
27.6
24.9
Robotics
13.6
7.1
9.1
10.5
7.4
15.3
16.3
16.3
12.8
10.8
Appendix
Table 8: Performance on BRIGHT (nDCG@10, %). Bold marks the best performance. The results of BM25, E5-Mistral and APIs are taken from BRIGHT ( Hongjin et al., 2025 ) .
Dataset
-wiki 1B
Contriever-wiki 1B
-code 1B
Contriever-code 1B
Apps
9.1
11.3
19.4
24.4
CosQA
22.6
24.3
30.2
32.4
ST2SQL
55.8
46.3
55.0
44.8
SN
54.2
49.8
64.0
41.5
SNCC
69.2
57.4
70.0
52.3
TransC
74.4
76.3
81.1
79.6
Appendix
Table 9: Performance on CoIR (nDCG@10, %). Bold marks the best score among the models.
Dataset
-wiki 1B
Contriever-wiki 1B
-code 1B
Contriever-code 1B
ArguAna
44.6
48.9
46.9
42.0
ClimateFEVER
15.8
29.0
14.9
13.7
DBPedia
27.6
31.6
21.8
14.6
FEVER
61.7
68.6
51.4
41.2
FiQA2018
30.4
27.7
33.3
31.7
HotpotQA
56.0
47.1
46.4
17.6
Appendix
Table 10: Performance of unsupervised/self-supervised retriever models on BEIR datasets (nDCG@10, %). Bold marks the best score per dataset.
0.1B
0.5B
1B
Dataset
bs4
bs8
bs16
bs4
bs8
bs16
bs4
bs8
bs16
AppsRetrieval
4.8
6.4
8.2
11.8
16.5
20.5
17.1
16.2
19.4
CosQA
25.0
25.4
26.2
27.5
28.4
27.5
28.0
28.1
30.2
SyntheticText2SQL
42.8
44.0
45.7
50.9
52.5
53.7
53.4
51.4
55.0
COIRCodeSearchNetRetrieval
39.5
46.5
49.9
57.0
52.9
57.9
59.3
57.9
64.0
CodeSearchNetCCRetrieval
56.0
60.4
63.4
62.5
63.7
68.0
69.0
69.2
70.0
Appendix
Table 11: Retrieval performance (nDCG@10, %) across the 10 CoIR tasks. We report three encoder sizes (135M, 500M, 1B) and three batch sizes ( bs4 , bs8 , bs16 ).
0.1B
0.5B
1B
Dataset
bs4
bs8
bs16
bs4
bs8
bs16
bs4
bs8
bs16
ArguAna
36.7
38.5
39.0
35.4
41.0
41.1
43.0
47.9
44.6
ClimateFEVER
14.0
15.5
15.3
13.6
15.2
13.8
16.4
14.0
15.8
DBPedia
15.5
15.7
18.3
23.1
20.8
21.3
25.5
20.7
27.6
FEVER
27.0
29.5
33.7
47.3
46.1
51.1
57.7
51.2
61.7
FiQA2018
17.4
18.2
19.2
24.8
27.8
27.2
29.4
28.8
30.4
Appendix
Table 12: Retrieval performance (nDCG@10, %) across 13 BEIR tasks for three encoder sizes and three batch sizes ( bs4 , bs8 , bs16 ).
0.1B
0.5B
1B
Dataset
0.1B
0.5B
1B
0.1B
0.5B
1B
0.1B
0.5B
1B
ArguAna
36.9
39.9
39.0
38.5
41.8
41.1
41.1
43.0
44.6
ClimateFEVER
14.1
16.3
15.3
18.3
20.0
13.8
15.0
15.8
15.8
DBPedia
17.6
17.0
18.3
24.2
22.7
21.3
25.0
26.4
27.6
FEVER
25.5
31.3
33.7
53.0
55.7
51.1
58.7
56.7
61.7
FiQA2018
16.9
19.3
19.2
26.6
24.9
27.2
29.3
28.4
30.4
Appendix
Table 13: nDCG@10 (%) on BEIR for three model sizes ( 0.1B , 0.5B , 1B ), with LM sizes in ascending order. Each triplet of columns highlights the highest value in bold .
0.1B
0.5B
1B
Task
0.1B
0.5B
1B
0.1B
0.5B
1B
0.1B
0.5B
1B
AppsRetrieval
5.9
5.2
8.2
18.2
15.0
20.5
17.9
15.7
19.4
CosQA
24.8
24.6
26.2
26.7
28.3
27.5
28.7
30.2
30.2
SyntheticText2SQL
43.9
46.2
45.7
50.8
52.0
53.7
52.1
53.0
55.0
COIRCodeSearchNetRetrieval
40.0
40.0
49.9
51.2
48.0
57.9
57.2
55.2
64.0
CodeSearchNetCCRetrieval
54.7
56.2
63.4
61.5
61.3
68.0
62.6
64.1
70.0
Appendix
Table 14: nDCG@10 (%) on CoIR for three model sizes ( 0.1B , 0.5B , 1B ), with LM sizes in ascending order. Bold in each row indicates the best performance for each retriever size.
Dataset
0.1B
0.5B
1B
AppsRetrieval
5.9
17.2
17.5
CosQA
25.8
28.0
24.5
SyntheticText2SQL
46.0
51.5
55.1
COIRCodeSearchNetRetrieval
41.1
51.5
58.3
CodeSearchNetCCRetrieval
59.3
62.7
66.6
CodeTransOceanContest
66.3
76.4
78.6
Appendix
Table 15: Comparison of ’s performance (nDCG@10, %) across CoIR benchmark tasks for three LM sizes trained on the mixture of Wikipedia and code-related corpus. Bold indicates the best performance in each row.
Dataset
0.1B
0.5B
1B
ArguAna
41.4
43.9
44.2
ClimateFEVER
16.2
16.1
15.3
DBPedia
18.8
21.7
25.4
FEVER
35.9
44.9
56.8
FiQA2018
21.2
28.3
34.6
HotpotQA
37.0
50.3
59.1
Appendix
Table 16: Comparison of ’s performance (nDCG@10, %) across BEIR benchmark tasks for three LM sizes trained on the mixture of Wikipedia and code-related corpus. The highest value in each row is highlighted in bold .
Dataset
0.1B
0.5B
1B
AppsRetrieval
1.9
8.6
11.3
CosQA
25.8
26.0
26.6
SyntheticText2SQL
42.0
54.3
53.7
COIRCodeSearchNetRetrieval
27.8
38.9
46.2
CodeSearchNetCCRetrieval
47.0
60.5
61.7
CodeTransOceanContest
60.2
68.6
75.2
Appendix
Table 17: Comparison of ’s performance (nDCG@10, %) across CoIR benchmark tasks for three LM sizes trained on an out-of-domain corpus, i.e., Fineweb-edu. Bold indicates the best performance in each row.
Dataset
0.1B
0.5B
1B
ArguAna
36.4
36.9
38.7
ClimateFEVER
13.9
12.7
12.3
DBPedia
17.2
20.2
24.5
FEVER
30.5
45.0
53.9
FiQA2018
20.1
24.9
30.3
HotpotQA
29.7
47.5
54.2
Appendix
Table 18: Comparison of ’s performance (nDCG@10, %) across 13 BEIR tasks for three LM sizes trained on an out-of-domain corpus, i.e., Fineweb-edu. Bold indicates the best performance in each row.
Model
ARC
COPA
OpenBookQA
PIQA
RTE
WiC
WinoGrande
Avg.
LLaMA-3.2-1B
28.0
77.0
25.0
73.0
59.5
47.5
57.5
52.5
LM-
29.0
74.0
25.5
71.0
58.5
52.0
55.5
52.2
Appendix
Table 19: Performance (Accuracy, %) of the LM before and after on various benchmarks.
Dense retrieval embedding models are a fundamental component of modern retrieval-based AI systems. Most dense retrievers are trained with contrastive objectives, which require labeled positive and negative document pairs that are often costly and difficult to obtain. In this work, we investigate whether the autoregressive next-token prediction objective of a large language model (LLM) can provide supervision for dense retrieval. The intuition is simple: if a document contains information relevant to a query, conditioning on that document should make the target output easier for the LLM to predict. A key challenge is that the next-token prediction loss is computed inside the LLM, while the retriever is a separate embedding model. To address this challenge, we propose DREAM (Dense Retrieval Embeddings via Autoregressive Modeling), which injects retriever-generated query-document similarity scores into selected attention heads of a frozen LLM. During training, these scores determine how much attention each candidate document receives while the LLM predicts the target output. The resulting prediction loss provides gradients for retriever training through the attention mechanism. We evaluate DREAM on retrieval benchmarks BEIR and RTEB using embedding backbones ranging from 0.5B to 3B parameters. DREAM consistently outperforms existing baselines across different model scales. These results demonstrate that DREAM provides a promising approach for training dense retrievers through autoregressive modeling.
Yixuan Tang, Yi Yang
The Hong Kong University of Science and Technology
Dense retrieval systems increasingly rely on multi-billion-parameter language models, whose memory and computational requirements make large-scale indexing, frequent corpus updates, and low-latency serving costly. We present a three-stage training pipeline for developing compact and efficient retrievers that remain competitive with substantially larger models. The pipeline combines cross-lingual alignment, relational knowledge distillation, and contrastive fine-tuning. It requires no original ground-truth relevance labels, relying exclusively on supervision generated by strong embedding models and rerankers utilised as teachers. Using this pipeline, we develop PolDense and EuroDense, both supporting contexts of up to 8,192 tokens. PolDense is a family of six Polish retrievers ranging from 17M to 1B parameters. EuroDense is a 435M-parameter retriever supporting nine European languages. We conduct an extensive evaluation covering 41 Polish and 150 multilingual retrieval tasks. The results demonstrate strong quality-efficiency trade-offs. PolDense-1B outperforms the evaluated retrievers with up to 9B parameters, while the PolDense family forms the Pareto frontier across model sizes. Among the evaluated models below 1B parameters, EuroDense ranks first in both task-averaged and language-averaged performance and leads in seven of nine languages. We release all models publicly.
State-of-the-art retrieval models increasingly rely on closed training data, creating a reproducibility gap. We present an open end-to-end recipe for training retrieval models and study how English supervision transfers to multilingual retrieval through translate-train. We first reconstruct and curate 665M English contrastive pre-training pairs from 1.4B pairs across 34 public sources and build 1.88M supervised fine-tuning pairs with mined hard negatives. Training yields two 149M-parameter models: DenseOn, a single-vector dense model, and LateOn, a ColBERT-style late-interaction model. They achieve 56.20 and 57.22 average nDCG@10 on BEIR, respectively, setting new state-of-the-art results for this size class. We then translate the validated English data into eight languages, yielding 2.8B pairs with cross-lingual samples, and train mDenseOn and mLateOn, two 307M-parameter models built on mmBERT-base. Despite sharing their backbone, data, and objectives, their representations behave differently: the dense model is strong on English and translated languages but degrades outside translate-train support, whereas the late-interaction model generalizes better to unseen languages and scripts. This suggests that token-level matching turns translate-train from a target-language expansion strategy into a multilingual generalization recipe. We publicly release the models, datasets, and training code.
Raphaël Sourty, Antoine Chaffin, Paulo Roberto Moura Junior +1