LLMs have fundamentally transformed dense retrieval, upgrading backbones from discriminative encoders to generative architectures. However, a critical disconnect remains: while LLMs possess strong reasoning capabilities, current retrievers predominantly utilize them as static encoders, leaving their potential for complex reasoning unexplored. To address this, existing approaches typically adopt rewrite-then-retrieve pipelines to generate explicit CoT rationales before retrieval. However, this incurs prohibitive latency. In this paper, we propose LaSER, a novel self-distillation framework that internalizes explicit reasoning into the latent space of dense retrievers. Operating on a shared LLM backbone, LaSER introduces a dual-view training mechanism: an Explicit view that explicitly encodes ground-truth reasoning paths, and a Latent view that performs implicit latent thinking. To bridge the gap between these views, we design a multi-grained alignment strategy. Beyond standard output alignment, we introduce a trajectory alignment mechanism that synchronizes the intermediate latent states of the latent path with the semantic progression of the explicit reasoning segments. This allows the retriever to think silently and effectively without autoregressive text generation. Extensive experiments on both in-domain and out-of-domain reasoning-intensive benchmarks demonstrate that LaSER significantly outperforms state-of-the-art baselines. Furthermore, analyses across diverse backbones and model scales validate the robustness of our approach, confirming that our unified learning framework is essential for eliciting effective latent thinking. Our method successfully combines the reasoning depth of explicit CoT pipelines with the inference efficiency of standard dense retrievers.
Figures & tables
Figure 1. Left: Comparison between our proposed pipeline and conventional retriever and rewrite-then-retrieve method. Right: Comparison of performance and latency of different methods on the Bright benchmark. Letency represents the time consumption of processing a single query. The “Rewrite-then-Retrieve” method uses a 0.6B retriever and a 3B rewriter.
Figure 2. Illustration of the LaSER. During training, each step involves two inference pathways: the explicit-view and the latent-view (shown on the left). The inputs for these two paths differ, with the explicit-view incorporating additional information. However, during the inference phase, only the latent-view mode is utilized. The loss function is computed based on the results from both paths (shown on the right).
Instruction:
Given a question, your mission is to follow the instructions below:
1. Identify the essential problem.
2. Think step by step to reason and describe what information could be relevant and helpful to address the questions in detail.
3. Draft an answer with as many thoughts as you have.
The given question:
[Begin of Question]
Table 1. The prompt used for annotating the explicit reasoning path of the given query.
Split
Dataset
# Q
# C
Q Len.
Reas. Len.
Train
ReasonEmb
81,659
–
222.1
984.8
Test
Bright (ID)
1,384
1,145,164
240.8
2,412.8
FollowIR (OOD)
104
98,312
76.6
–
BC-Plus (OOD)
830
100,195
123.2
–
Table 2. Statistics of the datasets used in our experiments. “Q” and “C” denote Query and Corpus respectively. “Len.” denotes average token count. “Reas. Len.” denotes average reasoning length annotated by external LLM.
Models
Size
StackExchange
Coding
Theorem-based
Avg.
Bio.
Earth.
Econ.
Psy.
Rob.
Stack.
Sus.
Leet.
Pony
AoPS
TheoQ.
TheoT.
Standard Dense Retrievers
BGE-M3
0.6B
7.2
13.3
12.5
12.8
12.6
10.0
10.1
15.6
30.7
1.5
5.6
5.0
11.4
Multilingual-E5-Large-Instruct
0.6B
15.0
24.6
14.0
14.6
16.1
10.3
13.8
15.1
2.1
3.0
11.6
6.9
12.3
E5-Mistral-7B-Instruct
7B
17.9
28.6
18.5
16.5
18.6
13.4
20.0
19.7
6.3
2.0
14.2
22.1
16.5
Qwen3-Embedding-0.6B
0.6B
12.7
26.3
17.9
16.5
12.5
12.4
12.2
14.3
0.7
3.1
17.2
26.5
14.4
Table 3. Main retrieval performance on the Bright benchmark. We report nDCG@10 for all subsets. The best results in the same backbone setting are formatted in bold and the second best are underlined . † indicates the pipeline method that uses an external LLM to rewrite queries during inference.
Models
Size
Robust04
News21
Core17
FollowIR-Avg
BrowseComp-Plus
MAP@5
p-MRR
nDCG@5
p-MRR
MAP@5
p-MRR
Score
p-MRR
R@5
R@100
R@1000
Standard Dense Retrievers
BGE-M3
0.6B
1.5
-1.6
21.4
-1.3
7.5
-8.8
10.1
-3.9
4.1
21.7
47.2
Multilingual-E5-Large-Instruct
0.6B
2.4
2.7
25.0
1.2
8.5
-6.0
12.0
-0.7
10.1
36.8
68.6
E5-Mistral-7B-Instruct
7B
2.7
2.5
28.8
0.8
13.7
-7.8
15.1
-1.5
9.3
37.1
70.2
Qwen3-Embedding-0.6B
0.6B
2.8
8.9
27.3
3.6
11.4
2.7
13.8
5.1
8.0
29.1
64.8
Table 4. Retrieval performance. Columns under Robust04, News21, and Core17 belong to the FollowIR benchmark.
Method
Bright
FollowIR
BrowseComp-Plus
nDCG@10
Score
p-MRR
R@5
R@1000
LaSER (Qwen3-0.6B)
23.10
11.49
-0.02
6.76
54.88
w/o Explicit View
20.59
9.78
-1.71
6.48
50.14
w/o Latent View
19.93
8.62
-0.11
5.93
52.25
w/o Process Align.
22.33
11.63
-1.58
6.18
52.43
w/o Output Distill.
19.97
7.55
0.43
7.03
53.02
Table 5. Ablation study of our proposed method. For FollowIR, we report the average score on all subsets.
Figure 3. Performance comparison of LaSER and baselines across different backbones and model sizes on Bright.
Figure 4. Comparison of the performance of using rewrite path on Bright between our method and basic training.
Figure 5. Effect of latent thinking steps during training and inference.
Reasoning-intensive retrieval requires text representations to capture not only semantic similarity, but also the reasoning needed to determine relevance under a given retrieval instruction. Existing reasoning-enhanced embedding models improve retrieval by incorporating reasoning information into dense representations, yet their supervision is typically dominated by the final retrieval objective. As a result, latent reasoning trajectories may learn shortcut reasoning patterns that preserve retrieval performance without producing meaningful incremental retrieval gains. We propose Retrieval Grounding Latent Reasoning (RGLT), a latent reasoning framework for dense retrieval that explicitly connects intermediate latent transitions with retrieval improvements. RGLT performs non-autoregressive reasoning in hidden space through an instruction-conditioned latent reasoning trajectory constructed from silent tokens. It combines process-supervised explicit-to-implicit distillation with retrieval-grounded supervision, using stage-wise CoT reconstruction to shape intermediate latent states and retrieval-effect credit to optimize incremental retrieval gains across the latent reasoning trajectories. Experiments on reasoning-intensive retrieval benchmarks show that RGLT consistently outperforms strong baselines while preserving efficient embedding inference.
Gang Zhou, Xiongxi Yu, Hu Tian +5
School of Artificial Intelligence, Beijing University of Posts and Telecommunications · Meituan LongCat Interaction Team · School of Management Science and Engineering, Central University of Finance and Economics +1
Explicit chain-of-thought (CoT) reasoning substantially improves the reasoning ability of large language models (LLMs), but incurs high inference cost due to lengthy autoregressive traces. Existing latent reasoning methods offer a promis- ing alternative, yet they often treat reasoning as uniformly compressible, causing precision-critical intermediate steps to be overly compressed and thereby degrading reasoning accuracy. In this work, we propose Selective Latent Thinking (SLT), a reasoning paradigm that selectively compresses redundant reasoning spans into latent representations while preserving precision-critical spans as explicit CoT within the same reasoning trajectory. Our primary instantiation uses a lightweight decoder to anticipate a short upcoming reasoning span and applies confidence- based gating to determine the longest span that can be reliably compressed; the resulting selective compression policy is further optimized with trajectory-level reinforcement learning to balance answer correctness against reasoning cost. Ex- periments across four mathematical reasoning benchmarks show that which spans are compressed is critical: with the same compressor, compressing arbitrary spans reduces the GSM-Hard accuracy of Qwen3-4B from 54.7% to 17.3%, whereas keeping numerical tokens explicit retains 51.8%. SLT reduces explicit reasoning length by 58.4% with only a 2.8-point accuracy drop compared to explicit CoT on Llama-3.2-1B. Notably, the principle extends beyond this design: instantiated within existing latent reasoning models via a learned compression gate, it improves accuracy at equal reasoning length (+2.6 points over Latent-SFT) and at shorter length (+5.6 points over Latent-GRPO with 13% fewer tokens).
Hui Xie, Jie Liu, Ziyue Qiao +1
Eindhoven University of Technology, Netherlands · School of Computing and Information Technology, Great Bay University, China
Dense retrieval embedding models are a fundamental component of modern retrieval-based AI systems. Most dense retrievers are trained with contrastive objectives, which require labeled positive and negative document pairs that are often costly and difficult to obtain. In this work, we investigate whether the autoregressive next-token prediction objective of a large language model (LLM) can provide supervision for dense retrieval. The intuition is simple: if a document contains information relevant to a query, conditioning on that document should make the target output easier for the LLM to predict. A key challenge is that the next-token prediction loss is computed inside the LLM, while the retriever is a separate embedding model. To address this challenge, we propose DREAM (Dense Retrieval Embeddings via Autoregressive Modeling), which injects retriever-generated query-document similarity scores into selected attention heads of a frozen LLM. During training, these scores determine how much attention each candidate document receives while the LLM predicts the target output. The resulting prediction loss provides gradients for retriever training through the attention mechanism. We evaluate DREAM on retrieval benchmarks BEIR and RTEB using embedding backbones ranging from 0.5B to 3B parameters. DREAM consistently outperforms existing baselines across different model scales. These results demonstrate that DREAM provides a promising approach for training dense retrievers through autoregressive modeling.
Yixuan Tang, Yi Yang
The Hong Kong University of Science and Technology