Advances in Multimodal Large Language Models (MLLMs) have driven substantial progress in multimodal embeddings. Recent methods exploit generative capabilities by augmenting embedded inputs with self-generated reasoning to produce more informative embeddings. However, conventional retrieval supervision for training embedding models specifies pairwise relevance relations without explicitly guiding the reasoning used for input augmentation. Such reasoning is neither directly observed nor uniquely determined by the input, allowing multiple plausible traces to introduce semantic cues of varying retrieval utility. Existing approaches largely optimize reasoning conditioned on the input alone, limiting the exploration of retrieval-relevant traces. We introduce PILOT, a framework that models reasoning as a latent variable and instantiates its prior and posterior distributions as distinct reasoning modes of a shared MLLM. The prior conditions only on the input, whereas the posterior uses additional training-time relational evidence to guide reasoning exploration. To translate the posterior's broader exploration into inference-time improvements, we derive an importance-weighted variational objective and instantiate it as a prior-posterior co-evolution framework that improves both modes under a shared inference-compatible reward while selectively transferring posterior trajectories to the prior according to their utility and prior reachability. On the 78-task MMEB-V2 benchmark, PILOT outperforms the primary baseline across all 12 task groups at both 2B and 7B parameters, improving overall scores by 3.2 and 3.3 points, respectively, with the largest gains in visual-document retrieval (+6.1 and +6.4 points). Our results show that relational supervision is most valuable not as a post-hoc reward but as a signal for shaping how reasoning is explored in training reasoning-enhanced embedding models.
Figures & tables
Figure 1: Relationally guided reasoning exploration in PILOT . (a) Input-only prior reasoning may miss discriminative cues, yielding weak embedding separation. (b) The posterior uses training-time relational evidence to discover target-grounded cues that help distinguish positive and negative candidates. Embedding layouts and reward curves are schematic.
Figure 2: Overview of PILOT. (a) A stronger offline annotator, given only a shuffled, unlabeled candidate pair u , synthesizes prior and consistency-filtered posterior reasoning traces. (b) Joint SFT and contrastive learning initialize both reasoning modes in a shared MLLM (Eq. 8 ). (c) GRPO optimizes both modes using rewards evaluated under the prior context (Eq. 12 ), while importance-weighted fitting transfers trajectories to the prior according to retrieval utility and prior reachability (Eq. 13 ). (d) Inference runs the prior alone.
Model
Training
Image
Video
VisDoc
All
# of Pairs
# of Annotators
CLS
QA
RET
GD
Avg.
CLS
QA
RET
MRET
Avg.
VDRv1
VDRv2
VR
OOD
Avg.
# of Datasets
10
10
12
4
36
5
5
5
3
18
10
4
6
4
24
78
Closed-Source Models
Seed-1.6-embedding
NR
NR
76.1
74.0
77.9
91.3
77.8
55.0
60.9
51.3
53.5
55.3
85.5
56.6
84.7
43.1
73.4
71.3
Small Backbones (2–3B)
ColPali-V1.3
127K
—
40.3
11.5
48.1
40.3
34.9
26.7
37.8
21.6
25.5
28.2
83.6
52.0
81.1
43.1
71.0
44.4
Table 1: Performance comparison across image, video, and visual-document (VisDoc) retrieval benchmarks. Shaded rows denote PILOT. Blue values in parentheses indicate absolute gains over UME-R1 at the same backbone size.
Training configuration
Performance
Variant
SFT initialization
Prior GRPO
Posterior during RL
IW prior fitting
Rollouts/query (prior + posterior)
Image
Video
VisDoc
All
PILOT
Dual-mode
✓
GRPO-updated
✓
8+8
67.5
45.9
69.9
63.3
Joint GRPO
Dual-mode
✓
GRPO-updated
×
8+8
66.8
45.1
68.1
62.2 ( −1.1 )
Frozen-posterior variant
Dual-mode
✓
Frozen
×
8+0
67.1
44.8
66.1
61.6 ( −1.7 )
Prior GRPO only
Dual-mode
✓
Not used
×
16+0
66.5
43.4
67.2
61.4 ( −1.9 )
Dual-mode SFT
Dual-mode
—
—
—
—
65.7
42.1
65.9
60.3 ( −3.0 )
Table 2: Component ablation on MMEB-V2. Each row specifies a complete training configuration. All variants are evaluated using only the inference-time prior.
Figure 3: Reasoning quality, candidate leakage, and verification under different synthesis prompts. (A–B) Overall and domain-level judged retrieval usefulness versus candidate leakage. (C) Comparison on shared query subsets with no leakage flagged under any variant or leakage flagged under every variant. (D) Comparison conditioned on blind-prediction correctness, with matched sample indices across conditions.
Figure 4: Posterior exploration and transfer. (a) Pass@K after dual-mode initialization. (b) Utility–reachability trade-off, colored by transfer score. (c) Changes in prior-solved, posterior-only, and both-failed query proportions relative to dual-mode initialization comparing the full method and ablations without transfer supervision (w/o TS) or importance weighting (w/o IW).
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Mode
Number of pairs
Proportion
Prior
787,202
53.79%
Posterior
676,158
46.21%
Total
1,463,360
100.00%
Appendix
Table 3: Composition of the dual-mode SFT dataset. Each pair contains a query and its positive candidate, with reasoning supervision on both sides.
Prompt condition
N
W/T/L
Net win
Margin
95% CI
Leak
Input + positive
1000
445/393/162
0.283
0.451
[0.388, 0.514]
0.274
Input + positive + negative (labeled)
1000
406/416/178
0.228
0.385
[0.322, 0.448]
0.236
Input + positive + negative (blind)
1000
608/216/176
0.432
0.772
[0.698, 0.845]
0.796
Appendix
Table 4: Model-based pairwise audit of query-side reasoning on 1,000 MMEB-2 queries (v3-delimited prompts; random negatives). Each candidate-conditioned trace is compared with the input-only trace for the same query and sample index. W/T/L counts wins, ties, and losses; net win is (W−L)/N . Margin is the mean signed preference score in {−2,−1,0,1,2} . Intervals are query-level bootstrap 95% confidence intervals for mean margin. Leak denotes the fraction flagged for candidate-only information.
Prompt condition
Stratum
N
W/T/L
Net win
Margin
Input + positive
No leak
726
238/367/121
0.161
0.249
Leak
274
207/26/41
0.606
0.985
Input + positive + negative (labeled)
No leak
764
226/392/146
0.105
0.173
Leak
236
180/24/32
0.627
1.072
Input + positive + negative (blind)
No leak
204
71/104/29
0.206
0.328
Leak
796
537/112/147
0.490
0.886
Appendix
Table 5: Pairwise reasoning quality stratified by candidate-only information leakage. Scores compare each trace with its matched input-only baseline. These post-generation strata are descriptive and do not identify the causal effect of leakage.
Domain
Prompt condition
N
W/T/L
Net win
Margin
Leak
Image
Input + positive
447
156/239/52
0.233
0.306
0.199
Input + positive + negative (labeled)
447
140/239/68
0.161
0.237
0.130
Input + positive + negative (blind)
447
196/150/101
0.213
0.391
0.725
Video
Input + positive
220
101/87/32
0.314
0.473
0.164
Input + positive + negative (labeled)
220
94/87/39
0.250
0.400
0.136
Input + positive + negative (blind)
220
122/57/41
0.368
0.541
0.700
Appendix
Table 6: Pairwise reasoning-quality audit by MMEB-2 domain. All prompt conditions include the query input and are compared against matched input-only traces. Rates are reported as fractions.
Blind prediction
N
W/T/L
Net win
Margin
Leak
Unsupported
Correct
971
602/212/157
0.458
0.818
0.803
0.092
Incorrect
29
6/4/19
-0.448
-0.759
0.552
0.483
Appendix
Table 7: Blind-conditioned reasoning stratified by whether the expert correctly selected the positive candidate. Unsupported denotes the fraction of traces flagged for unsupported claims. Differences are descriptive: correct and incorrect groups may differ in query difficulty and do not establish a causal effect of verification.
Prompt condition
N
W/T/L
Net win
Margin
95% CI
Leak
Input + positive
648
296/243/109
0.289
0.503
[0.350, 0.660]
0.250
Input + positive + negative (labeled)
648
265/262/121
0.222
0.395
[0.248, 0.545]
0.211
Input + positive + negative (blind)
648
441/117/90
0.542
0.937
[0.739, 1.131]
0.753
Appendix
Table 8: Repeated-sampling audit on 81 queries with eight samples per prompt condition. Top: aggregate results over 648 paired comparisons per condition; confidence intervals resample queries as clusters. Bottom: sample index 0 versus oracle-best-of-8, each with 81 comparisons per condition. The oracle maximizes each query’s paired margin against the input-only trace at the corresponding sample index. It is a retrospective paired-score upper bound, not an executable selection policy or an absolute-quality oracle.
Figure 5: Effective sample size (ESS) of the transfer weights during training (Full).
Figure 6: Signed log-probability gap between the posterior and prior prompts on posterior traces, per training subset.
MLLMs have been successfully applied to multimodal embedding tasks, yet their generative reasoning capabilities remain underutilized. Directly incorporating chain-of-thought reasoning into embedding learning introduces two fundamental challenges. First, structural misalignment between instance-level reasoning and pairwise contrastive supervision may lead to shortcut behavior, where the model merely learns the superficial format of reasoning. Second, reasoning is not universally beneficial for embedding tasks. Enforcing reasoning for all inputs may introduce unnecessary computation and latency, and can even obscure salient semantic signals for simple cases. To address these issues, we propose MMEmb-R1, an adaptive reasoning-based multimodal embedding framework. We formulate reasoning as a latent variable and introduce pair-aware reasoning selection that employs counterfactual intervention to identify reasoning paths beneficial for query-target alignment. Furthermore, we adopt reinforcement learning to selectively invoke reasoning only when necessary. Experiments on the MMEB-V2 benchmark demonstrate that our model achieves a score of 71.2 with only 4B parameters, establishing a new state-of-the-art while significantly reducing reasoning overhead and inference latency.
Yuchi Wang, Dingkang Yang, Haiyang Yu +5
MMLab, The Chinese University of Hong Kong · ByteDance Inc. · Fudan University
Universal multimodal retrieval requires compact embeddings that preserve task-relevant semantic information across diverse modalities. Prior works have incorporated latent reasoning into multimodal embedding learning to refine this information before embedding extraction. However, most existing approaches remain confined to deterministic latent paths, without exploring alternative trajectories to discover better embeddings. Thus, we propose VaME (Variational Multimodal Embeddings), a framework that models latent reasoning as a learnable distribution over trajectories. Specifically, we first introduce Variational Latent Reasoning (VLR) to enable autoregressive exploration in latent space, guided by answer reconstruction through a lightweight decoder. Meanwhile, we augment the original embedding-token readout with a latent-fused embedding to facilitate exploration during subsequent reinforcement learning. Finally, we optimize latent reasoning over stochastic variational trajectories through reinforcement learning, using Semantic Decoding Reward (SDR) to favor semantically meaningful trajectories with interpretable decoded outcomes. On the 78-task MMEB-V2 benchmark, spanning image, video, and visual-document retrieval, VaME outperforms most explicit CoT-based models and all latent-reasoning baselines. VaME also demonstrates robust performance on reasoning-intensive benchmarks such as MRMR, with substantial gains after reinforcement learning. Importantly, VaME achieves these gains with at least a 4.25x inference speedup over the deterministic latent autoregressive baselines. The code will be made publicly available.
Peixi Wu, Mingzhou Jiang, Feipeng Ma +11
University of Science and Technology of China · Tsinghua University · Kuaishou +1
Multimodal large language models (MLLMs) have emerged as a powerful backbone for multimodal embeddings. Recent methods introduce chain-of-thought (CoT) reasoning into the embedding pipeline to improve retrieval quality, but remain costly in both model size and inference cost. They typically employ separate reasoner and embedder with substantial parameter overhead, and generate CoT indiscriminately for every input. However, we observe that for simple inputs, discriminative embeddings already perform well, and redundant reasoning can even mislead the model, degrading performance. To address these limitations, we propose Think When Needed (TWN), a unified multimodal embedding framework with adaptive reasoning. TWN introduces a dual-LoRA architecture that attaches reasoning and embedding adapters to a shared frozen backbone, detaching gradients at their interface to mitigate gradient conflicts introduced by joint optimization while keeping parameters close to a single model. Building on this, an adaptive think mechanism uses a self-supervised routing gate to decide per input whether to generate CoT, skipping unnecessary reasoning to reduce inference overhead and even improve retrieval quality. We further explore embedding-guided RL to optimize CoT quality beyond supervised training. On the 78 tasks of MMEB-V2, TWN achieves state-of-the-art embedding quality while being substantially more efficient than existing generative methods, requiring only 3-5% additional parameters relative to the backbone and up to 50% fewer reasoning tokens compared to the full generative mode.