MLLMs have been successfully applied to multimodal embedding tasks, yet their generative reasoning capabilities remain underutilized. Directly incorporating chain-of-thought reasoning into embedding learning introduces two fundamental challenges. First, structural misalignment between instance-level reasoning and pairwise contrastive supervision may lead to shortcut behavior, where the model merely learns the superficial format of reasoning. Second, reasoning is not universally beneficial for embedding tasks. Enforcing reasoning for all inputs may introduce unnecessary computation and latency, and can even obscure salient semantic signals for simple cases. To address these issues, we propose MMEmb-R1, an adaptive reasoning-based multimodal embedding framework. We formulate reasoning as a latent variable and introduce pair-aware reasoning selection that employs counterfactual intervention to identify reasoning paths beneficial for query-target alignment. Furthermore, we adopt reinforcement learning to selectively invoke reasoning only when necessary. Experiments on the MMEB-V2 benchmark demonstrate that our model achieves a score of 71.2 with only 4B parameters, establishing a new state-of-the-art while significantly reducing reasoning overhead and inference latency.
Figures & tables
Figure 2: Two challenges of reasoning in embedding. (a) Shortcut behavior: UME-R1’s embedding token largely ignores CoT tokens, while MMEmb-R1 actively utilizes them. (b) Overthinking: reasoning helps the complex query (top) but introduces irrelevant noise for the simple target “cat” (bottom).
Figure 3: Overview of the MMEmb-R1 framework. Upper Left: Pair-aware reasoning selection—multiple heterogeneous workers generate diverse rationale candidates for the query and target, and an evaluator scores each candidate to produce selection weights w1,w2,w3 . Upper right: Joint reasoning and embedding training—the MLLM is trained with a direct embedding path ( Ldirect ), a reasoning-enhanced embedding path ( Lreason ), and a next-token prediction objective over CoT tokens ( LCoT ). Lower: Adaptive reasoning via GRPO—the policy πθ decides whether to invoke reasoning or emit <EMPTY> for each query, guided by three reward signals: adaptive reward Rada , format reward Rformat , and embedding reward Remb .
Model
Backbone
Image
Video
VisDoc
All
CLS
QA
RET
GD
Overall
CLS
QA
RET
MRET
Overall
VDRv1
VDRv2
VR
OOD
Overall
Small-size Models
GME
Qwen2-VL-2B
54.4
29.9
66.9
55.5
51.9
34.9
42.0
25.6
32.4
33.9
86.1
54.0
82.5
43.1
72.7
54.1
ColPali-V1.3
PaliGemma-3B
40.3
11.5
48.1
40.3
34.9
26.7
37.8
21.6
25.5
28.2
83.6
52.0
81.1
43.1
71.0
44.4
VLM2Vec
Qwen2-VL-2B
58.7
49.3
65.0
72.9
59.7
33.4
30.5
20.6
33.0
29.0
49.8
13.5
51.8
33.5
41.6
47.0
VLM2Vec-V2
Qwen2-VL-2B
62.9
56.3
69.5
77.3
64.9
39.3
34.3
28.8
38.5
34.9
75.5
44.9
79.4
39.4
65.4
58.0
Table 1: Comparison of baseline methods and MMEmb-R1 on MMEB-V2. Given the diversity of model backbones, we aggregate results by model size. Models with 2B–3B parameters are categorized as small , while those with 4B–7B parameters are categorized as medium . † indicates that for the TTE model, we adopt the student variant to ensure a fair comparison without relying on a large external teacher model. Metrics are abbreviated as follows: CLS (classification), QA (question answering), RET (retrieval), GD (grounding), MRET (moment retrieval), VDR (ViDoRe), VR (VisRAG), and OOD (out-of-domain).
Strategy
Throughput
Accuracy
MMEmb-R1 (Direct)
22.70 queries/s
59.2
UME-R1
3.21 queries/s
60.1
MMEmb-R1 (Reason)
5.79 queries/s
63.6
MMEmb-R1 (Adaptive)
10.02 queries/s
65.0
Table 2: Inference throughput on a subset and overall accuracy on MMEB-V2.
Figure 4: Reasoning invocation ratio vs. overall accuracy. The curve peaks at 74.3% reasoning ratio, after which accuracy declines.
Variant
Score
Δ
MMEmb-R1 (Full)
65.0
–
Pair-aware Reasoning Selection
Single-teacher rationale
61.2
− 3.8
w/o pair-aware selection (uniform)
62.8
− 2.2
w/o baseline calibration (use cr only)
64.1
− 0.9
Training Objective
Table 3: Ablation study on MMEB-V2. Each row removes or modifies one component from the full MMEmb-R1 framework.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Scaling behavior of MMEmb-R1 across backbone families and parameter scales. Performance improves consistently within each family, and newer architectures achieve higher scores at comparable or smaller model sizes.
Figure 6: Distribution of reasoning gains Δr across worker types. Median values are shown as horizontal bars. The dashed line indicates the selection threshold ϵ=−0.1 ; candidates below are filtered out.
Figure 7: Adaptive reasoning: MMEmb-R1 skips reasoning for a simple visual query (top, avoiding overthinking) and invokes it for a complex temporal query (bottom), outperforming UME-R1 and VLM2Vec-V2 respectively.
Figure 8: Pair-aware reasoning selection: three heterogeneous workers produce complementary rationales for the same query. The evaluator assigns the highest weight to the Proprietary worker ( w=0.55 ), which best bridges the query–target semantic gap.
Figure 9: Distribution of reasoning utility δi over the training set. Green bars ( δi≥0 ) indicate instances where reasoning improves retrieval; red bars ( δi<0 ) indicate the opposite.
Table 4: Instructions to different workers MLLM for prior distribution simulation.
Comparison (Ours-4B vs.)
Win/Loss/Tie
Mean Δ
95% CI of Δ
Wilcoxon p
Paired- tp
Embed-RL-4B (closest 4B rival)
69 / 9 / 0
+3.08
[+2.13, +4.02]
6.3×10−10
1.5×10−8
UME-R1-7B
63 / 13 / 2
+6.66
[+4.36, +9.09]
4.4×10−8
4.3×10−7
Embed-RL-2B
69 / 9 / 0
+4.39
[+3.27, +5.52]
3.4×10−11
5.8×10−11
Appendix
Table 5: Statistical significance analysis of MMEmb-R1 (Qwen3-VL-4B) against strong baselines over 78 tasks. Δ denotes the per-task performance difference between our model and the corresponding baseline.
Worker Configuration
MMEB-V2
InternVL3-2B (single)
54.8
GLM-4.1V-Thinking (single)
61.2
Qwen3VL-32B (single)
61.9
GLM-4.1V-Thinking + InternVL3-14B
63.0
MMEmb-R1 (Full)
65.0
MMEmb-R1 + Another 3 workers
62.1
Appendix
Table 6: MMEB-V2 performance under different worker configurations.
Model/Evaluator
MMEB-V2
UME-R1
60.1
Evaluator-Flash
63.2
InternVL3-38B
63.7
Qwen3-VL-32B (Alt. Prompt)
64.7
Qwen3-VL-32B (Original)
65.0
Appendix
Table 7: MMEB-V2 performance under different evaluator configurations. “Alt. Prompt” denotes the use of an alternative prompt template.
Benchmark
w/o Selection
w/ Pair-Aware Selection
MMStar
60.2
63.2
ROCOv2
43.26
45.78
Appendix
Table 8: Performance with and without the pair-aware selection mechanism on a generative task (MMStar) and an out-of-domain benchmark (ROCOv2). ROCOv2 is evaluated using CUI@5.
Setting
MMEB-V2
Original
65.0
w/o question text
52.8
Random question text
43.1
Appendix
Table 9: MMEB-V2 performance under different question-text perturbations. The target items remain unchanged across all settings.
Table 10: Prompts used for the pair-aware evaluator J . The baseline prompt (top) evaluates query–target relevance without reasoning. The with-rationale prompt (bottom) includes generated rationales, enabling the counterfactual comparison Δr=cr−c0 .
Table 11: Instruction template used during joint reasoning and embedding training (§ 3.3 ).
Model
Per Meta-Task Score
Average Score
Classification
VQA
Retrieval
Grounding
IND
OOD
Overall
# of Datasets
10
10
12
4
20
16
36
Baseline Models
CLIP Radford et al. (2021)
42.8
9.1
53.0
51.8
37.1
38.7
37.8
BLIP-2 Li et al. (2023)
27.0
4.2
33.9
47.0
25.3
25.1
25.2
SigLIP Zhai et al. (2023)
40.3
8.4
31.6
59.5
32.3
38.0
34.8
Appendix
Table 13: Results on the MMEB-V1 benchmark, which consists of 36 image embedding tasks. IND and OOD denote in-distribution and out-of-distribution datasets, respectively. Some results are adopted from Embed-RL Jiang et al. (2026) .
Multimodal large language models (MLLMs) have emerged as a powerful backbone for multimodal embeddings. Recent methods introduce chain-of-thought (CoT) reasoning into the embedding pipeline to improve retrieval quality, but remain costly in both model size and inference cost. They typically employ separate reasoner and embedder with substantial parameter overhead, and generate CoT indiscriminately for every input. However, we observe that for simple inputs, discriminative embeddings already perform well, and redundant reasoning can even mislead the model, degrading performance. To address these limitations, we propose Think When Needed (TWN), a unified multimodal embedding framework with adaptive reasoning. TWN introduces a dual-LoRA architecture that attaches reasoning and embedding adapters to a shared frozen backbone, detaching gradients at their interface to mitigate gradient conflicts introduced by joint optimization while keeping parameters close to a single model. Building on this, an adaptive think mechanism uses a self-supervised routing gate to decide per input whether to generate CoT, skipping unnecessary reasoning to reduce inference overhead and even improve retrieval quality. We further explore embedding-guided RL to optimize CoT quality beyond supervised training. On the 78 tasks of MMEB-V2, TWN achieves state-of-the-art embedding quality while being substantially more efficient than existing generative methods, requiring only 3-5% additional parameters relative to the backbone and up to 50% fewer reasoning tokens compared to the full generative mode.
Reasoning-driven universal multimodal embedding has advanced rapidly by introducing Chain-of-Thought (CoT) reasoning into the embedding pipeline. Despite the strong performance across both general and complex tasks, this paradigm suffers from two core limitations: (i) autoregressive CoT reasoning incurs high computational cost, making it impractical for low-latency retrieval; and (ii) embedding performance is heavily coupled with CoT annotation quality, making large-scale training unreliable. These raise fundamental questions: Is textual CoT the optimal form of reasoning for embedding, and can effective embedding reasoning be accomplished in latent space? To this end, we propose LaME (Latent Reasoning Multimodal Embedding), which formulates embedding-oriented latent reasoning as a weakly supervised information bottleneck. LaME employs K learnable reason tokens as a fixed-capacity bottleneck, completing all reasoning within a single forward pass. The two weak supervision signals structurally decouple contrastive from autoregressive objectives and eliminate dependence on CoT annotations, while a two-stage training pipeline ensures stable convergence. Experiments on MMEB-v2 and MRMR show that LaME achieves competitive performance, surpassing some explicit CoT-based models, while delivering 60x faster inference than explicit CoT methods and 2x faster than latent baselines with throughput comparable to discriminative embedding models. Code is available at https://github.com/PeppaWu/LaME.
Peixi Wu, Biao Yang, Feipeng Ma +7
University of Science and Technology of China · 2Kuaishou Technology · 3Zhejiang University +1
Advances in Multimodal Large Language Models (MLLMs) have driven substantial progress in multimodal embeddings. Recent methods exploit generative capabilities by augmenting embedded inputs with self-generated reasoning to produce more informative embeddings. However, conventional retrieval supervision for training embedding models specifies pairwise relevance relations without explicitly guiding the reasoning used for input augmentation. Such reasoning is neither directly observed nor uniquely determined by the input, allowing multiple plausible traces to introduce semantic cues of varying retrieval utility. Existing approaches largely optimize reasoning conditioned on the input alone, limiting the exploration of retrieval-relevant traces. We introduce PILOT, a framework that models reasoning as a latent variable and instantiates its prior and posterior distributions as distinct reasoning modes of a shared MLLM. The prior conditions only on the input, whereas the posterior uses additional training-time relational evidence to guide reasoning exploration. To translate the posterior's broader exploration into inference-time improvements, we derive an importance-weighted variational objective and instantiate it as a prior-posterior co-evolution framework that improves both modes under a shared inference-compatible reward while selectively transferring posterior trajectories to the prior according to their utility and prior reachability. On the 78-task MMEB-V2 benchmark, PILOT outperforms the primary baseline across all 12 task groups at both 2B and 7B parameters, improving overall scores by 3.2 and 3.3 points, respectively, with the largest gains in visual-document retrieval (+6.1 and +6.4 points). Our results show that relational supervision is most valuable not as a post-hoc reward but as a signal for shaping how reasoning is explored in training reasoning-enhanced embedding models.
Tianyu Yang, Yijun Tian, Lars Benedikt Kaesberg +4
University of Göttingen · University of Notre Dame · State and University Library of Göttingen