MMEmb-R1: Reasoning-Enhanced Multimodal Embedding with Pair-Aware Selection and Adaptive Control

Yuchi Wang; Dingkang Yang; Haiyang Yu; Weikang Bian; Jiefeng Long; Xiao Liang; Chao Feng; Hongsheng Li

cs.CVApr 7, 2026

MMEmb-R1: Reasoning-Enhanced Multimodal Embedding with Pair-Aware Selection and Adaptive Control

Authors: Yuchi Wang, Dingkang Yang, Haiyang Yu, Weikang Bian, Jiefeng Long, Xiao Liang, Chao Feng, Hongsheng Li

Organizations: MMLab, The Chinese University of Hong Kong · ByteDance Inc. · Fudan University

Abstract

MLLMs have been successfully applied to multimodal embedding tasks, yet their generative reasoning capabilities remain underutilized. Directly incorporating chain-of-thought reasoning into embedding learning introduces two fundamental challenges. First, structural misalignment between instance-level reasoning and pairwise contrastive supervision may lead to shortcut behavior, where the model merely learns the superficial format of reasoning. Second, reasoning is not universally beneficial for embedding tasks. Enforcing reasoning for all inputs may introduce unnecessary computation and latency, and can even obscure salient semantic signals for simple cases. To address these issues, we propose MMEmb-R1, an adaptive reasoning-based multimodal embedding framework. We formulate reasoning as a latent variable and introduce pair-aware reasoning selection that employs counterfactual intervention to identify reasoning paths beneficial for query-target alignment. Furthermore, we adopt reinforcement learning to selectively invoke reasoning only when necessary. Experiments on the MMEB-V2 benchmark demonstrate that our model achieves a score of 71.2 with only 4B parameters, establishing a new state-of-the-art while significantly reducing reasoning overhead and inference latency.

Figures & tables

Figure 2: Two challenges of reasoning in embedding. (a) Shortcut behavior: UME-R1’s embedding token largely ignores CoT tokens, while MMEmb-R1 actively utilizes them. (b) Overthinking: reasoning helps the complex query (top) but introduces irrelevant noise for the simple target “cat” (bottom).

Figure 3: Overview of the MMEmb-R1 framework. Upper Left: Pair-aware reasoning selection—multiple heterogeneous workers generate diverse rationale candidates for the query and target, and an evaluator scores each candidate to produce selection weights $w_{1},w_{2},w_{3}$ . Upper right: Joint reasoning and embedding training—the MLLM is trained with a direct embedding path ( $\mathcal{L}_{\text{direct}}$ ), a reasoning-enhanced embedding path ( $\mathcal{L}_{\text{reason}}$ ), and a next-token prediction objective over CoT tokens ( $\mathcal{L}_{\text{CoT}}$ ). Lower: Adaptive reasoning via GRPO—the policy $\pi_{\theta}$ decides whether to invoke reasoning or emit <EMPTY> for each query, guided by three reward signals: adaptive reward $R_{\text{ada}}$ , format reward $R_{\text{format}}$ , and embedding reward $R_{\text{emb}}$ .

Model	Backbone	Image					Video					VisDoc					All
Model	Backbone	CLS	QA	RET	GD	Overall	CLS	QA	RET	MRET	Overall	VDRv1	VDRv2	VR	OOD	Overall	All
Small-size Models
GME	Qwen2-VL-2B	54.4	29.9	66.9	55.5	51.9	34.9	42.0	25.6	32.4	33.9	86.1	54.0	82.5	43.1	72.7	54.1
ColPali-V1.3	PaliGemma-3B	40.3	11.5	48.1	40.3	34.9	26.7	37.8	21.6	25.5	28.2	83.6	52.0	81.1	43.1	71.0	44.4
VLM2Vec	Qwen2-VL-2B	58.7	49.3	65.0	72.9	59.7	33.4	30.5	20.6	33.0	29.0	49.8	13.5	51.8	33.5	41.6	47.0
VLM2Vec-V2	Qwen2-VL-2B	62.9	56.3	69.5	77.3	64.9	39.3	34.3	28.8	38.5	34.9	75.5	44.9	79.4	39.4	65.4	58.0

Table 1: Comparison of baseline methods and MMEmb-R1 on MMEB-V2. Given the diversity of model backbones, we aggregate results by model size. Models with 2B–3B parameters are categorized as small , while those with 4B–7B parameters are categorized as medium .

\dagger

indicates that for the TTE model, we adopt the student variant to ensure a fair comparison without relying on a large external teacher model. Metrics are abbreviated as follows: CLS (classification), QA (question answering), RET (retrieval), GD (grounding), MRET (moment retrieval), VDR (ViDoRe), VR (VisRAG), and OOD (out-of-domain).

Strategy	Throughput	Accuracy
MMEmb-R1 (Direct)	22.70 queries/s	59.2
UME-R1	3.21 queries/s	60.1
MMEmb-R1 (Reason)	5.79 queries/s	63.6
MMEmb-R1 (Adaptive)	10.02 queries/s	65.0

Table 2: Inference throughput on a subset and overall accuracy on MMEB-V2.

Figure 4: Reasoning invocation ratio vs. overall accuracy. The curve peaks at 74.3% reasoning ratio, after which accuracy declines.

Pair-aware Reasoning Selection
Variant	Score	$\Delta$
MMEmb-R1 (Full)	65.0	–
Single-teacher rationale	61.2	$-$ 3.8
w/o pair-aware selection (uniform)	62.8	$-$ 2.2
w/o baseline calibration (use $c_{r}$ only)	64.1	$-$ 0.9
Training Objective

Table 3: Ablation study on MMEB-V2. Each row removes or modifies one component from the full MMEmb-R1 framework.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Scaling behavior of MMEmb-R1 across backbone families and parameter scales. Performance improves consistently within each family, and newer architectures achieve higher scores at comparable or smaller model sizes.

Figure 6: Distribution of reasoning gains

\Delta_{r}

across worker types. Median values are shown as horizontal bars. The dashed line indicates the selection threshold

\epsilon=-0.1

; candidates below are filtered out.

Figure 7: Adaptive reasoning: MMEmb-R1 skips reasoning for a simple visual query (top, avoiding overthinking) and invokes it for a complex temporal query (bottom), outperforming UME-R1 and VLM2Vec-V2 respectively.

Figure 8: Pair-aware reasoning selection: three heterogeneous workers produce complementary rationales for the same query. The evaluator assigns the highest weight to the Proprietary worker ( $w\!=\!0.55$ ), which best bridges the query–target semantic gap.

Figure 9: Distribution of reasoning utility

\delta_{i}

over the training set. Green bars (

\delta_{i}\geq 0

) indicate instances where reasoning improves retrieval; red bars (

\delta_{i}<0

) indicate the opposite.

Panel of Table 4 — Table 4: Instructions to different workers MLLM for prior distribution simulation.

Comparison (Ours-4B vs.)	Win/Loss/Tie	Mean $\Delta$	95% CI of $\Delta$	Wilcoxon $p$	Paired- $t$ $p$
Embed-RL-4B (closest 4B rival)	69 / 9 / 0	+3.08	[+2.13, +4.02]	$6.3\times 10^{-10}$	$1.5\times 10^{-8}$
UME-R1-7B	63 / 13 / 2	+6.66	[+4.36, +9.09]	$4.4\times 10^{-8}$	$4.3\times 10^{-7}$
Embed-RL-2B	69 / 9 / 0	+4.39	[+3.27, +5.52]	$3.4\times 10^{-11}$	$5.8\times 10^{-11}$

Appendix

Table 5: Statistical significance analysis of MMEmb-R1 (Qwen3-VL-4B) against strong baselines over 78 tasks.

\Delta

denotes the per-task performance difference between our model and the corresponding baseline.

Worker Configuration	MMEB-V2
InternVL3-2B (single)	54.8
GLM-4.1V-Thinking (single)	61.2
Qwen3VL-32B (single)	61.9
GLM-4.1V-Thinking + InternVL3-14B	63.0
MMEmb-R1 (Full)	65.0
MMEmb-R1 + Another 3 workers	62.1

Appendix

Table 6: MMEB-V2 performance under different worker configurations.

Model/Evaluator	MMEB-V2
UME-R1	60.1
Evaluator-Flash	63.2
InternVL3-38B	63.7
Qwen3-VL-32B (Alt. Prompt)	64.7
Qwen3-VL-32B (Original)	65.0

Appendix

Table 7: MMEB-V2 performance under different evaluator configurations. “Alt. Prompt” denotes the use of an alternative prompt template.

Benchmark	w/o Selection	w/ Pair-Aware Selection
MMStar	60.2	63.2
ROCOv2	43.26	45.78

Appendix

Table 8: Performance with and without the pair-aware selection mechanism on a generative task (MMStar) and an out-of-domain benchmark (ROCOv2). ROCOv2 is evaluated using CUI@5.

Setting	MMEB-V2
Original	65.0
w/o question text	52.8
Random question text	43.1

Appendix

Table 9: MMEB-V2 performance under different question-text perturbations. The target items remain unchanged across all settings.

Panel of Table 10 — Table 10: Prompts used for the pair-aware evaluator $\mathcal{J}$ . The baseline prompt (top) evaluates query–target relevance without reasoning. The with-rationale prompt (bottom) includes generated rationales, enabling the counterfactual comparison $\Delta_{r}=c_{r}-c_{0}$ .

Panel of Table 11 — Table 11: Instruction template used during joint reasoning and embedding training (§ 3.3 ).

Baseline Models
Model	Per Meta-Task Score				Average Score
Model	Classification	VQA	Retrieval	Grounding	IND	OOD	Overall
# of Datasets	10	10	12	4	20	16	36
CLIP Radford et al. (2021)	42.8	9.1	53.0	51.8	37.1	38.7	37.8
BLIP-2 Li et al. (2023)	27.0	4.2	33.9	47.0	25.3	25.1	25.2
SigLIP Zhai et al. (2023)	40.3	8.4	31.6	59.5	32.3	38.0	34.8

Appendix

Table 13: Results on the MMEB-V1 benchmark, which consists of 36 image embedding tasks. IND and OOD denote in-distribution and out-of-distribution datasets, respectively. Some results are adopted from Embed-RL Jiang et al. (2026) .

Multimodal Embedding

Multimodal embedding maps text, images, audio, video, and other data into representations that support comparison and retrieval across modalities.

See all 160 papers

Think When Needed: Adaptive Reasoning-Driven Multimodal Embeddings with a Dual-LoRA Architecture
May 14, 2026Longxiang Zhang, Weilong Dai, Guanghao Zhang +2Efficient Multimodal Inference Multimodal Embedding
Alibaba Group
LaME: Learning to Think in Latent Space for Multimodal Embedding via Information Bottleneck
Jun 11, 2026Peixi Wu, Biao Yang, Feipeng Ma +7Efficient Multimodal Inference Multimodal Embedding
University of Science and Technology of China · 2Kuaishou Technology · 3Zhejiang University +1
Beyond Post-Hoc Rewards: Privileged Reasoning Exploration for Multimodal Embeddings
Oct 23, 2026Tianyu Yang, Yijun Tian, Lars Benedikt Kaesberg +4Multimodal Embedding Cross-Modal Retrieval
University of Göttingen · University of Notre Dame · State and University Library of Göttingen

May 14, 2026J/K move · Enter open · S save

Think When Needed: Adaptive Reasoning-Driven Multimodal Embeddings with a Dual-LoRA Architecture

Longxiang Zhang, Weilong Dai, Guanghao Zhang, Hao Jiang+1

Alibaba Group

Efficient Multimodal Inference Multimodal Embedding Cross-Modal Retrieval Multimodal CoT Reasoning

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) have emerged as a powerful backbone for multimodal embeddings. Recent methods introduce chain-of-thought (CoT) reasoning into the embedding pipeline to improve retrieval quality, but remain costly in both model size and inference cost. They typically employ separate reasoner and embedder with substantial parameter overhead, and generate CoT indiscriminately for every input. However, we observe that for simple inputs, discriminative embeddings already perform well, and redundant reasoning can even mislead the model, degrading performance. To address these limitations, we propose Think When Needed (TWN), a unified multimodal embedding framework with adaptive reasoning. TWN introduces a dual-LoRA architecture that attaches reasoning and embedding adapters to a shared frozen backbone, detaching gradients at their interface to mitigate gradient conflicts introduced by joint optimization while keeping parameters close to a single model. Building on this, an adaptive think mechanism uses a self-supervised routing gate to decide per input whether to generate CoT, skipping unnecessary reasoning to reduce inference overhead and even improve retrieval quality. We further explore embedding-guided RL to optimize CoT quality beyond supervised training. On the 78 tasks of MMEB-V2, TWN achieves state-of-the-art embedding quality while being substantially more efficient than existing generative methods, requiring only 3-5% additional parameters relative to the backbone and up to 50% fewer reasoning tokens compared to the full generative mode.

Abstract

Multimodal Embedding

Explore similar work

Think When Needed: Adaptive Reasoning-Driven Multimodal Embeddings with a Dual-LoRA Architecture

LaME: Learning to Think in Latent Space for Multimodal Embedding via Information Bottleneck

Beyond Post-Hoc Rewards: Privileged Reasoning Exploration for Multimodal Embeddings