Universal multimodal embedding (UME) maps heterogeneous inputs into a shared retrieval space with a single model. Recent approaches improve UME by generating explicit chain-of-thought (CoT) rationales before extracting embeddings, enabling multimodal large language models to better infer complex query intent. However, explicit CoT incurs substantial inference overhead and can compress rich multimodal evidence into a narrow textual bottleneck. We propose PLUME, a latent reasoning framework that advances UME by replacing verbalized CoT with a short autoregressive rollout of continuous latent states. To support diverse multimodal queries, PLUME further introduces a semantic-anchor-guided transition adapter that steers latent rollout along different reasoning trajectories under the same fixed computation budget. To stabilize training, PLUME adopts a progressive explicit-to-latent curriculum that uses verbalized reasoning only as a temporary training scaffold and gradually transfers this behavior into hidden-state computation, eliminating explicit CoT at inference. On the 78-task MMEB-v2 benchmark, PLUME outperforms strong explicit-CoT UME baselines while reducing reasoning from hundreds of generated tokens to fewer than 10 latent steps, delivering over 30x faster inference. PLUME is especially well suited to retrieval settings where relevant evidence is dense, structurally complex, and difficult to organize through verbalized intermediate rationales, such as video and visual document retrieval. These results show that structured latent computation can preserve the benefits of intermediate reasoning without the overhead of explicit rationale generation, providing a stronger and more efficient paradigm for practical retrieval systems.
Figures & tables
Figure 1: PLUME achieves a favorable accuracy–efficiency tradeoff on MMEB-v2. The x-axis shows inference throughput on a single H20 GPU and the y-axis shows average MMEB-v2 performance.
Figure 2: Comparison of three universal multimodal embedding paradigms. Left: early discriminative UME forms embeddings through single pass encoding, preserving efficiency but without explicitly modeling intermediate reasoning. Middle: explicit CoT UME improves reasoning by generating long textual traces before embedding extraction, but incurs substantial inference latency and token cost. Right: PLUME internalizes reasoning into a compact latent rollout and adapts the reasoning path with semantic-anchor-guided expert routing, achieving reasoning-aware embedding with substantially lower inference cost.
Figure 3: Overview of PLUME. Starting from a multimodal prefix, PLUME replaces explicit CoT decoding with a compact latent rollout inside the backbone. The bottom panel illustrates the latent rollout process, where the model performs several latent transitions before extracting the final retrieval embedding from the hidden state at <gen> . The top-left panel expands the semantic-anchor-guided transition adapter, which routes each latent step through shared and specialized experts, while the top-right panel shows the progressive explicit-to-latent curriculum that gradually rewrites explicit reasoning segments into latent blocks across training stages. The example in the bottom panel corresponds to an intermediate curriculum stage.
Model
Venue
Image
Video
VisDoc
All
CLS
QA
RET
GD
Overall
CLS
QA
RET
MRET
Overall
VDRv1
VDRv2
VR
OOD
Overall
# of Datasets
–
10
10
12
4
36
5
5
5
3
18
10
4
6
4
24
78
Early UME Baselines
LamRA
CVPR’25
59.2
26.5
70.0
62.7
54.1
39.3
42.6
24.3
34.6
35.2
22.0
11.5
37.4
21.0
23.9
40.4
VLM2Vec
ICLR’25
58.7
49.3
65.0
72.9
59.7
33.4
30.5
20.6
33.0
29.0
49.8
13.5
51.8
33.5
41.6
47.0
GME
CVPR’25
54.4
29.9
66.9
55.5
51.9
34.9
42.0
25.6
32.4
33.9
86.1
54.0
82.5
43.1
72.7
54.1
Table 1: Main comparison on MMEB-v2. We compare PLUME with early UME baselines and reasoning-enhanced UME methods. All methods share the same Qwen2-VL-2B backbone. Best and second-best results are highlighted in bold and underline .
Metric
PLUME
UME-R1
VLM2Vec-V2
Reasoning tokens/steps
8
403
0
Latency (ms/sample)
298 ± 12
9023 ± 187
156 ± 8
Throughput (samples/s)
3.3 ± 0.1
0.11 ± 0.01
6.4 ± 0.3
Speedup vs. UME-R1
30.3 ×
1.0 ×
–
Overhead vs. VLM2Vec-V2
1.9 ×
–
1.0 ×
Table 2: Inference efficiency on a single H20 GPU
Figure 4: Per task performance comparison on MMEB-v2.
Configuration
Image
Video
VisDoc
All
Full PLUME
66.3
44.1
67.5
61.6
w/o Latent Transition
63.6
41.0
64.8
58.8
w/o MoE (single MLP)
64.2
41.8
64.4
59.2
w/o Semantic Anchor
65.4
42.3
66.1
60.1
w/o Curriculum
60.2
36.5
60.2
54.8
Table 3: Ablation on the core components of PLUME.
K
Image
Video
VisDoc
All
Latency (ms)
4
64.3
43.3
65.7
59.9
232
6
65.9
43.6
66.7
61.1
268
8
66.3
44.1
67.5
61.6
300
Table 4: Effect of the latent steps K on accuracy and latency.
Configuration
Image
Video
VisDoc
All
Default ( Me=4 , Kr=2 , shared)
66.3
44.1
67.5
61.6
w/o Shared Expert
65.4
42.5
66.1
60.3
Top-1 expert (instead of Top-2)
65.8
43.0
66.8
60.8
Router: w/o c(x)
65.4
42.3
66.1
60.1
Router: w/o e(k)
65.7
41.9
66.2
60.4
Table 5: Ablation on the transition adapter design.
Figure 5: Activation preferences of specialized experts across image and video retrieval sub-tasks.
Figure 6: Average cosine similarity between intermediate states and the positive target over 200 samples, reported separately on image and video retrieval. PLUME shows a smoother trajectory with consistently smaller variance than UME-R1 across reasoning steps.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
# Stages
Image
Video
VisDoc
Avg.
2
64.0
42.1
64.3
59.0
4
66.3
44.1
67.5
61.6
6
66.4
43.7
67.3
61.4
8
65.6
43.6
66.7
60.9
Appendix
Table 6: Ablation on the number of curriculum stages. All models are trained with the same data, backbone, and total training recipe unless otherwise specified. The best result is in bold and the second best is underlined.
Figure 7: Failure cases on ChartQA. Each example shows the input chart and query at the top, and the top retrieved predictions from PLUME at the bottom. The green box marks the ground-truth target, while the preceding orange boxes denote higher-ranked distractors. In Case 1, PLUME fails on a count-over-threshold query, ranking 4 above the correct answer 5 . In Case 2, PLUME struggles with multi-step numerical aggregation, ranking several nearby values ahead of the correct average 21.5 . These examples suggest that although PLUME usually preserves the coarse semantic intent of the query, compressing reasoning into a short latent rollout can still weaken the preservation of fine-grained numerical relations and localized textual details required by chart-intensive question answering.
Figure 8: Failure case on InfographicsVQA. The figure shows an infographic-style input, the query, the ground-truth target, and the top retrieved predictions from PLUME. The green box marks the correct target, while the orange boxes denote higher-ranked or nearby distractors. In this example, the target answer is facebook , but PLUME ranks a broader candidate, twitter, facebook, pinterest , above the exact target. It also retrieves semantically related but overly coarse or mismatched variants such as twitter, facebook and facebook ads . This case suggests that on text-dense infographic inputs, PLUME can capture the relevant semantic region but may fail to preserve the precise answer granularity needed for exact retrieval.
Figure 9: Average activation rate of each expert across input modalities. T, I, V, D, TI, and TV denote text, image, video, document, text-image, and text-video inputs, respectively. Darker cells indicate higher expert activation rates.
Task
GME-2B
LamRA
VLM2Vec
VLM2Vec-V2.0
DUME-2B
UME-R1-2B
PLUME
Avg - All (78 tasks)
54.1
40.5
46.9
58.1
52.7
60.1
61.6
Avg - Image (36 tasks, Hit@1)
51.9
54.1
59.7
64.9
62.5
66.6
66.3
I-CLS (10)
54.6
59.3
58.6
62.9
59.3
64.8
66.5
I-QA (10)
29.8
26.5
49.2
56.4
54.9
62.8
59.2
I-RET (12)
66.8
70.1
65.0
69.6
66.3
67.6
67.6
I-VG (4)
55.6
62.6
73.1
77.1
78.0
77.2
79.7
Appendix
Table 7: Full MMEB-V2 results with selected baselines (Part I: Overall and Image). “VLM2Vec” denotes VLM2Vec-7B.
Task
GME-2B
LamRA
VLM2Vec
VLM2Vec-V2.0
DUME-2B
UME-R1-2B
PLUME
Avg - Video (18 tasks, Hit@1)
33.6
35.2
28.6
34.7
33.2
42.2
44.1
Avg - Visdoc (24 tasks, NDCG@5)
72.7
23.9
41.6
65.4
52.8
63.9
67.5
V-CLS (5)
34.8
39.4
33.3
39.2
37.7
44.3
45.0
V-QA (5)
41.8
42.7
30.7
34.7
46.6
51.0
52.3
V-RET (5)
25.4
24.5
20.5
28.4
17.1
32.9
33.5
V-MR (3)
31.8
33.6
30.7
37.6
30.0
39.7
46.7
Appendix
Table 8: Full MMEB-V2 results with selected baselines (Part II: Video and VisDoc). “VLM2Vec” denotes VLM2Vec-7B.
Figure 10: A case where explicit CoT gets trapped by a misleading textual detail while PLUME recovers the correct target. In this example, explicit CoT over-focuses on the spurious text cue “ARMYJUNK,” which biases the subsequent reasoning trace and leads to an incorrect retrieval result. In contrast, PLUME does not commit to a discrete verbal rationale and still retrieves the correct target half track after latent rollout.
Figure 11: A case where explicit CoT over-compresses the retrieval intent while PLUME preserves the correct target. In this example, explicit CoT summarizes the query into an overly coarse verbal description and is distracted toward an incorrect retrieval result. In contrast, PLUME avoids committing to a discrete textual rationale and successfully retrieves the correct target after latent rollout.
Figure 12: A case where explicit CoT is misled by a surface-level prior while PLUME recovers the correct answer. In this example, explicit CoT over-relies on the visual prior that egg cartons are typically made of cardboard and therefore predicts an incorrect result. In contrast, PLUME avoids being locked into this explicit verbal path and successfully retrieves the correct answer styrofoam after latent rollout.
Figure 13: A case where explicit CoT drifts toward an incorrect action description while PLUME recovers the correct target. In this example, explicit CoT over-compresses the video into a coarse description of two pens on a wooden floor and is distracted toward an incorrect retrieval result. In contrast, PLUME avoids committing to this explicit verbal path and successfully retrieves the correct target putting pen similar to others that are already on the table after latent rollout.
Tsinghua Shenzhen International Graduate School, Tsinghua University · School of Artificial Intelligence and Data Science, USTC · Kuaishou Technology +1