Organizations: Pattern Recognition Center, WeChat AI, Tencent Inc. · College of Computer Science, Sichuan University · Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education
Text embedding and text generation are two of the most useful capabilities of modern large language models, yet they are typically provided by separate specialized models (an embedder and a generator) trained with incompatible objectives and deployed side by side in retrieval-augmented generation and agent systems. Recent unified embedding-generation models such as GritLM share parameters between the two capabilities, but they either switch attention regimes between modes or optimize embedding and language modeling as parallel objectives that never supervise each other. We argue that these designs are only partially unified: the two capabilities coexist on the same weights, but do not shape each other. We propose ULME, a post-training framework that unifies embedding and language modeling on a causal decoder-only LLM through predictive coupling. ULME inserts a chain of learnable slots between the query and the answer, reads a semantic embedding and a reasoning embedding from two designated slots, and trains three objectives on a shared causal backbone: a contrastive loss anchoring the semantic slot to document representations, a language-modeling loss supervising answer generation, and a stop-gradient predictive loss that aligns the reasoning slot with a pooled summary of the answer, so that generation actively shapes what the embedding encodes. Across two model families (Qwen2.5, Qwen3) and eight scales spanning 0.5B-8B parameters, a single ULME checkpoint improves MTEB by 3.3-4.1 points over the strongest open-source baseline, tightens BRIGHT and AdvBench-IR simultaneously, and remains on par with a generation-only fine-tune on standard accuracy benchmarks (MMLU, GSM8K, MMLU-Pro, HumanEval), demonstrating that genuine unification yields mutually reinforcing rather than merely coexisting capabilities.
Figures & tables
Figure 1: Overview of ULME. A single causal decoder is trained with three objectives on a shared chain of learnable slot tokens (flame icons: trainable slots; grey blocks: hidden states). Left ( Lsem ): symmetric InfoNCE pulls the query and document slot embeddings Eq,Ed together. Middle ( Lgen ): next-token prediction over the answer span A , with the slot chain on the causal path from Q to A . Right ( Lpred ): a one-sided InfoNCE aligns the query-side slot with the answer-side slot under stop-gradient on the answer side. Total loss: L=Lsem+Lgen+Lpred .
Method
Retr.
Rerank.
Clust.
PairCls.
Class.
STS
Summ.
Avg.
Qwen3-1.7B
Echo ( Springer et al. 2025 )
4.4
37.7
38.1
59.3
70.4
52.1
-0.8
39.8
HyDE ( Gao et al. 2023 )
17.7
39.5
38.2
69.4
75.6
72.2
18.0
49.8
InBedder ( Peng et al. 2024 )
15.7
41.2
47.0
68.6
79.2
65.9
9.3
50.2
GIRCSE ( Tsai et al. 2025 )
27.3
40.4
47.2
78.2
70.6
66.6
22.2
52.5
LLM2Vec ( BehnamGhader et al. 2024 )
34.9
39.9
41.1
76.4
71.1
73.4
30.4
54.8
Table 1: Results on MTEB(eng, v2). Rows are grouped by backbone; columns 2–8 are per-category means over Retrieval, Reranking, Clustering, PairClassification, Classification, STS, and Summarization; the final column is the overall task-mean over all 41 tasks. Bold marks the best score in each column within a backbone group.
Model
MMLU
GSM8K
MMLU-Pro
HumanEval
HumanEval+
Qwen3-1.7B
Backbone
63.7
76.0
39.3
54.9
48.2
Gen-only FT
63.0
74.2
38.6
53.7
46.3
ULME (ours)
63.5 +0.5
75.3 +1.1
39.6 +1.0
56.1 +2.4
49.4 +3.1
Qwen3-4B
Backbone
72.8
87.9
52.9
68.3
62.2
Table 2: Generation results. Backbone (greyed) is the pre-training checkpoint we start from: Qwen2.5-Instruct ( Team and others 2024 ) or Qwen3 non-thinking ( Yang et al. 2025 ) , not trained by us. Gen-only FT fine-tunes the same backbone on the same OpenThoughts-114k data used by ULME. Bold marks the better of Gen-only FT and ULME per column.
Backbone
Method
AdvBench-IR ↓
BRIGHT ↑
Qwen3-1.7B
LLM2Vec
46.7
14.0
LLM2VEC-GEN
36.2
15.6
ULME (ours)
34.9 -1.3
16.3 +0.7
Qwen3-4B
LLM2Vec
50.8
15.7
LLM2VEC-GEN
42.5
18.8
ULME (ours)
40.9 -1.6
19.6 +0.8
Table 3: Reasoning-intensive retrieval on BRIGHT (nDCG@10, higher is better) and AdvBench-IR (lower is better). Bold marks the best result in each column within a backbone group.
Slots
MTEB-lite
MMLU
GSM8K
N=0
58.6
74.5
87.8
N=1
62.4
76.1
89.4
N=2
63.1
76.4
89.8
N=4
63.8
76.5
90.0
N=8
63.6
76.3
89.9
Table 4: Ablation on the number of embedding slots N . Highlighted row is our default setting.
Lsem
Lgen
Lpred
MTEB-lite
MMLU
GSM8K
✓
✓
31.2
76.6
90.1
✓
✓
61.0
61.8
74.2
✓
✓
60.2
76.4
89.9
✓
✓
✓
63.8
76.5
90.0
Table 5: Loss decomposition. ✓ = loss enabled, blank = removed. Highlighted row is our full model.
rqa pooling
MTEB-lite
MMLU
GSM8K
Mean over Ha
63.8
76.5
90.0
Last-token of a
61.9
76.3
89.8
Length-weighted mean
62.7
76.4
89.9
Table 6: Ablation on the predictive target rqa . Highlighted row is our default setting.
Figure 2: Per-task performance gap between the semantic slot (slot 0 ) and the reasoning slot (slot N−1 ) across the 41 tasks of MTEB(eng, v2). Positive bars favor slot 0 ; negative bars favor slot N−1 . The red dashed line marks the mean gap ( +0.65 ).
Hyperparameter
Value
Architecture
Number of embedding slots N
4
Semantic slot index
0
Reasoning slot index
N−1
Answer pooling rqa
mean over Ha
Contrastive projection
1-layer linear, 4096 dims
Table 7: Training hyperparameters used across all eight ULME configurations. The only setting that differs between runs is the backbone identifier.
Method
Retr.
Rerank.
Clust.
PairCls.
Class.
STS
Summ.
Avg.
Qwen2.5-0.5B-Instruct
LLM2Vec ( BehnamGhader et al. 2024 )
27.35
35.62
35.14
66.83
65.28
62.47
22.68
45.05
ULME (ours)
32.86
38.74
40.15
74.62
68.53
69.31
25.42
51.38
Qwen2.5-1.5B-Instruct
LLM2Vec ( BehnamGhader et al. 2024 )
35.14
38.85
38.72
71.68
69.42
68.16
26.85
52.34
ULME (ours)
40.16
42.03
44.86
80.14
72.85
76.28
29.76
59.34
Table 8: Full MTEB(eng, v2) results across all eight backbones. Bold marks the better of LLM2Vec and ULME per column.
Model
MMLU
GSM8K
MMLU-Pro
HumanEval
HumanEval+
Qwen2.5-0.5B-Instruct
Backbone
47.07
46.47
11.69
29.88
28.04
Gen-only FT
46.89
44.13
13.65
27.44
25.61
ULME (ours)
47.65 +0.76
43.75 -0.38
14.84 +1.19
29.27 +1.83
26.22 +0.61
Qwen2.5-1.5B-Instruct
Backbone
60.85
73.24
32.38
61.59
55.49
Table 9: Full generation results across all eight backbones. Bold marks the better of Gen-only FT and ULME per column; subscripts on ULME rows report the absolute gain over Gen-only FT.
Task
Query-side instruction
Document-side instruction
Clustering (single-side; d=∅ )
ArXivHierarchicalClusteringP2P
Represent this arXiv title and abstract for topic clustering:
∅
ArXivHierarchicalClusteringS2S
Represent this arXiv paper title for topic clustering:
∅
BiorxivClusteringP2P.v2
Represent this bioRxiv title and abstract for topic clustering:
∅
MedrxivClusteringP2P.v2
Represent this medRxiv title and abstract for topic clustering:
∅
MedrxivClusteringS2S.v2
Represent this medRxiv paper title for topic clustering:
∅
Table 10: Task-specific evaluation instructions used for MTEB(eng, v2). Every prompt is a one-line prefix ending in “: ” and the raw input is appended immediately after. “ =q ” indicates that the document-side prompt reuses the query-side wording verbatim; “ ∅ ” indicates an empty document-side prompt (single-side tasks).
Multimodal Large Language Models (MLLMs) have emerged as a promising foundation for universal multimodal embeddings. Recent studies have shown that reasoning-driven generative multimodal embeddings can outperform discriminative embeddings on several embedding tasks. However, Chain-of-Thought (CoT) reasoning tends to generate redundant thinking steps and introduce semantic ambiguity in the summarized answers in broader retrieval scenarios. To address this limitation, we propose Rewrite-driven Multimodal Embedding (RIME), a unified framework that jointly optimizes generation and embedding through a retrieval-friendly rewrite. Meanwhile, we present the Cross-Mode Alignment (CMA) to bridge the generative and discriminative embedding spaces, enabling flexible mutual retrieval to trade off efficiency and accuracy. Based on this, we also introduce Refine Reinforcement Learning (Refine-RL) that treats discriminative embeddings as stable semantic anchors to guide the rewrite optimization. Extensive experiments on MMEB-V2, MRMR and UVRB demonstrate that RIME substantially outperforms prior generative embedding models while significantly reducing the length of thinking. Code is available at https://github.com/PeppaWu/RIME.
Peixi Wu, Ke Mei, Feipeng Ma +15
WeChat Vision, Tencent Inc. · Zhejiang University · Tsinghua University +1
Reasoning-driven universal multimodal embedding has advanced rapidly by introducing Chain-of-Thought (CoT) reasoning into the embedding pipeline. Despite the strong performance across both general and complex tasks, this paradigm suffers from two core limitations: (i) autoregressive CoT reasoning incurs high computational cost, making it impractical for low-latency retrieval; and (ii) embedding performance is heavily coupled with CoT annotation quality, making large-scale training unreliable. These raise fundamental questions: Is textual CoT the optimal form of reasoning for embedding, and can effective embedding reasoning be accomplished in latent space? To this end, we propose LaME (Latent Reasoning Multimodal Embedding), which formulates embedding-oriented latent reasoning as a weakly supervised information bottleneck. LaME employs K learnable reason tokens as a fixed-capacity bottleneck, completing all reasoning within a single forward pass. The two weak supervision signals structurally decouple contrastive from autoregressive objectives and eliminate dependence on CoT annotations, while a two-stage training pipeline ensures stable convergence. Experiments on MMEB-v2 and MRMR show that LaME achieves competitive performance, surpassing some explicit CoT-based models, while delivering 60x faster inference than explicit CoT methods and 2x faster than latent baselines with throughput comparable to discriminative embedding models. Code is available at https://github.com/PeppaWu/LaME.
Peixi Wu, Biao Yang, Feipeng Ma +7
University of Science and Technology of China · 2Kuaishou Technology · 3Zhejiang University +1
Universal multimodal embedding (UME) maps heterogeneous inputs into a shared retrieval space with a single model. Recent approaches improve UME by generating explicit chain-of-thought (CoT) rationales before extracting embeddings, enabling multimodal large language models to better infer complex query intent. However, explicit CoT incurs substantial inference overhead and can compress rich multimodal evidence into a narrow textual bottleneck. We propose PLUME, a latent reasoning framework that advances UME by replacing verbalized CoT with a short autoregressive rollout of continuous latent states. To support diverse multimodal queries, PLUME further introduces a semantic-anchor-guided transition adapter that steers latent rollout along different reasoning trajectories under the same fixed computation budget. To stabilize training, PLUME adopts a progressive explicit-to-latent curriculum that uses verbalized reasoning only as a temporary training scaffold and gradually transfers this behavior into hidden-state computation, eliminating explicit CoT at inference. On the 78-task MMEB-v2 benchmark, PLUME outperforms strong explicit-CoT UME baselines while reducing reasoning from hundreds of generated tokens to fewer than 10 latent steps, delivering over 30x faster inference. PLUME is especially well suited to retrieval settings where relevant evidence is dense, structurally complex, and difficult to organize through verbalized intermediate rationales, such as video and visual document retrieval. These results show that structured latent computation can preserve the benefits of intermediate reasoning without the overhead of explicit rationale generation, providing a stronger and more efficient paradigm for practical retrieval systems.
Chenwei He, Xiangzhao Hao, Tianyu Yang +6
Southeast University · Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences