ULME: Predictive Coupling for Mutually Reinforcing Embedding and Language Modeling

Yang Liu; Zhiqiang Yuan; Shu-Dong Huang; Jiancheng Lv; Liwei Yang; Qiwei Yan; Kailin Lv; Jinchao Zhang

cs.CLOct 22, 2026

ULME: Predictive Coupling for Mutually Reinforcing Embedding and Language Modeling

Authors: Yang Liu, Zhiqiang Yuan, Shu-Dong Huang, Jiancheng Lv, Liwei Yang, Qiwei Yan, Kailin Lv, Jinchao Zhang

Organizations: Pattern Recognition Center, WeChat AI, Tencent Inc. · College of Computer Science, Sichuan University · Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education

Abstract

Text embedding and text generation are two of the most useful capabilities of modern large language models, yet they are typically provided by separate specialized models (an embedder and a generator) trained with incompatible objectives and deployed side by side in retrieval-augmented generation and agent systems. Recent unified embedding-generation models such as GritLM share parameters between the two capabilities, but they either switch attention regimes between modes or optimize embedding and language modeling as parallel objectives that never supervise each other. We argue that these designs are only partially unified: the two capabilities coexist on the same weights, but do not shape each other. We propose ULME, a post-training framework that unifies embedding and language modeling on a causal decoder-only LLM through predictive coupling. ULME inserts a chain of learnable slots between the query and the answer, reads a semantic embedding and a reasoning embedding from two designated slots, and trains three objectives on a shared causal backbone: a contrastive loss anchoring the semantic slot to document representations, a language-modeling loss supervising answer generation, and a stop-gradient predictive loss that aligns the reasoning slot with a pooled summary of the answer, so that generation actively shapes what the embedding encodes. Across two model families (Qwen2.5, Qwen3) and eight scales spanning 0.5B-8B parameters, a single ULME checkpoint improves MTEB by 3.3-4.1 points over the strongest open-source baseline, tightens BRIGHT and AdvBench-IR simultaneously, and remains on par with a generation-only fine-tune on standard accuracy benchmarks (MMLU, GSM8K, MMLU-Pro, HumanEval), demonstrating that genuine unification yields mutually reinforcing rather than merely coexisting capabilities.

Figures & tables

Figure 1: Overview of ULME. A single causal decoder is trained with three objectives on a shared chain of learnable slot tokens (flame icons: trainable slots; grey blocks: hidden states). Left (

\mathcal{L}_{\text{sem}}

): symmetric InfoNCE pulls the query and document slot embeddings

E_{q},E_{d}

together. Middle (

\mathcal{L}_{\text{gen}}

): next-token prediction over the answer span

A

, with the slot chain on the causal path from

Q

A

. Right (

\mathcal{L}_{\text{pred}}

): a one-sided InfoNCE aligns the query-side slot with the answer-side slot under stop-gradient on the answer side. Total loss:

\mathcal{L}=\mathcal{L}_{\text{sem}}+\mathcal{L}_{\text{gen}}+\mathcal{L}_{\text{pred}}

Method	Retr.	Rerank.	Clust.	PairCls.	Class.	STS	Summ.	Avg.
Qwen3-1.7B
Echo ( Springer et al. 2025 )	4.4	37.7	38.1	59.3	70.4	52.1	-0.8	39.8
HyDE ( Gao et al. 2023 )	17.7	39.5	38.2	69.4	75.6	72.2	18.0	49.8
InBedder ( Peng et al. 2024 )	15.7	41.2	47.0	68.6	79.2	65.9	9.3	50.2
GIRCSE ( Tsai et al. 2025 )	27.3	40.4	47.2	78.2	70.6	66.6	22.2	52.5
LLM2Vec ( BehnamGhader et al. 2024 )	34.9	39.9	41.1	76.4	71.1	73.4	30.4	54.8

Table 1: Results on MTEB(eng, v2). Rows are grouped by backbone; columns 2–8 are per-category means over Retrieval, Reranking, Clustering, PairClassification, Classification, STS, and Summarization; the final column is the overall task-mean over all 41 tasks. Bold marks the best score in each column within a backbone group.

Model	MMLU	GSM8K	MMLU-Pro	HumanEval	HumanEval+
Qwen3-1.7B
Backbone	63.7	76.0	39.3	54.9	48.2
Gen-only FT	63.0	74.2	38.6	53.7	46.3
ULME (ours)	63.5 +0.5	75.3 +1.1	39.6 +1.0	56.1 +2.4	49.4 +3.1
Qwen3-4B
Backbone	72.8	87.9	52.9	68.3	62.2

Table 2: Generation results. Backbone (greyed) is the pre-training checkpoint we start from: Qwen2.5-Instruct ( Team and others 2024 ) or Qwen3 non-thinking ( Yang et al. 2025 ) , not trained by us. Gen-only FT fine-tunes the same backbone on the same OpenThoughts-114k data used by ULME. Bold marks the better of Gen-only FT and ULME per column.

Backbone	Method	AdvBench-IR $\downarrow$	BRIGHT $\uparrow$
Qwen3-1.7B	LLM2Vec	46.7	14.0
	LLM2VEC-GEN	36.2	15.6
	ULME (ours)	34.9 -1.3	16.3 +0.7
Qwen3-4B	LLM2Vec	50.8	15.7
	LLM2VEC-GEN	42.5	18.8
	ULME (ours)	40.9 -1.6	19.6 +0.8

Table 3: Reasoning-intensive retrieval on BRIGHT (nDCG@10, higher is better) and AdvBench-IR (lower is better). Bold marks the best result in each column within a backbone group.

Slots	MTEB-lite	MMLU	GSM8K
$N=0$	58.6	74.5	87.8
$N=1$	62.4	76.1	89.4
$N=2$	63.1	76.4	89.8
$N=4$	63.8	76.5	90.0
$N=8$	63.6	76.3	89.9

Table 4: Ablation on the number of embedding slots

N

. Highlighted row is our default setting.

$\mathcal{L}_{\text{sem}}$	$\mathcal{L}_{\text{gen}}$	$\mathcal{L}_{\text{pred}}$	MTEB-lite	MMLU	GSM8K
	✓	✓	31.2	76.6	90.1
✓		✓	61.0	61.8	74.2
✓	✓		60.2	76.4	89.9
✓	✓	✓	63.8	76.5	90.0

Table 5: Loss decomposition. ✓ = loss enabled, blank = removed. Highlighted row is our full model.

$r_{qa}$ pooling	MTEB-lite	MMLU	GSM8K
Mean over $H_{a}$	63.8	76.5	90.0
Last-token of $a$	61.9	76.3	89.8
Length-weighted mean	62.7	76.4	89.9

Table 6: Ablation on the predictive target

r_{qa}

. Highlighted row is our default setting.

Figure 2: Per-task performance gap between the semantic slot (slot

0

) and the reasoning slot (slot

N{-}1

) across the 41 tasks of MTEB(eng, v2). Positive bars favor slot

0

; negative bars favor slot

N{-}1

. The red dashed line marks the mean gap (

+0.65

Architecture
Hyperparameter	Value
Number of embedding slots $N$	$4$
Semantic slot index	$0$
Reasoning slot index	$N{-}1$
Answer pooling $r_{qa}$	mean over $H_{a}$
Contrastive projection	1-layer linear, $4096$ dims

Table 7: Training hyperparameters used across all eight ULME configurations. The only setting that differs between runs is the backbone identifier.

Method	Retr.	Rerank.	Clust.	PairCls.	Class.	STS	Summ.	Avg.
Qwen2.5-0.5B-Instruct
LLM2Vec ( BehnamGhader et al. 2024 )	27.35	35.62	35.14	66.83	65.28	62.47	22.68	45.05
ULME (ours)	32.86	38.74	40.15	74.62	68.53	69.31	25.42	51.38
Qwen2.5-1.5B-Instruct
LLM2Vec ( BehnamGhader et al. 2024 )	35.14	38.85	38.72	71.68	69.42	68.16	26.85	52.34
ULME (ours)	40.16	42.03	44.86	80.14	72.85	76.28	29.76	59.34

Table 8: Full MTEB(eng, v2) results across all eight backbones. Bold marks the better of LLM2Vec and ULME per column.

Model	MMLU	GSM8K	MMLU-Pro	HumanEval	HumanEval+
Qwen2.5-0.5B-Instruct
Backbone	47.07	46.47	11.69	29.88	28.04
Gen-only FT	46.89	44.13	13.65	27.44	25.61
ULME (ours)	47.65 +0.76	43.75 -0.38	14.84 +1.19	29.27 +1.83	26.22 +0.61
Qwen2.5-1.5B-Instruct
Backbone	60.85	73.24	32.38	61.59	55.49

Table 9: Full generation results across all eight backbones. Bold marks the better of Gen-only FT and ULME per column; subscripts on ULME rows report the absolute gain over Gen-only FT.

Clustering (single-side; $d=\emptyset$ )
Task	Query-side instruction	Document-side instruction
ArXivHierarchicalClusteringP2P	Represent this arXiv title and abstract for topic clustering:	$\emptyset$
ArXivHierarchicalClusteringS2S	Represent this arXiv paper title for topic clustering:	$\emptyset$
BiorxivClusteringP2P.v2	Represent this bioRxiv title and abstract for topic clustering:	$\emptyset$
MedrxivClusteringP2P.v2	Represent this medRxiv title and abstract for topic clustering:	$\emptyset$
MedrxivClusteringS2S.v2	Represent this medRxiv paper title for topic clustering:	$\emptyset$

Table 10: Task-specific evaluation instructions used for MTEB(eng, v2). Every prompt is a one-line prefix ending in “: ” and the raw input is appended immediately after. “

=q

” indicates that the document-side prompt reuses the query-side wording verbatim; “

\emptyset

” indicates an empty document-side prompt (single-side tasks).

Text Embeddings

Text embeddings map words, sentences, or documents to vectors for measuring meaning, retrieving information, and organizing text.

See all 56 papers

Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings
Apr 24, 2026Peixi Wu, Ke Mei, Feipeng Ma +15Cross-Modal Alignment Multimodal Embedding
WeChat Vision, Tencent Inc. · Zhejiang University · Tsinghua University +1
LaME: Learning to Think in Latent Space for Multimodal Embedding via Information Bottleneck
Jun 11, 2026Peixi Wu, Biao Yang, Feipeng Ma +7Efficient Multimodal Inference Multimodal Embedding
University of Science and Technology of China · 2Kuaishou Technology · 3Zhejiang University +1
PLUME: Latent Reasoning Based Universal Multimodal Embedding
Apr 2, 2026Chenwei He, Xiangzhao Hao, Tianyu Yang +6Multimodal Embedding Multimodal IR
Southeast University · Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences

Apr 24, 2026J/K move · Enter open · S save

Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings

Peixi Wu, Ke Mei, Feipeng Ma, Bosong Chai+14

WeChat Vision, Tencent Inc. · Zhejiang University · Tsinghua University+1

Cross-Modal Alignment Multimodal Embedding Cross-Modal Retrieval

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have emerged as a promising foundation for universal multimodal embeddings. Recent studies have shown that reasoning-driven generative multimodal embeddings can outperform discriminative embeddings on several embedding tasks. However, Chain-of-Thought (CoT) reasoning tends to generate redundant thinking steps and introduce semantic ambiguity in the summarized answers in broader retrieval scenarios. To address this limitation, we propose Rewrite-driven Multimodal Embedding (RIME), a unified framework that jointly optimizes generation and embedding through a retrieval-friendly rewrite. Meanwhile, we present the Cross-Mode Alignment (CMA) to bridge the generative and discriminative embedding spaces, enabling flexible mutual retrieval to trade off efficiency and accuracy. Based on this, we also introduce Refine Reinforcement Learning (Refine-RL) that treats discriminative embeddings as stable semantic anchors to guide the rewrite optimization. Extensive experiments on MMEB-V2, MRMR and UVRB demonstrate that RIME substantially outperforms prior generative embedding models while significantly reducing the length of thinking. Code is available at https://github.com/PeppaWu/RIME.

Abstract

Text Embeddings

Explore similar work

Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings

LaME: Learning to Think in Latent Space for Multimodal Embedding via Information Bottleneck

PLUME: Latent Reasoning Based Universal Multimodal Embedding