TSEmbed: Unlocking Task Scaling in Universal Multimodal Embeddings

Yebo Wu; Feng Liu; Ziwei Xie; Zhiyuan Liu; Changwang Zhang; Jun Wang; Li Li

cs.CLMar 5, 2026

TSEmbed: Unlocking Task Scaling in Universal Multimodal Embeddings

Authors: Yebo Wu, Feng Liu, Ziwei Xie, Zhiyuan Liu, Changwang Zhang, Jun Wang, Li Li

Organizations: State Key Laboratory of IOTSC, University of Macau · OPPO Research Institute

Abstract

Despite the exceptional reasoning capabilities of Multimodal Large Language Models (MLLMs), their adaptation into universal embedding models is significantly impeded by task conflict. To address this, we propose TSEmbed, a universal multimodal embedding framework that synergizes Mixture-of-Experts (MoE) with Low-Rank Adaptation (LoRA) to explicitly disentangle conflicting task objectives. Moreover, we introduce Expert-Aware Negative Sampling (EANS), a novel strategy that leverages expert routing distributions as an intrinsic proxy for semantic similarity. By dynamically prioritizing informative hard negatives that share expert activation patterns with the query, EANS effectively sharpens the model's discriminative power and refines embedding boundaries. To ensure training stability, we further devise a two-stage learning paradigm that solidifies expert specialization before optimizing representations via EANS. TSEmbed achieves state-of-the-art performance on both the Massive Multimodal Embedding Benchmark (MMEB) and real-world industrial production datasets, laying a foundation for task-level scaling in universal multimodal embeddings.

Figures & tables

Panel of Figure 1 — Figure 1: Impact of task conflict on model performance. Red annotations ( $\downarrow$ ) indicate the performance drop when switching from task-specific models to the unified VLM2VEC.

Panel of Figure 2 — Figure 2: A Multidimensional Anatomy of Task Conflict in Monolithic Adapters. (a) Divergent optimization trajectories of isolated task-specific adapters. (b) Heterogeneous convergence dynamics of individual tasks during training. (c) Layer-wise cosine similarity between the jointly trained adapter and its isolated counterparts.

Figure 3: Overview of existing work and our proposed TSEmbed .

Encoder-Only Models
Model	Backbone	Model Size	Per Meta-Task Score				Average Score
Model	Backbone	Model Size	Classification	VQA	Retrieval	Grounding	IND	OOD	Overall
# of datasets $\rightarrow$			10	10	12	4	20	16	36
CLIP [ 28 ]	-	0.428B	42.8	9.1	53.0	51.8	37.1	38.7	37.8
BLIP-2 [ 20 ]	-	3.74B	27.0	4.2	33.9	47.0	25.3	25.1	25.2
SigLIP [ 33 ]	-	0.203B	40.3	8.4	31.6	59.5	32.3	38.0	34.8

Table 1: Performance comparison of various multimodal embedding models on the MMEB [ 16 ] . Bold and underlined values denote the best and second-best results within each parameter scale, respectively. † indicates a link to the model’s official homepage.

Panel of Figure 6 — Figure 6: Sensitivity analysis of the warm-up duration $T_{\text{warmup}}$ . We evaluate the impact of $T_{\text{warmup}}$ on four meta-task categories using both 2B and 7B models.

Configuration	Components			Per Meta-Task Score				Overall
Configuration	MoE	EANS	Two-Stage	Cls.	VQA	Ret.	Grd.	Overall
	Qwen2-VL-2B [ 35 ]
Baseline (VLM2VEC)	✗	✗	✗	59.00	49.40	65.40	73.40	59.30
+ MoE	✓	✗	✗	67.87	64.17	72.03	85.65	70.20
+ MoE + EANS	✓	✓	✗	68.31	64.26	71.59	85.08	70.14
TSEmbed (Full)	✓	✓	✓	68.77	64.29	72.09	85.73	70.52

Table 3: Ablation study of TSEmbed . ✓/✗ indicate enabled/disabled modules.

Panel of Figure 7 — Figure 7: Impact of expert number $N$ on model performance. We evaluate $N\in\{2,4,6,8\}$ across four meta-task categories on both 2B and 7B models.

Panel of Figure 8 — Figure 8: Impact of layer coverage on EANS routing aggregation. We evaluate the effect of aggregating router distributions from 25%, 50%, 75%, and 100% of layers.

Multimodal Embedding

Multimodal embedding maps text, images, audio, video, and other data into representations that support comparison and retrieval across modalities.

See all 160 papers

MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Models
Sep 8, 2026Xuanming Cui, Shlok Kumar Mishra, Wentao Bao +6Efficient Multimodal Inference Multimodal Embedding
University of Central Florida · AI at Meta
Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings
Apr 24, 2026Peixi Wu, Ke Mei, Feipeng Ma +15Cross-Modal Alignment Multimodal Embedding
WeChat Vision, Tencent Inc. · Zhejiang University · Tsinghua University +1
Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMs
Feb 5, 2026Qi Li, Yanzhe Zhao, Yongxin Zhou +4Efficient Multimodal Inference Multimodal Large Language Models
Honor Device Co., Ltd., China

Sep 8, 2026J/K move · Enter open · S save

MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Models

Xuanming Cui, Shlok Kumar Mishra, Wentao Bao, Aashu Singh+5

University of Central Florida · AI at Meta

Efficient Multimodal Inference Multimodal Embedding Mixture-of-Experts Models Adaptive Computation

Open paper AskPDF arXiv

Universal multimodal embedding (UME) increasingly demands encoder's capacity for handling a broad range of tasks and modalities with increased complexity. Prior scaling methods either increase the representation size, retrieval effort, or scales the encoder into a heavy multimodal LLM. Recent works, such as Think-Then-Embed (TTE), explore scaling via reasoning tokens. However, embedding models are hard to scale up: increasing parameters directly tradeoffs for the large training batch size that contrastive learning needs, and retrieval has to be served under tight latency. Moreover, UME tasks are diverse in complexity, where scaling up embedders can bring significant redundant computation. In this work, we propose MOEMB, which instead scales UME along the expert axis through mixture-of-experts (MoE), growing encoder capacity while preserving single-vector, non-autoregressive encoding. Through a systematic study of the design space and training recipes for MoE-based UME, MoEMB sets a new state of the art on both MMEB-V2 and MRMR among models trained on public MMEB-family data: with only 3B active parameters, MoEMB surpasses TTE-based methods with >4x active parameters, using significantly less computes. To further improve the scalability and efficiency, we conduct the first comprehensive study of adaptive computation for MoE-based embedding, spanning diverse strategies across training-based and inference-only methods. Together, these results support expert scaling as an effective and efficient direction for UME, with adaptive computation further improving efficiency for MLLM-based embedding models towards large-scale retrieval and recommendation systems.

Abstract

Multimodal Embedding

Explore similar work

MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Models

Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings

Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMs