WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report

Junjie Zhou; Ke Mei; Lei Li; Tianyi Wang; Fengyun Rao; Jing Lyu

cs.CVAug 25, 2026

WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report

Authors: Junjie Zhou, Ke Mei, Lei Li, Tianyi Wang, Fengyun Rao, Jing Lyu

Organizations: WeChat Vision, Tencent Inc.

Abstract

Universal multimodal embeddings are becoming a core component of modern AI systems, enabling heterogeneous content to be represented in a shared space for applications such as retrieval, recommendation, classification, and agentic systems. In this report, we present WeMM-Embedding, a family of universal multimodal embedding models supporting text, images, videos, visual documents, and arbitrarily interleaved multimodal inputs with flexible output dimensions. The family comprises 2B, 4B, and 9B variants and is trained in two stages: a large-scale multimodal alignment stage, followed by a refinement stage using curated data, fine-grained relevance supervision, and cross-scale knowledge transfer. Across extensive evaluations, WeMM-Embedding achieves leading performance on multiple public benchmarks. Notably, the 2B variant already surpasses the previously leading 8B open-source baseline on MMEB-v2, while the 9B variant further achieves a new state-of-the-art overall score of 80.6. WeMM-Embedding also demonstrates strong practical performance across WeChat applications, with substantial gains on a 26-task in-house benchmark and consistent improvements across 14 online A/B tests. It has been deployed at scale across recommendation and search applications, including WeChat Channels, Official Accounts, Moments, and e-commerce services. We have released the model weights and code to facilitate future research at https://github.com/Tencent/WeMM-Embedding.

Figures & tables

Figure 1: Performance and efficiency overview of WeMM-Embedding. Left: MMEB-v2 overall performance across different model sizes, compared with representative baselines. Right: Aggregate performance on the MMEB-v2 image and video subsets, a 12-dataset cross-modal retrieval suite, and the 26-task in-house benchmark.

Figure 2: Overview of our multimodal training data. Major data families and representative coverage across diverse task settings and content domains.

Model	Size	AVG	Image					Video					VisDoc
Model	Size	AVG	Overall	CLS	QA	Ret	GD	Overall	CLS	QA	V-Ret	M-Ret	Overall
# datasets $\rightarrow$		78	36	10	10	12	4	18	5	5	5	3	24
VLM2Vec [ 19 ]	2B	47.8	59.7	58.7	49.3	65.0	72.9	29.0	33.4	30.5	20.6	33.0	44.0
GME [ 50 ]	2B	55.4	51.9	54.4	29.9	66.9	55.5	33.9	34.9	42.0	25.6	32.4	76.8
VLM2Vec-V2 [ 28 ]	2B	59.3	64.9	62.9	56.3	69.5	77.3	34.9	39.3	34.3	28.8	38.5	69.2
Ops-MM-embedding-v1	2B	64.6	69.0	68.1	65.1	69.2	80.9	47.6	53.6	55.7	41.8	33.7	70.8

Table 1: Benchmarking results on MMEB-v2. Baseline results are taken from the official MMEB leaderboard . † Closed-source leaderboard submission without publicly released model weights or a public inference endpoint. ⋆ Proprietary commercial model with an undisclosed parameter count. CLS: classification; QA: question answering; Ret: retrieval; GD: visual grounding; V-Ret: video retrieval; M-Ret: moment retrieval.

Model	Size	V3-All	Text						Agent				MCMR	Audio
Model	Size	V3-All	Overall	RR	IF	LC	MC	GR	Overall	Tool	GUI	Memory	MCMR	Audio
# tasks $\rightarrow$		190	53	20	9	6	5	13	47	35	8	4	1	11
VLM2Vec-V2 [ 28 ]	2B	38.3	24.5	7.8	29.2	11.5	50.3	41.2	28.7	27.6	36.2	23.3	4.1	0.0
Omni-Embed-Nemotron [ 45 ]	3B	43.5	39.2	17.2	42.8	40.0	69.7	56.9	36.5	38.1	32.0	32.2	26.1	36.5
E5-Omni [ 5 ]	3B	44.6	26.7	11.5	44.4	8.3	60.7	34.4	36.9	37.7	35.6	32.3	31.9	30.8
Qwen3-VL-Embedding [ 22 ]	2B	50.9	39.2	16.6	40.6	53.1	61.2	58.0	39.3	42.6	30.4	28.4	42.0	0.0

Table 2: Evaluation results on MMEB-v3. V3-All averages all 190 tasks, comprising the 78 MMEB-v2 tasks, 53 Text tasks, 47 Agent tasks, 11 Audio tasks, and MCMR [ 10 ] . Following the evaluation protocol defined in the MMEB-v3 paper [ 15 ] , Text results are reported using NDCG@5. Unsupported tasks are assigned a score of zero. RR: reasoning retrieval; IF: instruction following; LC: long-context retrieval; MC: multi-condition retrieval; GR: general retrieval.

		Gemini Embedding 2 †	Amazon Nova MME †	Voyage Multimodal 3.5 †	Qwen3-VL- Embedding	WeMM- Embedding	WeMM- Embedding	WeMM- Embedding
Size		–	–	–	8B	2B	4B	9B
Text $\rightarrow$ Image Recall@1	MSCOCO [ 7 ]	62.9	57.2	58.1	58.7	62.6	63.5	64.6
	Flickr30k [ 31 ]	89.1	81.6	89.9	84.6	86.9	87.0	87.8
	DOCCI [ 29 ]	93.4	84.0	83.8	95.0	94.0	95.8	96.7
	TextCaps [ 36 ]	89.6	76.0	79.4	79.3	82.5	85.8	85.2
	Mean	83.8	74.7	77.8	79.4	81.5	83.0	83.6

Table 3: Cross-modal retrieval results on 12 public benchmarks. AVG denotes the average across the 12 datasets. † Proprietary commercial model with an undisclosed parameter count.

Model	Size	AVG	Classification	Search	Cross-DM	Article Rel.	Video Rel.
# tasks $\rightarrow$		26	4	7	4	4	7
Qwen3-VL-Embedding [ 22 ]	2B	60.9	60.1	51.9	55.5	75.5	64.9
WeMM-Embedding	2B	72.0	72.6	65.5	73.7	86.4	68.8

Table 4: Evaluation results on the in-house benchmark. AVG denotes the average over all 26 tasks. Cross-DM denotes cross-domain content matching; Article Rel. denotes article relevance; Video Rel. denotes video relevance.

Figure 3: MRL analysis of WeMM-Embedding-2B on MMEB-v2. Left: Performance retained on the image, video, and visual-document subsets across embedding dimensions. Right: Performance retained for classification (CLS), question answering (QA), and retrieval (RET), each averaged over the corresponding image and video tasks.

Stage-1 Configuration	AVG	Image	Video	VisDoc
Full configuration	71.9	76.1	59.1	75.3
w/o task-specific instructions	71.1	75.7	58.2	73.8
w/o task-consistent batching	68.5	71.5	56.3	73.1
w/o duplicate-aware masking	71.4	75.3	58.6	75.1

Table 5: Results on MMEB-v2 from a small-scale Stage-1 ablation study using a 2B model. AVG denotes the average across all 78 MMEB-v2 tasks.

Cumulative Configuration	AVG	Image	Video	VisDoc
Stage-1 checkpoint	75.7	77.7	67.5	78.9
+ curated data	76.6	78.9	69.2	78.5
+ reranker supervision	76.7	78.8	69.4	79.1
+ embedding-teacher distillation	77.6	79.4	70.0	80.4
+ expanded visual input budget (final)	77.9	79.6	70.8	80.7

Table 6: Cumulative Stage-2 results of the 2B model on MMEB-v2. Each row reports the configuration after introducing the listed strategy. AVG denotes the average across all 78 MMEB-v2 tasks.

Multimodal Embedding

Multimodal embedding maps text, images, audio, video, and other data into representations that support comparison and retrieval across modalities.

See all 160 papers

Oct 27, 2026

EmbeddingGemma 2: Unified and Efficient Multimodal Representations

We present EmbeddingGemma 2, an open multimodal embedding model which natively represents text, images, audio, video, and combinations thereof in a unified space. Our approach introduces improvements such as simplified per-layer embeddings and a multi-stage curriculum with modality and objective progression which ensure strong quality across unimodal, cross-modal, and mixed-modal evaluations. Notably, in comprehensive embedding benchmarks across all modalities, EmbeddingGemma 2 establishes a new Pareto frontier, outperforming both specialist models and models two to twelve times larger. Furthermore, our design offers flexibility for resource-constrained environments, supporting selective modality encoder loading, adjustable vision token budgets, truncatable embedding dimensions, and quantized weights. We present ablations analyzing key decisions and release EmbeddingGemma 2 to promote further research.

Henrique Schechter Vera, Sahil Dua, Jovan Andonov +70

Multimodal Embedding Unified Multimodal Models Multimodal Pretraining Efficient Multimodal Inference

Ask PDF

May 26, 2026

Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini

We introduce Gemini Embedding 2, a native multimodal embedding model that allows embedding video, audio, image, and text modalities in a unified representation space. We leverage the multimodal capabilities of Gemini to produce embeddings for arbitrary combinations of interleaved inputs across all these modalities that generalize well across a wide variety of tasks. Applying large-scale contrastive learning in a multi-task multi-stage training setup, we achieve state-of-the-art performance on key embedding benchmarks including unimodal, cross-modal, and multimodal retrieval spanning a diverse set of tasks. We show that our embedding model demonstrates strong performance (with a score of 62.9 R@1 on MSCOCO, 68.8 NDCG@10 on Vatex, 69.9 on MTEB multilingual and 84.0 on MTEB Code) across a variety of tasks surpassing the performance of specialized models. These unified capabilities make Gemini Embedding 2 a promising candidate for downstream use cases such as RAG, recommendation and search. Furthermore, its robust zero-shot performance across distinct fields - from astronomy and bioscience to fine arts and the culinary arts - establishes it as a highly reliable, out-of-the-box representation even for specialized domains.

Madhuri Shanbhogue, Zhe Li, Shanfeng Zhang +86

Cross-Modal Learning Unified Multimodal Models Multimodal Embedding Cross-Modal Retrieval Multimodal Foundation Models Multimodal Understanding

Ask PDF

Sep 21, 2026

Ovis-Embedding: Pushing the Frontiers of Universal Omni-Modal Embeddings

In this report, we introduce Ovis-Embedding, a state-of-the-art omni-modal embedding family built on native integration of text, image, video, and audio. Instead of assembling separate modality towers, Ovis-Embedding uses a shared multimodal backbone to encode different modalities in a common representation space. Specifically, we make three key advances: (1) native omni-modal initialization: we adopt a pretrained Qwen-omni model as the embedding backbone and adapt it through contrastive training with low-rank initialization; (2) data-centric omni-modal training: we construct a broad, high-quality corpus spanning text, images, video, audio, and interleaved multimodal data. To improve data efficiency, we introduce homogeneous-source sampling to form task-consistent batches with informative in-batch negatives; and (3) embedding-specific training and inference optimization: we use focal loss to emphasize hard examples and similarity-based Embedding Distillation to transfer fine-grained similarity structure from complementary experts. At inference time, low-rank feature decomposition enables compact embeddings with flexible dimensionality and minimal performance loss. Empirical evaluations show that the Ovis-Embedding family achieves state-of-the-art performance on MMEB-v3, MMEB-v2, MVEB, MAEB, and RTEB, demonstrating its effectiveness across text, image, video, and audio modalities. These results highlight the potential of unified omni-modal training to overcome modality fragmentation and advance universal embedding models for any-to-any retrieval.

Ovis-Embedding Team

Alibaba Token Hub, Alibaba Group

Unified Multimodal Models Multimodal Embedding Cross-Modal Retrieval Multimodal Contrastive Learning

Ask PDF

Abstract

Multimodal Embedding

Explore similar work

EmbeddingGemma 2: Unified and Efficient Multimodal Representations

Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini

Ovis-Embedding: Pushing the Frontiers of Universal Omni-Modal Embeddings