cs.CVOct 10, 2026

What Really Works for Universal Multimodal Embeddings? A Controlled Study of Contrastive Training Strategies

Authors: Duo Li, Haojun Pan, Zuhao Yang, Xintong Wang, Chengfu Huo, Shijian Lu, Liang Ding

Organizations: CCDS, Nanyang Technological University · Alibaba Group · Universität Hamburg · Sichuan University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Contrastive learning has become the standard paradigm for multimodal representation learning, aligning semantically matched pairs while separating mismatched ones in a shared embedding space. Although various contrastive training strategies have been proposed across negative selection, batch construction, and gradient modulation, their actual effectiveness under controlled, reproducible conditions remains poorly understood. This motivates a fundamental question: Do sophisticated contrastive strategies consistently bring meaningful downstream gains in multimodal embedding learning? To address this, we systematically evaluate representative strategies under a unified benchmarking framework on MMEB. Surprisingly, direct evaluations reveal that most existing strategies offer marginal gains over a standard aligned baseline, with several even degrading overall performance. To uncover the underlying failure modes, we introduce an MLLM-as-a-Judge diagnostic framework that dissects top-1 retrieval behaviors into labeled positives, false negatives (unlabeled valid matches), and hard/easy negative errors. Our systematic analysis yields three key insights: (1) Gradient distortion: Upweighting gradients from high-similarity negatives consistently hurts downstream generalization; (2) Distribution illusion: Wider separation between positive and negative similarity distributions does not necessarily translate to better retrieval performance; and (3) Synergistic training: While false-negative mitigation or hard-batch construction yields limited gains in isolation, integrating hard-negative mining, false-negative mitigation, and hard-batch construction achieves optimal performance, delivering an average improvement of 6.0% on MMEB with Qwen2-VL-2B. Overall, our findings demystify popular contrastive strategies and provide actionable recipes for training universal multimodal embedding models.

Figures & tables

 DependenciesOptimization Dimensions
MethodExternal ScorerPre- processingGradient ModulationFalse-Negative MitigationBatch ConstructionHard-Negative Mining
LLaVE––Adaptive–––
EGA––Adaptive–––
GloFND–––Adaptive––
FN Risk–✓–Adaptive––
FALCON–✓––Adaptive–
Table 1: Overview of representative contrastive training strategies. Dependencies indicate whether a method requires external scoring or additional preprocessing. Entries under Optimization Dimensions indicate what aspects each contrastive learning method targets explicitly. Static denotes that the corresponding optimization signal is fixed before training, whereas Adaptive denotes that it evolves with the student model.
Training ConfigurationMMEB
LRMax. Visual TokensLoRA RankPooling StrategiesCls.VQARet.VGOverall
2e-52,8808Last Token58.749.365.072.959.8
1e-42,8808Last Token60.056.567.376.063.2
1e-41,2808Last Token59.756.567.376.163.2
1e-42,8804Last Token60.755.465.276.562.5
1e-42,88032Last Token61.957.769.276.564.8
Table 2: Training-setting alignment. We align the learning rate (LR), maximum number of visual tokens, LoRA rank, and pooling strategy for the VLM2Vec baseline on MMEB using Qwen2-VL-2B-Instruct. Cls. , VQA , Ret. , and VG denote image classification, visual question answering, image-level retrieval, and visual grounding, respectively. Overall denotes the mean Precision@1 across all 36 MMEB evaluation datasets. The configuration adopted for all subsequent experiments is highlighted in bold .
MethodTraining Time (h:mm)Per Meta-Task ScoreAverage Score
Image ClassificationVisual Question AnsweringImage-level RetrievalVisual GroundingINDOODOverall
No. of Datasets →\rightarrow1010124201636
VLM2Vec (Baseline)19:1263.357.668.677.969.859.265.1
Baseline + GM
LLaVE19:2457.858.468.978.069.357.263.9
EGA19:2663.357.468.177.369.958.564.8
Table 3: Comparison of multimodal contrastive training strategies on MMEB using Qwen2-VL-2B. The evaluation metric is mean Precision@1 (%) across the corresponding datasets. IND and OOD denote the averages over in-distribution and out-of-distribution datasets, respectively. GM, FN, HN, and BC denote gradient modulation, false-negative mitigation, hard-negative mining, and batch construction, respectively. Training time denotes the training wall-clock time; one-time preprocessing costs are excluded. The best results are shown in bold .
Figure 1: Overview of the MLLM-as-a-Judge analysis framework. Given the same query and labeled positive, we collect top-1 predictions produced by different contrastive training strategies. The MLLM judge categorizes the prediction as a labeled positive, an alternative valid positive (i.e., false negative), a hard-negative error, or an easy-negative error. This decomposition enables a more fine-grained analysis than standard exact-match Precision@1.
MethodPositive Outcomes (%)Negative Outcomes (%)Uncertain (%)
Labeled Positive ↑\uparrowFalse Negative ↑\uparrowSemantic Valid ↑\uparrowHard Negative ↓\downarrowEasy Negative ↓\downarrowSemantic Error ↓\downarrow
VLM2Vec67.148.2275.3617.196.8524.040.60
Baseline + GM
LLaVE65.688.5374.2117.587.6725.250.54
EGA66.688.3174.9917.247.0424.280.74
Baseline + FN
Table 4: MLLM-as-a-Judge analysis of top-1 predictions on MMEB. All methods are evaluated on the same 7,200 cases, with 200 cases sampled from each of the 36 datasets. Labeled Positive matches the annotated target; False Negative is a different but semantically valid answer; and Semantic Valid is their sum. Hard Negative denotes a related but incorrect prediction, while Easy Negative denotes a clearly irrelevant prediction; Semantic Error is their sum. GM, FN, HN, and BC denote gradient modulation, false-negative mitigation, hard-negative mining, and batch construction, respectively. Arrows indicate the preferred direction, and the best results are shown in bold .
Figure 2: Similarity distributions of nine training strategies on MMEB. All methods are evaluated on the same 7,200 queries, comprising 200 randomly sampled cases from each of the 36 datasets. Positive (P) denotes the similarity between a query and its designated target, while Hard Negative (HN) and Easy Negative (EN) denote the mean similarities to its Top-5 and Bottom-5 non-positive candidates, respectively. OVL(P,HN) and OVL(P,EN) measure the distribution overlap between the Positive distribution and the HN and EN distributions, respectively; lower values indicate better distributional separation.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

MethodOptimization DimensionsPer Meta-Task ScoreOverall Score
Image ClassificationVisual Question AnsweringImage-level RetrievalVisual Grounding
No. of Datasets →\rightarrow101012436
Qwen3.5-2B
VLM2Vec (Baseline)–65.267.467.384.668.7
EGAGM65.467.867.482.568.6
FN RiskFN65.366.067.084.368.2
Appendix
Table B.1: Comparison of multimodal contrastive training strategies on MMEB using Qwen3.5-2B and Qwen2-VL-7B. We extend the unified comparison in Table 3 under the same training and evaluation protocol. Scores are mean Precision@1 (%) over the corresponding datasets, and Overall is the average across all 36 MMEB datasets. GM, FN, HN, and BC denote gradient modulation, false-negative mitigation, hard-negative mining, and batch construction, respectively. The best result in each column under each backbone is highlighted in bold .
 Positive Outcomes (%)Negative Outcomes (%) 
Judge / SetLabeled Positive ↑\uparrowFalse Negative ↑\uparrowSemantic Valid ↑\uparrowHard Negative ↓\downarrowEasy Negative ↓\downarrowSemantic Error ↓\downarrowUncertain (%)
Qwen3.8-Flash Full set67.358.0875.4317.166.7523.910.66
Qwen3.8-Flash Subset67.148.2275.3617.196.8524.040.60
GPT-5.4 Subset67.148.0775.2116.936.7123.641.15
Appendix
Table B.2: Robustness of MLLM-as-Judge analysis. We evaluate predictions from the same VLM2Vec-2B checkpoint using different MLLM judges and evaluation sets. Full set includes all available evaluation cases from the 36 MMEB datasets, while the Subset contains the 7,200 cases used in the main analysis, with 200 cases randomly sampled from each dataset. Outcome definitions follow Table 4 . All values are percentages.
MethodOptimization DimensionsPer Meta-Task ScoreOverall Score
Image ClassificationVisual Question AnsweringImage-level RetrievalVisual Grounding
No. of Datasets →\rightarrow101012436
Qwen2-VL-2B
B3 (Full)FN + HN + BC67.363.071.481.469.0
Random BatchFN + HN67.360.771.078.167.9
w/o FN FilteringHN + BC67.361.770.780.768.4
Appendix
Table B.3: Ablation study of B3 using Qwen2-VL-2B on MMEB. We remove false-negative mitigation (FN), hard-negative mining (HN), or batch construction (BC) from the full B3 framework while keeping the remaining training and evaluation settings unchanged. Scores are mean Precision@1 (%) over the corresponding datasets, and Overall is the average across all 36 MMEB datasets. The best result in each column is highlighted in bold .
Figure C.1: Overview of the MLLM-as-Judge prompt. The judge receives the query, the dataset-designated positive, the model’s top-1 prediction, and an explicit mapping between images and their roles. It first verifies the designated positive and then categorizes the top-1 prediction as the designated positive, a false negative, a hard negative, an easy negative, or an uncertain case. Task-specific criteria guide judgments across the five MMEB task families, and a constrained JSON schema enables automatic parsing and aggregation.
More in this topic

Multimodal Embedding

Multimodal embedding maps text, images, audio, video, and other data into representations that support comparison and retrieval across modalities.

See all 156 papers

Explore similar work

CardsList
Aug 19, 2026

UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrieval

Universal multimodal retrieval aims to support diverse instruction-aware retrieval tasks, demanding both efficient corpus-scale matching and fine-grained semantic reasoning. Recent MLLM-based embedding methods typically derive representations from hidden states, while Chain-of-Thought (CoT) reasoning is emerging as a promising strategy for embedding enhancement by encoding intermediate semantic evidence into the representation space. However, existing CoT methods typically use item-wise reasoning over queries and candidates in isolation, providing no explicit evidence to distinguish a positive from a semantically confusable hard negative. Moreover, contrastive embeddings capture global similarity but struggle with meta-tasks requiring answer verification, category judgment or fine-grained reasoning. In this paper, we propose UMER, a Unified Multimodal Embedding and Ranking framework for universal multimodal retrieval. UMER replaces item-wise reflection with Pair-Aware Discriminative Reasoning, which compares query--candidate pairs to identify instruction-relevant matching and discrepancy evidence. UMER jointly learns contrastive embeddings for efficient global matching and discriminative ranking for explicit pairwise relevance judgment within a single MLLM. A complementary mutual distillation strategy further transfers reliable pairwise preferences between the embedding and ranking functions. On the MMEB-V2 benchmark, UMER achieves state-of-the-art performance under comparable experimental settings while supporting budget-adjustable inference.

Libiao Chen, Xiyang Liu, Yanheng Wei +2

Beihang University, Beijing, China

Multimodal EmbeddingCross-Modal RetrievalMultimodal RerankingMultimodal ReasoningMultimodal Contrastive Learning
AskPDF
Jun 20, 2026

MultiMem: Measuring and Mitigating Memorization in Multi-Modal Contrastive Learning

Memorization in machine learning models enables high performance on rare in-distribution samples by capturing their atypical patterns. However, it also causes harmful retention of noise and outliers, degrading generalization. While memorization has been extensively studied in both supervised and self-supervised learning in the vision domain, it remains unexplored in multi-modal contrastive learning. We address this gap by introducing MultiMem, the first metric designed to quantify memorization in multi-modal contrastive learning. Through our systematic analysis, we demonstrate that cross-modal semantic misalignment has the strongest influence on memorization, with text being the dominant modality driving memorization, followed by video, image, and audio. We show that targeted augmentations applied across all modalities effectively reduce memorization as measured by our MultiMem metric and improve model performance. Overall, this work establishes the first framework for measuring and mitigating memorization in multi-modal contrastive learning, preventing harmful data retention and contributing to higher-performing models.

Wenhao Wang, Franziska Boenisch, Michael Backes +1

CISPA Helmholtz Center for Information Security

Multimodal Contrastive Learning
AskPDF
Mar 2, 2026

Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learning

Multimodal embedding models, rooted in multimodal large language models (MLLMs), have yielded significant performance improvements across diverse tasks such as retrieval and classification. However, most existing approaches rely heavily on large-scale contrastive learning, with limited exploration of how the architectural and training paradigms of MLLMs affect embedding quality. While effective for generation, the causal attention and next-token prediction paradigm of MLLMs does not explicitly encourage the formation of globally compact representations, limiting their effectiveness as multimodal embedding backbones. To address this, we propose CoCoA, a Content reconstruction pre-training paradigm based on Collaborative Attention for multimodal embedding optimization. Specifically, we restructure the attention flow and introduce an EOS-based reconstruction task, encouraging the model to reconstruct input from the corresponding <EOS> embeddings. This drives the multimodal model to compress the semantic information of the input into the <EOS> token, laying the foundations for subsequent contrastive learning. Extensive experiments on MMEB-V1 demonstrate that CoCoA built upon Qwen2-VL and Qwen2.5-VL significantly improves embedding quality. Results validate that content reconstruction serves as an effective strategy to maximize the value of existing data, enabling multimodal embedding models generate compact and informative representations, raising their performance ceiling.

Jiahan Chen, Da Li, Hengran Zhang +6

State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China · Baidu Inc., Beijing, China

Multimodal EmbeddingMultimodal LearningMultimodal PretrainingMultimodal Large Language Models
AskPDF