Contrastive learning has become the standard paradigm for multimodal representation learning, aligning semantically matched pairs while separating mismatched ones in a shared embedding space. Although various contrastive training strategies have been proposed across negative selection, batch construction, and gradient modulation, their actual effectiveness under controlled, reproducible conditions remains poorly understood. This motivates a fundamental question: Do sophisticated contrastive strategies consistently bring meaningful downstream gains in multimodal embedding learning? To address this, we systematically evaluate representative strategies under a unified benchmarking framework on MMEB. Surprisingly, direct evaluations reveal that most existing strategies offer marginal gains over a standard aligned baseline, with several even degrading overall performance. To uncover the underlying failure modes, we introduce an MLLM-as-a-Judge diagnostic framework that dissects top-1 retrieval behaviors into labeled positives, false negatives (unlabeled valid matches), and hard/easy negative errors. Our systematic analysis yields three key insights: (1) Gradient distortion: Upweighting gradients from high-similarity negatives consistently hurts downstream generalization; (2) Distribution illusion: Wider separation between positive and negative similarity distributions does not necessarily translate to better retrieval performance; and (3) Synergistic training: While false-negative mitigation or hard-batch construction yields limited gains in isolation, integrating hard-negative mining, false-negative mitigation, and hard-batch construction achieves optimal performance, delivering an average improvement of 6.0% on MMEB with Qwen2-VL-2B. Overall, our findings demystify popular contrastive strategies and provide actionable recipes for training universal multimodal embedding models.
Figures & tables
Dependencies
Optimization Dimensions
Method
External Scorer
Pre- processing
Gradient Modulation
False-Negative Mitigation
Batch Construction
Hard-Negative Mining
LLaVE
–
–
Adaptive
–
–
–
EGA
–
–
Adaptive
–
–
–
GloFND
–
–
–
Adaptive
–
–
FN Risk
–
✓
–
Adaptive
–
–
FALCON
–
✓
–
–
Adaptive
–
Table 1: Overview of representative contrastive training strategies. Dependencies indicate whether a method requires external scoring or additional preprocessing. Entries under Optimization Dimensions indicate what aspects each contrastive learning method targets explicitly. Static denotes that the corresponding optimization signal is fixed before training, whereas Adaptive denotes that it evolves with the student model.
Training Configuration
MMEB
LR
Max. Visual Tokens
LoRA Rank
Pooling Strategies
Cls.
VQA
Ret.
VG
Overall
2e-5
2,880
8
Last Token
58.7
49.3
65.0
72.9
59.8
1e-4
2,880
8
Last Token
60.0
56.5
67.3
76.0
63.2
1e-4
1,280
8
Last Token
59.7
56.5
67.3
76.1
63.2
1e-4
2,880
4
Last Token
60.7
55.4
65.2
76.5
62.5
1e-4
2,880
32
Last Token
61.9
57.7
69.2
76.5
64.8
Table 2: Training-setting alignment. We align the learning rate (LR), maximum number of visual tokens, LoRA rank, and pooling strategy for the VLM2Vec baseline on MMEB using Qwen2-VL-2B-Instruct. Cls. , VQA , Ret. , and VG denote image classification, visual question answering, image-level retrieval, and visual grounding, respectively. Overall denotes the mean Precision@1 across all 36 MMEB evaluation datasets. The configuration adopted for all subsequent experiments is highlighted in bold .
Method
Training Time (h:mm)
Per Meta-Task Score
Average Score
Image Classification
Visual Question Answering
Image-level Retrieval
Visual Grounding
IND
OOD
Overall
No. of Datasets →
10
10
12
4
20
16
36
VLM2Vec (Baseline)
19:12
63.3
57.6
68.6
77.9
69.8
59.2
65.1
Baseline + GM
LLaVE
19:24
57.8
58.4
68.9
78.0
69.3
57.2
63.9
EGA
19:26
63.3
57.4
68.1
77.3
69.9
58.5
64.8
Table 3: Comparison of multimodal contrastive training strategies on MMEB using Qwen2-VL-2B. The evaluation metric is mean Precision@1 (%) across the corresponding datasets. IND and OOD denote the averages over in-distribution and out-of-distribution datasets, respectively. GM, FN, HN, and BC denote gradient modulation, false-negative mitigation, hard-negative mining, and batch construction, respectively. Training time denotes the training wall-clock time; one-time preprocessing costs are excluded. The best results are shown in bold .
Figure 1: Overview of the MLLM-as-a-Judge analysis framework. Given the same query and labeled positive, we collect top-1 predictions produced by different contrastive training strategies. The MLLM judge categorizes the prediction as a labeled positive, an alternative valid positive (i.e., false negative), a hard-negative error, or an easy-negative error. This decomposition enables a more fine-grained analysis than standard exact-match Precision@1.
Method
Positive Outcomes (%)
Negative Outcomes (%)
Uncertain (%)
Labeled Positive ↑
False Negative ↑
Semantic Valid ↑
Hard Negative ↓
Easy Negative ↓
Semantic Error ↓
VLM2Vec
67.14
8.22
75.36
17.19
6.85
24.04
0.60
Baseline + GM
LLaVE
65.68
8.53
74.21
17.58
7.67
25.25
0.54
EGA
66.68
8.31
74.99
17.24
7.04
24.28
0.74
Baseline + FN
Table 4: MLLM-as-a-Judge analysis of top-1 predictions on MMEB. All methods are evaluated on the same 7,200 cases, with 200 cases sampled from each of the 36 datasets. Labeled Positive matches the annotated target; False Negative is a different but semantically valid answer; and Semantic Valid is their sum. Hard Negative denotes a related but incorrect prediction, while Easy Negative denotes a clearly irrelevant prediction; Semantic Error is their sum. GM, FN, HN, and BC denote gradient modulation, false-negative mitigation, hard-negative mining, and batch construction, respectively. Arrows indicate the preferred direction, and the best results are shown in bold .
Figure 2: Similarity distributions of nine training strategies on MMEB. All methods are evaluated on the same 7,200 queries, comprising 200 randomly sampled cases from each of the 36 datasets. Positive (P) denotes the similarity between a query and its designated target, while Hard Negative (HN) and Easy Negative (EN) denote the mean similarities to its Top-5 and Bottom-5 non-positive candidates, respectively. OVL(P,HN) and OVL(P,EN) measure the distribution overlap between the Positive distribution and the HN and EN distributions, respectively; lower values indicate better distributional separation.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Optimization Dimensions
Per Meta-Task Score
Overall Score
Image Classification
Visual Question Answering
Image-level Retrieval
Visual Grounding
No. of Datasets →
10
10
12
4
36
Qwen3.5-2B
VLM2Vec (Baseline)
–
65.2
67.4
67.3
84.6
68.7
EGA
GM
65.4
67.8
67.4
82.5
68.6
FN Risk
FN
65.3
66.0
67.0
84.3
68.2
Appendix
Table B.1: Comparison of multimodal contrastive training strategies on MMEB using Qwen3.5-2B and Qwen2-VL-7B. We extend the unified comparison in Table 3 under the same training and evaluation protocol. Scores are mean Precision@1 (%) over the corresponding datasets, and Overall is the average across all 36 MMEB datasets. GM, FN, HN, and BC denote gradient modulation, false-negative mitigation, hard-negative mining, and batch construction, respectively. The best result in each column under each backbone is highlighted in bold .
Positive Outcomes (%)
Negative Outcomes (%)
Judge / Set
Labeled Positive ↑
False Negative ↑
Semantic Valid ↑
Hard Negative ↓
Easy Negative ↓
Semantic Error ↓
Uncertain (%)
Qwen3.8-Flash Full set
67.35
8.08
75.43
17.16
6.75
23.91
0.66
Qwen3.8-Flash Subset
67.14
8.22
75.36
17.19
6.85
24.04
0.60
GPT-5.4 Subset
67.14
8.07
75.21
16.93
6.71
23.64
1.15
Appendix
Table B.2: Robustness of MLLM-as-Judge analysis. We evaluate predictions from the same VLM2Vec-2B checkpoint using different MLLM judges and evaluation sets. Full set includes all available evaluation cases from the 36 MMEB datasets, while the Subset contains the 7,200 cases used in the main analysis, with 200 cases randomly sampled from each dataset. Outcome definitions follow Table 4 . All values are percentages.
Method
Optimization Dimensions
Per Meta-Task Score
Overall Score
Image Classification
Visual Question Answering
Image-level Retrieval
Visual Grounding
No. of Datasets →
10
10
12
4
36
Qwen2-VL-2B
B3 (Full)
FN + HN + BC
67.3
63.0
71.4
81.4
69.0
Random Batch
FN + HN
67.3
60.7
71.0
78.1
67.9
w/o FN Filtering
HN + BC
67.3
61.7
70.7
80.7
68.4
Appendix
Table B.3: Ablation study of B3 using Qwen2-VL-2B on MMEB. We remove false-negative mitigation (FN), hard-negative mining (HN), or batch construction (BC) from the full B3 framework while keeping the remaining training and evaluation settings unchanged. Scores are mean Precision@1 (%) over the corresponding datasets, and Overall is the average across all 36 MMEB datasets. The best result in each column is highlighted in bold .
Figure C.1: Overview of the MLLM-as-Judge prompt. The judge receives the query, the dataset-designated positive, the model’s top-1 prediction, and an explicit mapping between images and their roles. It first verifies the designated positive and then categorizes the top-1 prediction as the designated positive, a false negative, a hard negative, an easy negative, or an uncertain case. Task-specific criteria guide judgments across the five MMEB task families, and a constrained JSON schema enables automatic parsing and aggregation.
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China · Baidu Inc., Beijing, China