Organizations: State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 101408, China · Beijing Academy of Artificial Intelligence, Beijing 100084, China
Multimodal large language models (MLLMs) require visual encoders that extract fine-grained visual features while constructing semantically disentangled and discriminative representations. Conventional methodologies encounter a fundamental trade-off: contrastive learning enhances inter-class discrimination by suppressing intra-class variability, whereas generative reconstruction emphasizes perceptual fidelity but lacks explicit comparative supervision. The naive aggregation of these objectives via composite loss functions often induces optimization conflicts and instability due to gradient interference within the shared latent space. To resolve this, we introduce a unified representation-learning paradigm that reformulates generative reconstruction as a discriminative ranking problem within the generative output manifold. Our approach maintains perceptual reconstruction as a supervisory signal and integrates a ranking-based discriminative constraint by systematically ordering matched and mismatched reconstruction outputs. This formulation embeds discriminative information seamlessly into the generative process, alleviating the risk of loss competition. Specifically, we propose AUC Generative Reconstruction (AUCGR), which optimizes the area under the ROC curve (AUC) by ensuring higher scores for matched pairs than for mismatched pairs, and Partial AUC Generative Reconstruction (pAUCGR), which addresses supervision bias by focusing on mismatched comparisons within reliable feature subspaces. The proposed objectives are designed to be plug-and-play, requiring no architectural modifications to existing generative reconstruction frameworks. Theoretical analysis establishes rigorous connections to AUC/pAUC optimization and provides generalization guarantees. Extensive experiments validate that our method advances both discriminative accuracy and fine-grained perceptual representation.
Figures & tables
Fig. 1: Motivation of our method. (a) Reconstructive learning improves P-Ability by preserving a sparser representation space. (b) Contrastive learning improves D-Ability by organizing representations into compact semantic clusters. (c) The naive method searches for a balanced representation through heuristic weight tuning between contrastive learning and reconstructive learning, but the proper weights are difficult to determine. (d) The target balanced representation, which is directly optimized by our method without heuristic weight tuning.
Fig. 2: Training results of linearly combining contrastive learning and reconstructive learning (Naive Method) on the OpenAI CLIP ViT-L@224 backbone, which verify that the naive method indeed falls into the Tragic Triad regime.
Fig. 3: An overview of our method. (a) A generic comparison principle for generative reconstruction. (b) AUC-oriented instantiations, including AUCGR and pAUCGR. (c) Reconstruction learning pipeline. Our method serves as a plug-and-play optimization module that replaces the original optimization strategy in existing reconstruction learning methods, thereby yielding balanced enhanced visual representations.
Fig. 4: Empirical verification of noisy negatives. (a) The training data exhibits a clear long-tail distribution. (b) We randomly sample one mini-batch with batch size 128 and compute pairwise image similarity and label-text similarity within the batch. The results show that highly similar image pairs indeed exist during training.
CLIP Backbone
Resol.
#Params
Method
Avg
OpenAI ViT-L-14
224
427.6
Original
13.3
13.3
20.0
20.0
13.3
53.3
20.0
6.7
13.3
19.2
DIVA
13.3
6.7
13.3
26.7
13.3
46.7
26.7
20.0
26.7
21.5
+DCR
6.7
20.0
33.3
0.0
26.7
66.7
20.0
6.7
20.0
22.2
+AUCGR
0.0
33.3
26.7
6.7
0.0
73.3
33.3
13.3
13.3
22.2
+pAUCGR
6.7
13.3
46.7
6.7
6.7
66.7
20.0
20.0
33.3
24.4
GenHancer
13.3
13.3
13.3
13.3
13.3
66.7
40.0
20.0
33.3
25.2
TABLE I: Performance of Fine-grained Perceptual Ability (P-Ability) on the MMVP-VLM benchmark in terms of ACC . ‘+DCR’ denotes our conference version. The visual patterns are symbolized as: : Orientation and Direction, : Presence of Specific Features, : State and Condition, : Quantity and Count, : Positional and Relational Context, : Color and Appearance, : Structural and Physical Characteristics, : Texts, : Viewpoint and Perspective.
Backbone
Method
CIFAR-10 [ 114 ]
CIFAR-100 [ 114 ]
MNIST [ 115 ]
Caltech-101 [ 58 ]
SUN397 [ 116 ]
FGVC Aircraft [ 117 ]
Country-211 [ 11 ]
Stanford Cars [ 118 ]
DTD [ 59 ]
Eurosat [ 119 ]
FER2013 [ 120 ]
Flowers-102 [ 121 ]
Food-101 [ 122 ]
GTSRB [ 123 ]
PCam [ 124 ]
Pets [ 125 ]
Rendered SST2 [ 11 ]
Resisc45 [ 126 ]
STL10 [ 127 ]
VOC2007 [ 128 ]
ImageNet-1K [ 57 ]
ImageNet-V2 [ 129 ]
ImageNet-Adv. [ 130 ]
ImageNet-Ren. [ 131 ]
ImageNet-OOD [ 130 ]
ObjectNet [ 132 ]
Avg. ACC
O-1
Original
83.57
51.32
67.95
49.63
50.03
35.70
13.19
54.94
49.41
69.30
25.62
82.14
85.01
46.69
62.92
64.90
52.22
70.71
95.74
41.03
53.92
42.39
35.77
38.84
27.85
44.24
53.66
DIVA
80.57
41.80
62.71
45.57
47.21
22.77
7.83
20.07
48.46
65.02
28.45
68.58
68.82
36.64
65.03
41.37
52.11
67.56
97.83
37.47
36.80
31.38
17.31
17.29
28.70
26.20
44.75
+DCR
75.47
42.18
58.57
42.94
49.80
30.60
12.05
41.43
42.13
53.22
28.53
74.96
77.55
35.31
61.42
51.05
50.69
68.71
95.25
39.77
45.35
35.85
32.47
14.01
26.90
41.45
47.22
+AUCGR
86.14
43.49
65.82
47.05
48.14
28.11
10.53
37.68
48.99
77.93
26.02
77.04
76.32
43.59
64.87
49.74
54.20
75.38
94.39
39.82
43.71
36.43
25.37
21.12
27.65
39.77
49.59
+pAUCGR
83.47
51.72
66.43
49.88
50.71
34.56
12.78
55.74
51.81
69.57
25.61
78.89
83.98
47.20
62.69
61.41
52.11
73.71
95.76
46.65
53.89
41.57
37.97
39.55
27.80
45.46
53.88
GenHancer
44.25
19.39
60.34
33.54
30.20
14.46
7.17
12.29
30.96
65.44
30.59
36.67
32.04
26.88
64.03
21.10
51.78
56.13
55.83
26.70
18.20
20.87
11.31
7.43
24.00
7.85
31.13
TABLE II: Performance of Discriminative Ability (D-Ability) on 26 standard zero-shot clustering benchmarks in terms of ACC . ‘+DCR’ denotes our conference version. O-1, M-1, and S-1 separately represent OpenAI CLIP ViT-L@224, MetaCLIP ViT-L@224, and SigLIP ViT-SO@224. Results in terms of ARI and NMI are provided in Tab. V and Tab. VI .
Fig. 5: Qualitative results of P-Ability on the MMVP-VLM benchmark. The predictions from the original CLIP and our improved version are indicated by red and green arrows, respectively. The improved CLIP effectively addresses the original model’s limitations in capturing fine-grained visual details. More images are provided in Fig. 12 and Fig. 13 .
Fig. 6: Qualitative results of D-Ability on the CIFAR-10 zero-shot clustering benchmark by using the t-SNE method. The first three panels use DIVA with SigLIP ViT-SO@224 (S-1), and the last three panels use GenHancer with MetaCLIP ViT-L@224 (M-1). The improved CLIP achieves better class separability.
Fig. 7: Performance on zero-shot retrieval on the OpenAI CLIP ViT-L@224. Compared with the original DIVA/GenHancer model, our method consistently improves both image-to-text and text-to-image retrieval on Flickr30k and COCO. The results on the MetaCLIP ViT-L@224 and SigLIP ViT-SO@224 are provided in Fig. 14 and Fig. 15 .
Fig. 8: Average performance on 26 standard zero-shot classification benchmarks. O-1, M-1, and S-1 separately represent OpenAI CLIP ViT-L@224, MetaCLIP ViT-L@224, and SigLIP ViT-SO@224. Detailed results are provided in Tab. VII , Tab. VIII and Tab. IX .
LLM
CLIP
Vision-Centric Benchmarks
Conventional MLLM Benchmarks
MMVP- MLLM [ 53 ]
NaturalBench [ 63 ]
CV-Bench 2D [ 62 ]
CV-Bench 3D [ 62 ]
POPE [ 133 ]
SciQA- IMG [ 134 ]
Hallusion Avg. [ 135 ]
Acc
Q-Acc
I-Acc
G-Acc
ADE20K
COCO
rand
pop
adv
Vicuna-7B
Original
24.7
76.4
53.6
56.4
17.6
49.6
60.9
58.7
87.3
86.1
84.2
66.8
27.6
DIVA
31.3
75.3
51.7
56.1
22.3
51.3
63.4
60.2
87.9
87.0
84.6
66.3
28.6
+pAUCGR
31.3
76.3
53.9
56.1
25.5
54.1
64.7
62.6
87.3
87.2
84.9
66.4
28.6
GenHancer
30.7
77.3
55.6
59.1
24.4
52.9
63.6
63.2
88.1
86.7
84.6
66.5
28.4
TABLE III: Performance of MLLMs (LLaVA-1.5 [ 1 ] ) on various benchmarks. The champion is highlighted in bold .
Fig. 9: Ablation comparison with naive linear weighting and gradient-balancing methods. We evaluate the trade-off between P-Ability and D-Ability on two generative reconstruction baselines, DIVA and GenHancer, across OpenAI CLIP ViT-L@224 (O-1), MetaCLIP ViT-L@224 (M-1), and SigLIP ViT-SO@224 (S-1). Here, D-Ability is measured by ACC on zero-shot clustering benchmarks, while +N, +P, +C, and +G denote +Naive, +PCGrad, +CAGrad, and +GradNorm, respectively. Compared with both naive linear weighting and gradient-balancing methods, AUCGR and pAUCGR achieve a better balance between the two abilities.
Fig. 10: Ablation study on different generative model structures. We evaluate our method with Stable Diffusion v1.5, Stable Diffusion v2.1, and Rectified Flow as the generative reconstruction backbone. All experiments use OpenAI CLIP ViT-L@224 as the visual backbone. The results show that our framework is not tied to a specific generative architecture and consistently improves both P-Ability and D-Ability .
Fig. 11: Ablation study on the pAUCGR parameter. We vary the retained-negative threshold γ(α) . All experiments use OpenAI CLIP ViT-L@224 as the visual backbone.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Notation
Description
x∈RH×W×3
Input image with height H and width W .
D={xi}i=1n
Training dataset.
x~ , x~t
VAE latent of x and its noisy latent at step t .
t
Diffusion or flow timestep.
fϕ
visual encoder.
z=fϕ(x)
Visual representation extracted by fϕ .
Appendix
TABLE IV: A summary of key notations and descriptions in the paper.
Fig. 12: Expanded qualitative results of AUCGR on the MMVP-VLM benchmark. The predictions from the original CLIP and the AUCGR-enhanced CLIP are indicated by red and green arrows, respectively. AUCGR helps the model capture fine-grained visual details that are missed by the original CLIP.
Fig. 13: Expanded qualitative results of pAUCGR on the MMVP-VLM benchmark. The predictions from the original CLIP and the pAUCGR-enhanced CLIP are indicated by red and green arrows, respectively. pAUCGR further improves fine-grained visual perception by reducing misleading negative comparisons during optimization.
Dataset
OpenAI CLIP ViT-L@224 (O-1)
MetaCLIP ViT-L@224 (M-1)
SigLIP ViT-SO@224 (S-1)
Original
DIVA
+DCR
+AUCGR
+pAUCGR
GenHancer
+DCR
+AUCGR
+pAUCGR
Original
DIVA
+DCR
+AUCGR
+pAUCGR
GenHancer
+DCR
+AUCGR
+pAUCGR
Original
DIVA
+DCR
+AUCGR
+pAUCGR
GenHancer
+DCR
+AUCGR
+pAUCGR
CIFAR-10
0.786
0.675
0.616
0.759
0.784
0.252
0.538
0.573
0.565
0.791
0.630
0.774
0.754
0.801
0.399
0.751
0.789
0.793
0.878
0.404
0.740
0.801
0.937
0.936
0.938
0.939
0.939
CIFAR-100
0.384
0.286
0.274
0.303
0.382
0.088
0.209
0.214
0.226
0.396
0.309
0.340
0.330
0.431
0.182
0.366
0.364
0.403
0.551
0.206
0.537
0.465
0.538
0.548
0.549
0.537
0.534
MNIST
0.449
0.407
0.333
0.463
0.501
0.365
0.422
0.585
0.406
0.496
0.513
0.476
0.492
0.472
0.524
0.449
0.475
0.475
0.555
0.544
0.605
0.551
0.562
0.571
0.559
0.579
0.561
Caltech-101
0.302
0.299
0.250
0.283
0.306
0.215
0.282
0.276
0.258
0.302
0.298
0.312
0.309
0.298
0.235
0.294
0.305
0.309
0.268
0.207
0.287
0.268
0.289
0.281
0.276
0.294
0.289
SUN397
0.379
0.363
0.371
0.355
0.380
0.186
0.306
0.304
0.336
0.386
0.326
0.321
0.320
0.387
0.113
0.302
0.292
0.369
0.432
0.052
0.418
0.404
0.430
0.430
0.431
0.428
0.431
Appendix
TABLE V: Performance of Discriminative Ability (D-Ability) on 26 standard zero-shot clustering benchmarks in terms of ARI . ‘+DCR’ denotes our conference. This table provides supplementary results for Tab. II .
Dataset
OpenAI CLIP ViT-L@224 (O-1)
MetaCLIP ViT-L@224 (M-1)
SigLIP ViT-SO@224 (S-1)
Original
DIVA
+DCR
+AUCGR
+pAUCGR
GenHancer
+DCR
+AUCGR
+pAUCGR
Original
DIVA
+DCR
+AUCGR
+pAUCGR
GenHancer
+DCR
+AUCGR
+pAUCGR
Original
DIVA
+DCR
+AUCGR
+pAUCGR
GenHancer
+DCR
+AUCGR
+pAUCGR
CIFAR-10
0.841
0.745
0.716
0.815
0.839
0.364
0.636
0.677
0.662
0.860
0.765
0.839
0.831
0.865
0.508
0.811
0.845
0.862
0.907
0.531
0.931
0.863
0.937
0.935
0.937
0.937
0.938
CIFAR-100
0.679
0.611
0.592
0.633
0.675
0.381
0.557
0.541
0.573
0.710
0.651
0.669
0.667
0.722
0.498
0.683
0.682
0.708
0.798
0.520
0.786
0.744
0.791
0.789
0.791
0.791
0.788
MNIST
0.617
0.556
0.572
0.617
0.647
0.525
0.538
0.735
0.526
0.639
0.690
0.614
0.627
0.661
0.662
0.596
0.619
0.624
0.696
0.680
0.746
0.691
0.702
0.705
0.700
0.685
0.700
Caltech-101
0.809
0.780
0.757
0.793
0.810
0.662
0.750
0.772
0.759
0.823
0.807
0.818
0.827
0.820
0.662
0.811
0.816
0.826
0.805
0.601
0.805
0.807
0.809
0.809
0.807
0.813
0.811
SUN397
0.759
0.743
0.744
0.740
0.755
0.588
0.693
0.697
0.716
0.764
0.715
0.715
0.715
0.763
0.494
0.708
0.703
0.752
0.796
0.312
0.790
0.782
0.795
0.796
0.795
0.794
0.795
Appendix
TABLE VI: Performance of Discriminative Ability (D-Ability) on 26 standard zero-shot clustering benchmarks in terms of NMI . ‘+DCR’ denotes our conference version. This table provides supplementary results for Tab. II .
Dataset
OpenAI CLIP ViT-L@224 (O-1)
MetaCLIP ViT-L@224 (M-1)
SigLIP ViT-SO@224 (S-1)
DIVA
+AUCGR
+pAUCGR
GenHancer
+AUCGR
+pAUCGR
DIVA
+AUCGR
+pAUCGR
GenHancer
+AUCGR
+pAUCGR
DIVA
+AUCGR
+pAUCGR
GenHancer
+AUCGR
+pAUCGR
CIFAR-10
86.93
92.22
95.60
76.36
75.76
79.45
96.66
97.45
97.56
58.19
96.53
97.33
95.13
92.30
96.68
96.71
96.69
96.70
CIFAR-100
58.44
70.89
75.69
36.34
43.81
51.89
79.83
81.11
84.12
28.63
81.32
84.18
76.86
78.41
84.11
83.70
84.21
84.23
MNIST
53.63
67.28
76.65
49.02
66.28
48.99
62.90
62.21
63.13
29.85
63.99
64.71
74.84
81.77
88.12
88.33
89.11
89.13
Caltech-101
79.97
81.35
83.35
69.50
80.38
78.54
83.81
89.47
87.07
34.91
84.77
86.52
71.80
85.83
85.98
86.06
86.01
86.03
SUN397
55.47
63.43
67.80
30.30
52.73
48.99
63.63
70.62
73.36
12.27
71.79
74.07
26.38
72.55
74.43
74.42
74.45
74.42
Appendix
TABLE VII: Performance on 26 standard zero-shot classification benchmarks in terms of ACC@1 . This table presents the detailed results corresponding to Fig. 8 .
Dataset
OpenAI CLIP ViT-L@224 (O-1)
MetaCLIP ViT-L@224 (M-1)
SigLIP ViT-SO@224 (S-1)
DIVA
+AUCGR
+pAUCGR
GenHancer
+AUCGR
+pAUCGR
DIVA
+AUCGR
+pAUCGR
GenHancer
+AUCGR
+pAUCGR
DIVA
+AUCGR
+pAUCGR
GenHancer
+AUCGR
+pAUCGR
CIFAR-10
99.13
99.41
99.64
96.78
97.61
98.48
99.94
99.95
99.93
93.39
99.90
99.95
99.43
99.37
99.81
99.83
99.86
99.86
CIFAR-100
81.95
90.58
92.74
64.19
70.01
76.23
96.10
95.92
96.93
58.32
95.85
96.66
91.60
93.11
96.07
95.83
96.03
96.03
MNIST
87.91
89.14
93.52
84.49
88.97
88.08
92.54
95.52
94.38
81.94
93.63
94.67
94.14
97.16
98.81
98.99
99.00
99.00
Caltech-101
92.23
94.23
95.43
87.87
96.34
92.36
97.73
96.06
96.42
64.39
94.53
95.27
94.95
96.17
95.94
96.04
95.79
95.76
SUN397
84.77
91.52
93.70
65.04
84.52
78.28
90.78
94.66
95.85
32.82
95.02
96.06
54.22
95.19
95.81
95.88
95.83
95.83
Appendix
TABLE VIII: Performance on 26 standard zero-shot classification benchmarks in terms of ACC@5 . This table presents the detailed results corresponding to Fig. 8 . Since PCam and Rendered SST2 contain only two categories, ACC@5 cannot be computed on these two datasets.
Dataset
OpenAI CLIP ViT-L@224 (O-1)
MetaCLIP ViT-L@224 (M-1)
SigLIP ViT-SO@224 (S-1)
DIVA
+AUCGR
+pAUCGR
GenHancer
+AUCGR
+pAUCGR
DIVA
+AUCGR
+pAUCGR
GenHancer
+AUCGR
+pAUCGR
DIVA
+AUCGR
+pAUCGR
GenHancer
+AUCGR
+pAUCGR
CIFAR-10
86.95
92.22
95.62
76.35
75.76
79.44
96.67
97.45
97.57
58.09
96.58
97.33
95.19
92.31
96.68
96.71
96.69
96.69
CIFAR-100
58.45
70.88
75.72
36.35
43.79
51.88
79.82
81.09
84.10
28.64
81.32
84.20
76.86
78.42
84.12
83.71
84.22
84.26
MNIST
53.68
66.76
76.32
49.37
65.46
48.97
63.20
62.89
63.74
29.76
64.30
65.12
74.33
81.52
87.98
88.21
89.01
89.00
Caltech-101
87.27
90.89
92.49
78.14
81.66
83.82
93.05
95.18
95.29
29.23
93.85
95.18
84.14
95.40
95.99
96.11
96.03
96.03
SUN397
50.66
63.31
68.45
30.74
52.44
43.27
62.02
70.32
74.71
10.59
70.32
75.09
25.21
73.79
75.82
76.05
75.93
75.92
Appendix
TABLE IX: Performance on 26 standard zero-shot classification benchmarks in terms of mRecall . This table presents the detailed results corresponding to Fig. 8 .
Fig. 14: Performance on zero-shot retrieval on MetaCLIP ViT-L@224. Compared with the original DIVA/GenHancer model, our method consistently improves both image-to-text and text-to-image retrieval on Flickr30k and COCO.
Fig. 15: Performance on zero-shot retrieval on SigLIP ViT-SO@224. Compared with the original DIVA/GenHancer model, our method consistently improves both image-to-text and text-to-image retrieval on Flickr30k and COCO.
Fig. 16: Expanded qualitative results on the MLLM benchmarks. Ours denotes the model that replaces the original CLIP vision encoder with the pAUCGR-enhanced CLIP encoder.
Method
P-Ability
D-Ability
DIVA
21.48
44.75
Naive
22.22
47.23
Naive+Random
20.74
43.32
Naive+Partial
17.78
42.75
AUCGR
22.22
49.59
AUCGR+Random
21.48
48.88
Appendix
TABLE X: Comparison of noisy-negative filtering strategies. All experiments use OpenAI CLIP ViT-L@224. P-Ability is measured by ACC on MMVP-VLM, and D-Ability is measured by the average ACC over 26 zero-shot clustering benchmarks. Naive+Random and AUCGR+Random apply random negative removal to the corresponding objectives; Naive+Partial applies image-similarity-based filtering without AUC-oriented optimization; and pAUCGR combines the same filtering principle with AUCGR. The best result for each is highlighted in bold .
TABLE XI: Performance with different surrogate losses. We instantiate pAUCGR with the exponential, square, hinge, and logistic pairwise surrogates using OpenAI CLIP ViT-L@224. P-Ability is measured by ACC on MMVP-VLM, and D-Ability is measured by the average ACC over 26 zero-shot clustering benchmarks.
Fig. 17: Sensitivity to surrogate loss hyperparameters. We evaluate the trade-off between P-Ability and D-Ability of pAUCGR on DIVA and GenHancer using OpenAI CLIP ViT-L@224. Each point corresponds to a different margin λ for the hinge or square surrogate, or a different temperature τ for the logistic surrogate. The dashed lines indicate the original performance of DIVA or GenHancer, respectively.
Fig. 18: Training stability across five random seeds. We evaluate pAUCGR on DIVA and GenHancer using OpenAI CLIP ViT-L@224. P-Ability is measured by ACC on MMVP-VLM, and D-Ability is measured by the average ACC over 26 zero-shot clustering benchmarks. The boxplots summarize the pAUCGR results across five runs. Boxes indicate the interquartile ranges, and the horizontal lines inside them indicate the medians. Circular markers denote the corresponding original DIVA or GenHancer results. The mean P-Ability and D-Ability of pAUCGR over the five seeds both exceed the corresponding baseline values.
Backbone
Training Costs
DIVA
+AUCGR
+pAUCGR
OpenAI CLIP ViT-L@224
Time
3.48
4.97 ( ↑42.82% )
5.72 ( ↑64.37% )
GPU Memory
13.72
21.17 ( ↑54.28% )
22.17 ( ↑61.55% )
OpenAI CLIP ViT-L@336
Time
4.80
7.43 ( ↑54.79% )
8.23 ( ↑71.46% )
GPU Memory
22.27
38.82 ( ↑74.34% )
41.28 ( ↑85.38% )
MetaCLIP ViT-L@224
Time
3.60
4.54 ( ↑26.11% )
5.73 ( ↑59.17% )
GPU Memory
13.75
20.94 ( ↑52.27% )
22.17 ( ↑61.25% )
Appendix
TABLE XII: Training costs of our method on DIVA. Time is measured in hours (h), and GPU memory is measured in gigabytes (GB). The percentages in parentheses denote the relative increases over DIVA with the same backbone and cost metric.
Fig. 19: Computational costs under different batch sizes. Training time and GPU memory are measured for our method on DIVA with OpenAI CLIP ViT-L@224. The pairwise objective constructs a B×B score matrix, so its per-batch computation and intermediate storage scale as O(B2) . We use B=16 in all experiments because it provides a favorable balance between performance and resource consumption.
Unified multimodal models (UMMs) unify visual understanding and generation within a single architecture. However, conventional training relies on image-text pairs (or sequences) whose captions are typically sparse and miss fine-grained visual details, even when they use hundreds of words to describe a simple image. We introduce Reconstruction Alignment (RECA), a resource-efficient post-training method that leverages visual understanding encoder embeddings as dense "text prompts", providing rich supervision without captions. Concretely, RECA conditions a UMM on its own visual understanding embeddings and optimizes it to reconstruct the input image with a self-supervised reconstruction loss, thereby realigning understanding and generation. Despite its simplicity, RECA is broadly applicable: across autoregressive, masked-autoregressive, and diffusion-based UMMs, it consistently improves generation and editing fidelity. With only 27 GPU hours, post-training with RECA substantially improves image generation performance on GenEval (0.73 → 0.90) and DPGBench (80.93 → 88.15), while also boosting editing benchmarks (ImgEdit 3.38 → 3.75, GEdit 6.94 → 7.27). Notably, RECA surpasses much larger open-source models and applies broadly across diverse UMM architectures, establishing it as an efficient and general post-training alignment strategy for UMMs.
Ji Xie, Trevor Darrell, Luke Zettlemoyer +1
1UC Berkeley · University of Washington · 3Duke University
The limited understanding capacity of the visual encoder in Contrastive Language-Image Pre-training (CLIP) has become a key bottleneck for downstream performance. This capacity includes both Discriminative Ability (D-Ability), which reflects class separability, and Detail Perceptual Ability (P-Ability), which focuses on fine-grained visual cues. Recent solutions use diffusion models to enhance representations by conditioning image reconstruction on CLIP visual tokens. We argue that such paradigms may compromise D-Ability and therefore fail to effectively address CLIP's representation limitations. To address this, we integrate contrastive signals into diffusion-based reconstruction to pursue more comprehensive visual representations. We begin with a straightforward design that augments the diffusion process with contrastive learning on input images. However, empirical results show that the naive combination suffers from gradient conflict and yields suboptimal performance. To balance the optimization, we introduce the Diffusion Contrastive Reconstruction (DCR), which unifies the learning objective. The key idea is to inject contrastive signals derived from each reconstructed image, rather than from the original input, into the diffusion process. Our theoretical analysis shows that the DCR loss can jointly optimize D-Ability and P-Ability. Extensive experiments across various benchmarks and multi-modal large language models validate the effectiveness of our method. The code is available at https://github.com/boyuh/DCR.
Boyu Han, Qianqian Xu, Shilong Bao +4
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · School of Computer Science and Tech., University of Chinese Academy of Sciences · Beijing Academy of Artificial Intelligence +3
Discriminative and generative vision models excel in their respective domains but remain semantically misaligned, hindering progress toward unified visual learning. We introduce LEASE (LEArning from SEmantic Dictionaries), a self-supervised framework that bridges this gap using a paired generative-discriminative codebook design. LEASE operates entirely in a discrete token space produced through a one-time precomputation step, enabling efficient training without data augmentations, teacher models, or online tokenizers. LEASE integrates two complementary objectives: a masked token reconstruction loss that captures fine-grained generative detail, and a codebook contrast loss that aligns encoder features with discriminative semantics via adaptive centroid weighting. This dual supervision yields a unified latent space that supports both high-quality generation and strong representation learning. On ImageNet-1K, LEASE achieves state-of-the-art unified performance, outperforming prior VQGAN-based methods such as MAGE and Sorcen across linear probing (up to +1.7%), unconditional generation (-1.26 FID and +10.19 IS w.r.t MAGE), few-shot learning (+0.56% on average against Sorcen), transfer (+0.75% average improvement against MAGE and Sorcen), and robustness benchmarks (+5.86% and +4.25% average improvement against MAGE and Sorcen, respectively). It also competes favorably with domain-specialized contrastive and generative models while surpassing previous MIM methods. The unsupervised LEASE model can also be extended to conditional generation by building upon its learned representations, proving competitive with specialized baselines. Overall, LEASE provides an efficient and effective step toward general-purpose vision models that jointly understand and generate visual content.
Imanol G. Estepa, Jesús M Rodríguez-de-Vera, Bhalaji Nagarajan +1
1Universitat de Barcelona, Barcelona, Spain · 2Barcelona Supercomputing Center (BSC), Barcelona, Spain