When AUC Meets Generative Reconstruction: A Unified Principle for Balanced Visual Representation Learning
Organizations: State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 101408, China · Beijing Academy of Artificial Intelligence, Beijing 100084, China
Abstract
Multimodal large language models (MLLMs) require visual encoders that extract fine-grained visual features while constructing semantically disentangled and discriminative representations. Conventional methodologies encounter a fundamental trade-off: contrastive learning enhances inter-class discrimination by suppressing intra-class variability, whereas generative reconstruction emphasizes perceptual fidelity but lacks explicit comparative supervision. The naive aggregation of these objectives via composite loss functions often induces optimization conflicts and instability due to gradient interference within the shared latent space. To resolve this, we introduce a unified representation-learning paradigm that reformulates generative reconstruction as a discriminative ranking problem within the generative output manifold. Our approach maintains perceptual reconstruction as a supervisory signal and integrates a ranking-based discriminative constraint by systematically ordering matched and mismatched reconstruction outputs. This formulation embeds discriminative information seamlessly into the generative process, alleviating the risk of loss competition. Specifically, we propose AUC Generative Reconstruction (AUCGR), which optimizes the area under the ROC curve (AUC) by ensuring higher scores for matched pairs than for mismatched pairs, and Partial AUC Generative Reconstruction (pAUCGR), which addresses supervision bias by focusing on mismatched comparisons within reliable feature subspaces. The proposed objectives are designed to be plug-and-play, requiring no architectural modifications to existing generative reconstruction frameworks. Theoretical analysis establishes rigorous connections to AUC/pAUC optimization and provides generalization guarantees. Extensive experiments validate that our method advances both discriminative accuracy and fine-grained perceptual representation.
Figures & tables
| CLIP Backbone | Resol. | #Params | Method | Avg | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| OpenAI ViT-L-14 | 224 | 427.6 | Original | 13.3 | 13.3 | 20.0 | 20.0 | 13.3 | 53.3 | 20.0 | 6.7 | 13.3 | 19.2 |
| DIVA | 13.3 | 6.7 | 13.3 | 26.7 | 13.3 | 46.7 | 26.7 | 20.0 | 26.7 | 21.5 | |||
| +DCR | 6.7 | 20.0 | 33.3 | 0.0 | 26.7 | 66.7 | 20.0 | 6.7 | 20.0 | 22.2 | |||
| +AUCGR | 0.0 | 33.3 | 26.7 | 6.7 | 0.0 | 73.3 | 33.3 | 13.3 | 13.3 | 22.2 | |||
| +pAUCGR | 6.7 | 13.3 | 46.7 | 6.7 | 6.7 | 66.7 | 20.0 | 20.0 | 33.3 | 24.4 | |||
| GenHancer | 13.3 | 13.3 | 13.3 | 13.3 | 13.3 | 66.7 | 40.0 | 20.0 | 33.3 | 25.2 |
| Backbone | Method | CIFAR-10 [ 114 ] | CIFAR-100 [ 114 ] | MNIST [ 115 ] | Caltech-101 [ 58 ] | SUN397 [ 116 ] | FGVC Aircraft [ 117 ] | Country-211 [ 11 ] | Stanford Cars [ 118 ] | DTD [ 59 ] | Eurosat [ 119 ] | FER2013 [ 120 ] | Flowers-102 [ 121 ] | Food-101 [ 122 ] | GTSRB [ 123 ] | PCam [ 124 ] | Pets [ 125 ] | Rendered SST2 [ 11 ] | Resisc45 [ 126 ] | STL10 [ 127 ] | VOC2007 [ 128 ] | ImageNet-1K [ 57 ] | ImageNet-V2 [ 129 ] | ImageNet-Adv. [ 130 ] | ImageNet-Ren. [ 131 ] | ImageNet-OOD [ 130 ] | ObjectNet [ 132 ] | Avg. ACC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| O-1 | Original | 83.57 | 51.32 | 67.95 | 49.63 | 50.03 | 35.70 | 13.19 | 54.94 | 49.41 | 69.30 | 25.62 | 82.14 | 85.01 | 46.69 | 62.92 | 64.90 | 52.22 | 70.71 | 95.74 | 41.03 | 53.92 | 42.39 | 35.77 | 38.84 | 27.85 | 44.24 | 53.66 |
| DIVA | 80.57 | 41.80 | 62.71 | 45.57 | 47.21 | 22.77 | 7.83 | 20.07 | 48.46 | 65.02 | 28.45 | 68.58 | 68.82 | 36.64 | 65.03 | 41.37 | 52.11 | 67.56 | 97.83 | 37.47 | 36.80 | 31.38 | 17.31 | 17.29 | 28.70 | 26.20 | 44.75 | |
| +DCR | 75.47 | 42.18 | 58.57 | 42.94 | 49.80 | 30.60 | 12.05 | 41.43 | 42.13 | 53.22 | 28.53 | 74.96 | 77.55 | 35.31 | 61.42 | 51.05 | 50.69 | 68.71 | 95.25 | 39.77 | 45.35 | 35.85 | 32.47 | 14.01 | 26.90 | 41.45 | 47.22 | |
| +AUCGR | 86.14 | 43.49 | 65.82 | 47.05 | 48.14 | 28.11 | 10.53 | 37.68 | 48.99 | 77.93 | 26.02 | 77.04 | 76.32 | 43.59 | 64.87 | 49.74 | 54.20 | 75.38 | 94.39 | 39.82 | 43.71 | 36.43 | 25.37 | 21.12 | 27.65 | 39.77 | 49.59 | |
| +pAUCGR | 83.47 | 51.72 | 66.43 | 49.88 | 50.71 | 34.56 | 12.78 | 55.74 | 51.81 | 69.57 | 25.61 | 78.89 | 83.98 | 47.20 | 62.69 | 61.41 | 52.11 | 73.71 | 95.76 | 46.65 | 53.89 | 41.57 | 37.97 | 39.55 | 27.80 | 45.46 | 53.88 | |
| GenHancer | 44.25 | 19.39 | 60.34 | 33.54 | 30.20 | 14.46 | 7.17 | 12.29 | 30.96 | 65.44 | 30.59 | 36.67 | 32.04 | 26.88 | 64.03 | 21.10 | 51.78 | 56.13 | 55.83 | 26.70 | 18.20 | 20.87 | 11.31 | 7.43 | 24.00 | 7.85 | 31.13 |
| LLM | CLIP | Vision-Centric Benchmarks | Conventional MLLM Benchmarks | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MMVP- MLLM [ 53 ] | NaturalBench [ 63 ] | CV-Bench 2D [ 62 ] | CV-Bench 3D [ 62 ] | POPE [ 133 ] | SciQA- IMG [ 134 ] | Hallusion Avg. [ 135 ] | ||||||||
| Acc | Q-Acc | I-Acc | G-Acc | ADE20K | COCO | rand | pop | adv | ||||||
| Vicuna-7B | Original | 24.7 | 76.4 | 53.6 | 56.4 | 17.6 | 49.6 | 60.9 | 58.7 | 87.3 | 86.1 | 84.2 | 66.8 | 27.6 |
| DIVA | 31.3 | 75.3 | 51.7 | 56.1 | 22.3 | 51.3 | 63.4 | 60.2 | 87.9 | 87.0 | 84.6 | 66.3 | 28.6 | |
| +pAUCGR | 31.3 | 76.3 | 53.9 | 56.1 | 25.5 | 54.1 | 64.7 | 62.6 | 87.3 | 87.2 | 84.9 | 66.4 | 28.6 | |
| GenHancer | 30.7 | 77.3 | 55.6 | 59.1 | 24.4 | 52.9 | 63.6 | 63.2 | 88.1 | 86.7 | 84.6 | 66.5 | 28.4 | |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Notation | Description |
|---|---|
| Input image with height and width . | |
| Training dataset. | |
| , | VAE latent of and its noisy latent at step . |
| Diffusion or flow timestep. | |
| visual encoder. | |
| Visual representation extracted by . |
| Dataset | OpenAI CLIP ViT-L@224 (O-1) | MetaCLIP ViT-L@224 (M-1) | SigLIP ViT-SO@224 (S-1) | ||||||||||||||||||||||||
| Original | DIVA | +DCR | +AUCGR | +pAUCGR | GenHancer | +DCR | +AUCGR | +pAUCGR | Original | DIVA | +DCR | +AUCGR | +pAUCGR | GenHancer | +DCR | +AUCGR | +pAUCGR | Original | DIVA | +DCR | +AUCGR | +pAUCGR | GenHancer | +DCR | +AUCGR | +pAUCGR | |
| CIFAR-10 | 0.786 | 0.675 | 0.616 | 0.759 | 0.784 | 0.252 | 0.538 | 0.573 | 0.565 | 0.791 | 0.630 | 0.774 | 0.754 | 0.801 | 0.399 | 0.751 | 0.789 | 0.793 | 0.878 | 0.404 | 0.740 | 0.801 | 0.937 | 0.936 | 0.938 | 0.939 | 0.939 |
| CIFAR-100 | 0.384 | 0.286 | 0.274 | 0.303 | 0.382 | 0.088 | 0.209 | 0.214 | 0.226 | 0.396 | 0.309 | 0.340 | 0.330 | 0.431 | 0.182 | 0.366 | 0.364 | 0.403 | 0.551 | 0.206 | 0.537 | 0.465 | 0.538 | 0.548 | 0.549 | 0.537 | 0.534 |
| MNIST | 0.449 | 0.407 | 0.333 | 0.463 | 0.501 | 0.365 | 0.422 | 0.585 | 0.406 | 0.496 | 0.513 | 0.476 | 0.492 | 0.472 | 0.524 | 0.449 | 0.475 | 0.475 | 0.555 | 0.544 | 0.605 | 0.551 | 0.562 | 0.571 | 0.559 | 0.579 | 0.561 |
| Caltech-101 | 0.302 | 0.299 | 0.250 | 0.283 | 0.306 | 0.215 | 0.282 | 0.276 | 0.258 | 0.302 | 0.298 | 0.312 | 0.309 | 0.298 | 0.235 | 0.294 | 0.305 | 0.309 | 0.268 | 0.207 | 0.287 | 0.268 | 0.289 | 0.281 | 0.276 | 0.294 | 0.289 |
| SUN397 | 0.379 | 0.363 | 0.371 | 0.355 | 0.380 | 0.186 | 0.306 | 0.304 | 0.336 | 0.386 | 0.326 | 0.321 | 0.320 | 0.387 | 0.113 | 0.302 | 0.292 | 0.369 | 0.432 | 0.052 | 0.418 | 0.404 | 0.430 | 0.430 | 0.431 | 0.428 | 0.431 |
| Dataset | OpenAI CLIP ViT-L@224 (O-1) | MetaCLIP ViT-L@224 (M-1) | SigLIP ViT-SO@224 (S-1) | ||||||||||||||||||||||||
| Original | DIVA | +DCR | +AUCGR | +pAUCGR | GenHancer | +DCR | +AUCGR | +pAUCGR | Original | DIVA | +DCR | +AUCGR | +pAUCGR | GenHancer | +DCR | +AUCGR | +pAUCGR | Original | DIVA | +DCR | +AUCGR | +pAUCGR | GenHancer | +DCR | +AUCGR | +pAUCGR | |
| CIFAR-10 | 0.841 | 0.745 | 0.716 | 0.815 | 0.839 | 0.364 | 0.636 | 0.677 | 0.662 | 0.860 | 0.765 | 0.839 | 0.831 | 0.865 | 0.508 | 0.811 | 0.845 | 0.862 | 0.907 | 0.531 | 0.931 | 0.863 | 0.937 | 0.935 | 0.937 | 0.937 | 0.938 |
| CIFAR-100 | 0.679 | 0.611 | 0.592 | 0.633 | 0.675 | 0.381 | 0.557 | 0.541 | 0.573 | 0.710 | 0.651 | 0.669 | 0.667 | 0.722 | 0.498 | 0.683 | 0.682 | 0.708 | 0.798 | 0.520 | 0.786 | 0.744 | 0.791 | 0.789 | 0.791 | 0.791 | 0.788 |
| MNIST | 0.617 | 0.556 | 0.572 | 0.617 | 0.647 | 0.525 | 0.538 | 0.735 | 0.526 | 0.639 | 0.690 | 0.614 | 0.627 | 0.661 | 0.662 | 0.596 | 0.619 | 0.624 | 0.696 | 0.680 | 0.746 | 0.691 | 0.702 | 0.705 | 0.700 | 0.685 | 0.700 |
| Caltech-101 | 0.809 | 0.780 | 0.757 | 0.793 | 0.810 | 0.662 | 0.750 | 0.772 | 0.759 | 0.823 | 0.807 | 0.818 | 0.827 | 0.820 | 0.662 | 0.811 | 0.816 | 0.826 | 0.805 | 0.601 | 0.805 | 0.807 | 0.809 | 0.809 | 0.807 | 0.813 | 0.811 |
| SUN397 | 0.759 | 0.743 | 0.744 | 0.740 | 0.755 | 0.588 | 0.693 | 0.697 | 0.716 | 0.764 | 0.715 | 0.715 | 0.715 | 0.763 | 0.494 | 0.708 | 0.703 | 0.752 | 0.796 | 0.312 | 0.790 | 0.782 | 0.795 | 0.796 | 0.795 | 0.794 | 0.795 |
| Dataset | OpenAI CLIP ViT-L@224 (O-1) | MetaCLIP ViT-L@224 (M-1) | SigLIP ViT-SO@224 (S-1) | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DIVA | +AUCGR | +pAUCGR | GenHancer | +AUCGR | +pAUCGR | DIVA | +AUCGR | +pAUCGR | GenHancer | +AUCGR | +pAUCGR | DIVA | +AUCGR | +pAUCGR | GenHancer | +AUCGR | +pAUCGR | |
| CIFAR-10 | 86.93 | 92.22 | 95.60 | 76.36 | 75.76 | 79.45 | 96.66 | 97.45 | 97.56 | 58.19 | 96.53 | 97.33 | 95.13 | 92.30 | 96.68 | 96.71 | 96.69 | 96.70 |
| CIFAR-100 | 58.44 | 70.89 | 75.69 | 36.34 | 43.81 | 51.89 | 79.83 | 81.11 | 84.12 | 28.63 | 81.32 | 84.18 | 76.86 | 78.41 | 84.11 | 83.70 | 84.21 | 84.23 |
| MNIST | 53.63 | 67.28 | 76.65 | 49.02 | 66.28 | 48.99 | 62.90 | 62.21 | 63.13 | 29.85 | 63.99 | 64.71 | 74.84 | 81.77 | 88.12 | 88.33 | 89.11 | 89.13 |
| Caltech-101 | 79.97 | 81.35 | 83.35 | 69.50 | 80.38 | 78.54 | 83.81 | 89.47 | 87.07 | 34.91 | 84.77 | 86.52 | 71.80 | 85.83 | 85.98 | 86.06 | 86.01 | 86.03 |
| SUN397 | 55.47 | 63.43 | 67.80 | 30.30 | 52.73 | 48.99 | 63.63 | 70.62 | 73.36 | 12.27 | 71.79 | 74.07 | 26.38 | 72.55 | 74.43 | 74.42 | 74.45 | 74.42 |
| Dataset | OpenAI CLIP ViT-L@224 (O-1) | MetaCLIP ViT-L@224 (M-1) | SigLIP ViT-SO@224 (S-1) | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DIVA | +AUCGR | +pAUCGR | GenHancer | +AUCGR | +pAUCGR | DIVA | +AUCGR | +pAUCGR | GenHancer | +AUCGR | +pAUCGR | DIVA | +AUCGR | +pAUCGR | GenHancer | +AUCGR | +pAUCGR | |
| CIFAR-10 | 99.13 | 99.41 | 99.64 | 96.78 | 97.61 | 98.48 | 99.94 | 99.95 | 99.93 | 93.39 | 99.90 | 99.95 | 99.43 | 99.37 | 99.81 | 99.83 | 99.86 | 99.86 |
| CIFAR-100 | 81.95 | 90.58 | 92.74 | 64.19 | 70.01 | 76.23 | 96.10 | 95.92 | 96.93 | 58.32 | 95.85 | 96.66 | 91.60 | 93.11 | 96.07 | 95.83 | 96.03 | 96.03 |
| MNIST | 87.91 | 89.14 | 93.52 | 84.49 | 88.97 | 88.08 | 92.54 | 95.52 | 94.38 | 81.94 | 93.63 | 94.67 | 94.14 | 97.16 | 98.81 | 98.99 | 99.00 | 99.00 |
| Caltech-101 | 92.23 | 94.23 | 95.43 | 87.87 | 96.34 | 92.36 | 97.73 | 96.06 | 96.42 | 64.39 | 94.53 | 95.27 | 94.95 | 96.17 | 95.94 | 96.04 | 95.79 | 95.76 |
| SUN397 | 84.77 | 91.52 | 93.70 | 65.04 | 84.52 | 78.28 | 90.78 | 94.66 | 95.85 | 32.82 | 95.02 | 96.06 | 54.22 | 95.19 | 95.81 | 95.88 | 95.83 | 95.83 |
| Dataset | OpenAI CLIP ViT-L@224 (O-1) | MetaCLIP ViT-L@224 (M-1) | SigLIP ViT-SO@224 (S-1) | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DIVA | +AUCGR | +pAUCGR | GenHancer | +AUCGR | +pAUCGR | DIVA | +AUCGR | +pAUCGR | GenHancer | +AUCGR | +pAUCGR | DIVA | +AUCGR | +pAUCGR | GenHancer | +AUCGR | +pAUCGR | |
| CIFAR-10 | 86.95 | 92.22 | 95.62 | 76.35 | 75.76 | 79.44 | 96.67 | 97.45 | 97.57 | 58.09 | 96.58 | 97.33 | 95.19 | 92.31 | 96.68 | 96.71 | 96.69 | 96.69 |
| CIFAR-100 | 58.45 | 70.88 | 75.72 | 36.35 | 43.79 | 51.88 | 79.82 | 81.09 | 84.10 | 28.64 | 81.32 | 84.20 | 76.86 | 78.42 | 84.12 | 83.71 | 84.22 | 84.26 |
| MNIST | 53.68 | 66.76 | 76.32 | 49.37 | 65.46 | 48.97 | 63.20 | 62.89 | 63.74 | 29.76 | 64.30 | 65.12 | 74.33 | 81.52 | 87.98 | 88.21 | 89.01 | 89.00 |
| Caltech-101 | 87.27 | 90.89 | 92.49 | 78.14 | 81.66 | 83.82 | 93.05 | 95.18 | 95.29 | 29.23 | 93.85 | 95.18 | 84.14 | 95.40 | 95.99 | 96.11 | 96.03 | 96.03 |
| SUN397 | 50.66 | 63.31 | 68.45 | 30.74 | 52.44 | 43.27 | 62.02 | 70.32 | 74.71 | 10.59 | 70.32 | 75.09 | 25.21 | 73.79 | 75.82 | 76.05 | 75.93 | 75.92 |
| Method | P-Ability | D-Ability |
| DIVA | 21.48 | 44.75 |
| Naive | 22.22 | 47.23 |
| Naive+Random | 20.74 | 43.32 |
| Naive+Partial | 17.78 | 42.75 |
| AUCGR | 22.22 | 49.59 |
| AUCGR+Random | 21.48 | 48.88 |
| Backbone | Training Costs | DIVA | +AUCGR | +pAUCGR |
|---|---|---|---|---|
| OpenAI CLIP ViT-L@224 | Time | 3.48 | 4.97 ( ) | 5.72 ( ) |
| GPU Memory | 13.72 | 21.17 ( ) | 22.17 ( ) | |
| OpenAI CLIP ViT-L@336 | Time | 4.80 | 7.43 ( ) | 8.23 ( ) |
| GPU Memory | 22.27 | 38.82 ( ) | 41.28 ( ) | |
| MetaCLIP ViT-L@224 | Time | 3.60 | 4.54 ( ) | 5.73 ( ) |
| GPU Memory | 13.75 | 20.94 ( ) | 22.17 ( ) |
Representation Learning
Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.
See all 863 papers