Authors: Jinhao Li, Sarah Erfani, Lei Feng, Guangrui Li, James Bailey, Feng Liu
Organizations: School of Computing and Information Systems, The University of Melbourne, Australia · School of Computer Science and Engineering, Southeast University, China · AWS AI Labs, USA · Department of Data Science and Artificial Intelligence, Monash University, Australia
Image-text alignment models such as CLIP are typically trained with contrastive learning, where unpaired examples are treated as strict negatives. While this assumption introduces noise by penalising semantically related pairs, existing solutions often rely on heuristic similarity thresholds that lack statistical grounding and sensitivity to dataset-specific noise distributions. In this paper, we propose Permute-then-Adapt (PTA), a framework that leverages efficient weak models to identify missed positives while controlling Type-I error relative to a conditional permutation reference, up to calibration approximation error. Unlike standard distillation or arbitrary thresholding, PTA estimates a random-pairing reference distribution from the weak teacher's similarity scores and uses its upper tail, determined by α, to select auxiliary positives. The calibrated positive set is then trained against by a single multi-positive contrastive objective: each anchor maximises the total probability mass it assigns to its positive set. Offline calibration with the weak model adds negligible training overhead. Experiments show gains over heuristic soft-label approaches in object recognition, cross-modal retrieval, and data efficiency. Project repository: https://github.com/JinhaoLee/Permute-then-Adapt.
Figures & tables
Figure 1: The risk of uncalibrated heuristics. While a fixed threshold (red dashed) works on COCO ( Left ), it fails on fine-grained Stanford Cars ( Right ), where the random-pairing null distribution shifts rightward. The fixed threshold accepts 12 × more false positives from the null on Cars. In contrast, PTA (green solid) estimates a dataset-specific permutation null and adapts its cutoff to control Type-I error relative to that reference at level α , up to calibration approximation error.
Figure 2: Overview of Permute-then-Adapt (PTA). Given paired image–text samples, PTA uses a frozen weak teacher to estimate a permutation null distribution and dataset-adaptive threshold. This threshold selects calibrated off-diagonal auxiliary positives, which join ground-truth diagonal pairs to form positive sets. The student is optimized with a positive-set marginal likelihood objective.
Method
ImageNet
IN-V2
IN-A
IN-R
Cars
DTD
C-100
C-10
Food
CUB
Aircraft
Flowers
Pets
Avg.
66.23
59.49
37.15
71.82
59.38
43.36
67.46
90.40
84.55
53.30
19.86
63.27
87.11
61.80
SA
± 0.05
± 0.13
± 0.27
± 0.22
± 0.23
± 0.23
± 0.41
± 0.40
± 0.13
± 0.24
± 0.48
± 0.12
± 0.17
± 0.06
65.78
58.74
40.89
72.55
58.29
43.17
65.88
89.80
82.58
49.88
20.15
62.07
87.63
61.34
VPL
± 0.05
± 0.07
± 0.31
± 0.14
± 0.15
± 0.29
± 0.51
± 0.22
± 0.14
± 0.31
± 0.17
± 0.41
± 0.20
± 0.08
65.02
57.76
40.49
72.15
56.87
42.19
65.05
89.63
81.80
48.05
18.83
61.39
87.31
60.50
FT
± 0.05
± 0.06
± 0.30
± 0.16
± 0.33
± 0.27
± 0.57
± 0.25
± 0.08
± 0.22
± 0.43
± 0.51
± 0.21
± 0.06
Table 1: Zero-shot classification accuracy (%). CLIP ViT-B/16 students are fine-tuned on COCO train2014 with a CLIP ViT-B/32 teacher. Upper and lower rows show the mean and sample SD over seeds 41–45. PTA uses finite-rank calibration (implementation details). Best means are bold.
Flickr30k
MS-COCO
Method
Image to Text
Text to Image
Image to Text
Text to Image
R@1
R@5
R@1
R@5
R@1
R@5
R@1
R@5
SFT
81.00
95.60
79.70
95.30
22.69
41.69
21.94
41.29
SA
81.90
96.40
81.70
96.00
23.65
43.69
22.26
42.06
VPL
81.30
96.10
81.50
95.80
24.91
45.35
24.24
44.81
FT
83.00
96.20
81.30
96.40
23.91
43.72
23.84
43.69
Table 2: Cross-modal retrieval (recall@1/5). All systems are fine-tuned on COCO train2014; best scores are bold. SFT: supervised fine-tuning. Other abbreviations follow Table 1 .
Figure 3: Sensitivity to weak-teacher corruption. Teacher and PTA student accuracy under inference-time teacher dropout, with the student setup fixed. Points reproduce reported benchmark averages; per-task results are in Appendix A.8 .
Variant
Avg. gain (pp)
SFT (ceiling)
–
CN-only (App. A.1.1 )
+1.076
{\color[rgb]{0.35,0.35,0.35}\pm 0.061}
Asymmetric (t → i: CE)
+1.147
{\color[rgb]{0.35,0.35,0.35}\pm 0.035}
PTA (full)
{\color[rgb]{0.1094,0.418,0.3945}\mathbf{+1.524}}
{\color[rgb]{0.35,0.35,0.35}\pm 0.108}
Table 3: PTA mechanism decomposition. Independent ablations of Eq. 13 : paired accuracy gains (pp) over matched SFT controls on 13 benchmarks, mean ± SD over five seeds (three for asymmetric).
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
# permuted scores
Across-seed range of Q
Std of Q
1,000
8.50 – 9.00
0.25
5,000 (default)
8.44 – 8.69
0.13
10,000
8.49 – 8.62
0.07
20,000
8.51 – 8.57
0.03
Appendix
Table 4: Threshold stability. Range and standard deviation of Q across seeds {41,42,43} for different numbers of permuted scores in Sperm . Threshold units are cosine similarity ⋅exp(logit_scale) . The default is 5,000 scores from R=5 random permutations of 1,000 pairs.
Method
Avg. accuracy (%)
Δ vs. SFT
SFT
62.42
0.00
FT
63.08
0.66
Naive percentile
64.59
2.17
PTA
65.60
+3.18
Appendix
Table 5: Thresholding ablation. Average accuracy (%) on a legacy benchmark suite at B=4096 with the last four vision blocks unfrozen (single seed); included to mirror prior protocol.
α
R
Q
Reference exceedance (%)
Auxiliary selection (%)
0.005
1
28.262±0.359
0.462±0.067
0.451±0.077
0.005
10
28.152±0.142
0.473±0.024
0.471±0.031
0.005
100
28.151±0.048
0.468±0.008
0.471±0.011
0.010
1
26.520±0.294
0.969±0.094
0.932±0.106
0.010
10
26.384±0.126
1.007±0.036
0.977±0.046
0.010
100
26.387±0.036
1.008±0.011
0.975±0.013
Appendix
Table 6: Global-permutation calibration on fixed features ( n=5,000 ). Values are mean ± sample standard deviation across 30 permutation seeds. Q is in scaled-similarity units. Rates are percentages, with their standard deviations in percentage points. Variability is conditional on the fixed records and features; reference exceedance measures random-pair selection, not semantic error.
α
M
COCO ViT-B/32
COCO ViT-B/16
Flickr30k ViT-B/32
Flickr30k ViT-B/16
0.005
100
1.258±1.166
1.170±0.869
1.384±1.248
1.176±0.945
0.005
200
0.909±0.742
0.904±0.684
0.951±0.817
0.942±0.707
0.005
500
0.643±0.330
0.631±0.342
0.595±0.321
0.605±0.309
0.005
1,000
0.560±0.250
0.548±0.222
0.581±0.293
0.572±0.220
0.005
5,000
0.518±0.109
0.509±0.101
0.509±0.136
0.525±0.121
0.005
10,000
0.515±0.089
0.508±0.094
0.496±0.108
0.519±0.094
Appendix
Table 7: Held-out random-pairing exceedance rate (%) for the linear-quantile calibration rule. Entries are mean ± sample standard deviation over 100 resampling seeds; standard deviations are in percentage points. M is the number of retained permuted calibration scores. These rates concern random pairing, rather than semantic false positives.
α
Dropout
Q
Auxiliary selection (%)
Known-positive recall (%)
0.005
0.0
27.698±0.623
0.589±0.145
80.72±5.44
0.005
0.1
27.509±0.537
0.581±0.139
74.49±4.96
0.005
0.3
26.827±0.376
0.540±0.122
46.70±5.12
0.005
0.5
27.756±0.157
0.510±0.127
4.88±0.88
0.005
0.8
25.418±0.178
0.633±0.146
0.65±0.16
0.010
0.0
25.990±0.599
1.143±0.255
91.51±2.49
Appendix
Table 8: Calibration-only weak-teacher corruption diagnostic. Entries are mean ± sample standard deviation across five paired seeds. Auxiliary selection and known-positive recall are percentages; their standard deviations are in percentage points. No student is trained in this experiment.
Figure 7: PTA data efficiency on CC3M.
Dropout
Caltech101
CIFAR10
CIFAR100
DTD
Flowers102
Food101
SUN397
ImageNet
Average
0.0
59.58
58.38
42.66
36.52
60.60
88.71
80.55
61.00
61.00
0.1
52.77
50.22
36.06
39.74
51.66
89.26
77.36
56.72
56.72
0.3
28.31
21.74
26.17
26.36
38.93
81.08
50.43
39.00
39.00
0.5
7.32
4.44
10.96
12.51
10.02
33.37
13.09
13.10
13.10
0.8
0.11
0.50
2.07
9.77
1.06
10.32
0.97
3.54
3.54
Appendix
Table 9: Zero-shot classification accuracy (%) of CLIP-B/32 with different dropout rates during inference. Higher dropout rates simulate progressively weaker models.
Weak Dropout
Caltech101
CIFAR10
CIFAR100
DTD
Flowers102
Food101
SUN397
ImageNet
Average
0.0
64.14
60.40
44.57
50.51
66.45
89.38
83.77
65.60
65.60
0.1
64.13
59.47
44.52
50.94
66.58
89.36
83.98
65.57
65.57
0.3
62.80
56.06
44.31
51.17
65.56
89.47
83.04
64.63
64.63
0.5
58.74
47.10
44.73
49.75
62.43
88.72
77.12
61.23
61.23
0.8
56.84
47.78
40.53
46.51
60.81
87.74
78.30
59.79
59.79
Appendix
Table 10: Classification accuracy (%) after fine-tuning CLIP-B/16 using PTA with CLIP-B/32 weak models at different dropout rates. All use identical hyperparameters on COCO Captions.
Method
Runs
GPU mix across runs
Elapsed/run (h)
GPU-hours
SA
5
L40S (5)
1.08–1.35
5.99
VPL
5
L40S (5)
1.34–1.98
8.15
FT
5
L40S (4), H100 (1)
0.99–1.62
6.81
MD
5
L40S (5)
1.43–1.74
7.99
WSA
5
L40S (5)
1.44–1.67
7.78
PTA
5
A100 (2), H100 (3)
1.00–2.14
7.24
Appendix
Table 11: Recorded training allocations for Table 1 . Each job uses one GPU. Counts in parentheses indicate how many of the five seeds ran on each GPU model. Elapsed time is the per-job range; GPU-hours sum the five allocations, using unrounded times.
Figure 8: Qualitative retrieval. Three selected examples. Within each top-3 strip, ranks run left to right; green outlines mark the matching image, ranked third by SA and first by PTA.
Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.
Image--text contrastive pretraining has become a dominant paradigm for visual representation learning, yet existing methods often yield representations that remain partially organized by modality rather than by semantics. We propose ITO, a framework addressing this limitation through two complementary mechanisms with distinct roles. Multimodal multiple alignment enriches supervision by constructing diverse cross-modal correspondences from multi-view image augmentations, providing the primary source of discriminative gain. A lightweight training-time multimodal fusion module then acts as a training-time regularizer, encouraging the encoders to produce features that are compatible under fusion. Crucially, the fusion module is discarded at inference, preserving the efficiency of standard dual-encoder architectures while incurring training-time overhead only. Extensive experiments across pretraining scales from millions to billions of image--text pairs show that ITO outperforms strong contrastive baselines at moderate scale and yields consistent gains over CLIP under identical data, backbone, and compute at the 100M--1B scale, on classification, retrieval, and multimodal benchmarks. Controlled comparisons show that these gains are not explained by additional training compute or image augmentation alone. Our analysis further reveals that the contribution of fusion grows with data scale and that it stabilizes optimization, mitigating the late-stage overfitting commonly observed in aggressive contrastive learning. Code is available at https://github.com/showstarpro/ITO.
Hanpeng Liu, Yaqian Li, Zidan Wang +6
School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China · Li Auto Inc. · Institute of AI for Industries, Chinese Academy of Sciences
Image-text alignment is a core problem in computer vision with applications in caption evaluation, hallucination detection, data curation, and the benchmarking of text-to-image (T2I) generators. As T2I models improve, benchmarking has become demanding, requiring metrics capable of finding a series of issues like missing objects, swapped attributes, miscounts, and ignored negations. Recent work addresses this by fine-tuning evaluators on preference data or by prompting a vision-language model, either holistically with the caption or with decomposed verification questions. However, existing approaches fall short: fine-tuned metrics remain bound to one backbone and training distribution; holistic metrics miss fine-grained details; and decomposed metrics rely on a fixed-YES assumption that penalizes faithful images whenever that assumption fails. In contrast, we propose DEPICT, a training-free metric that replaces fixed reference answers with expected agreement between image-based and caption-only answers, weighting questions by how decisively the caption determines them. By replacing fixed references, our agreement rule increases negation accuracy from 19% to 88%. To recover the context lost during decomposition, DEPICT merges this agreement score with a holistic score. We evaluate DEPICT on five benchmarks and eleven backbones from three model families and find that it surpasses all training-free metrics and exceeds fine-tuned evaluators on two out of three human-correlation benchmarks.
Contrastive Language-Image Pretraining learns a shared representation space through large-scale contrastive learning. However, existing methods that enforce global consistency regularization overlook a key challenge: the inherent information asymmetry between images and text: captions typically describe only one specific aspect of an image, thus images with similar visual content can be paired with completely divergent textual content and semantic information. Consequently, global regularizers inadvertently impose constraints between visually similar images whose captions describe divergent aspects, introducing semantic distortion into the representation space. We propose AspectCLIP, a framework that reformulates consistency regularization to respect this one-to-many structure. AspectCLIP first partitions training samples into attribute clusters based on textual similarity to identify aspect-coherent groups, then applies full cyclic consistency within each cluster while restricting cross-cluster regularization to prototype-level comparisons. This aspect-guided regularization enforces strict geometric alignment only when images and texts describe a consistent facet, while allowing flexibility across divergent aspects. Extensive experiments on downstream tasks demonstrate that AspectCLIP consistently outperforms traditional methods and achieves a more structured representation space.
Yiyang Yao, Shanglin Liu, Jianming Lv +4
School of Computer Science and Technology, South China University of Technology, Guangzhou, China