Authors: Jinhao Li, Sarah Erfani, Lei Feng, Guangrui Li, James Bailey, Feng Liu
Organizations: School of Computing and Information Systems, The University of Melbourne, Australia · School of Computer Science and Engineering, Southeast University, China · AWS AI Labs, USA · Department of Data Science and Artificial Intelligence, Monash University, Australia
Image-text alignment models such as CLIP are typically trained with contrastive learning, where unpaired examples are treated as strict negatives. While this assumption introduces noise by penalising semantically related pairs, existing solutions often rely on heuristic similarity thresholds that lack statistical grounding and sensitivity to dataset-specific noise distributions. In this paper, we propose Permute-then-Adapt (PTA), a framework that leverages efficient weak models to identify missed positives while controlling Type-I error relative to a conditional permutation reference, up to calibration approximation error. Unlike standard distillation or arbitrary thresholding, PTA estimates a random-pairing reference distribution from the weak teacher's similarity scores and uses its upper tail, determined by α, to select auxiliary positives. The calibrated positive set is then trained against by a single multi-positive contrastive objective: each anchor maximises the total probability mass it assigns to its positive set. Offline calibration with the weak model adds negligible training overhead. Experiments show gains over heuristic soft-label approaches in object recognition, cross-modal retrieval, and data efficiency. Project repository: https://github.com/JinhaoLee/Permute-then-Adapt.
Figures & tables
Figure 1: The risk of uncalibrated heuristics. While a fixed threshold (red dashed) works on COCO ( Left ), it fails on fine-grained Stanford Cars ( Right ), where the random-pairing null distribution shifts rightward. The fixed threshold accepts 12 × more false positives from the null on Cars. In contrast, PTA (green solid) estimates a dataset-specific permutation null and adapts its cutoff to control Type-I error relative to that reference at level α , up to calibration approximation error.
Figure 2: Overview of Permute-then-Adapt (PTA). Given paired image–text samples, PTA uses a frozen weak teacher to estimate a permutation null distribution and dataset-adaptive threshold. This threshold selects calibrated off-diagonal auxiliary positives, which join ground-truth diagonal pairs to form positive sets. The student is optimized with a positive-set marginal likelihood objective.
Method
ImageNet
IN-V2
IN-A
IN-R
Cars
DTD
C-100
C-10
Food
CUB
Aircraft
Flowers
Pets
Avg.
66.23
59.49
37.15
71.82
59.38
43.36
67.46
90.40
84.55
53.30
19.86
63.27
87.11
61.80
SA
± 0.05
± 0.13
± 0.27
± 0.22
± 0.23
± 0.23
± 0.41
± 0.40
± 0.13
± 0.24
± 0.48
± 0.12
± 0.17
± 0.06
65.78
58.74
40.89
72.55
58.29
43.17
65.88
89.80
82.58
49.88
20.15
62.07
87.63
61.34
VPL
± 0.05
± 0.07
± 0.31
± 0.14
± 0.15
± 0.29
± 0.51
± 0.22
± 0.14
± 0.31
± 0.17
± 0.41
± 0.20
± 0.08
65.02
57.76
40.49
72.15
56.87
42.19
65.05
89.63
81.80
48.05
18.83
61.39
87.31
60.50
FT
± 0.05
± 0.06
± 0.30
± 0.16
± 0.33
± 0.27
± 0.57
± 0.25
± 0.08
± 0.22
± 0.43
± 0.51
± 0.21
± 0.06
Table 1: Zero-shot classification accuracy (%). CLIP ViT-B/16 students are fine-tuned on COCO train2014 with a CLIP ViT-B/32 teacher. Upper and lower rows show the mean and sample SD over seeds 41–45. PTA uses finite-rank calibration (implementation details). Best means are bold.
Flickr30k
MS-COCO
Method
Image to Text
Text to Image
Image to Text
Text to Image
R@1
R@5
R@1
R@5
R@1
R@5
R@1
R@5
SFT
81.00
95.60
79.70
95.30
22.69
41.69
21.94
41.29
SA
81.90
96.40
81.70
96.00
23.65
43.69
22.26
42.06
VPL
81.30
96.10
81.50
95.80
24.91
45.35
24.24
44.81
FT
83.00
96.20
81.30
96.40
23.91
43.72
23.84
43.69
Table 2: Cross-modal retrieval (recall@1/5). All systems are fine-tuned on COCO train2014; best scores are bold. SFT: supervised fine-tuning. Other abbreviations follow Table 1 .
Figure 3: Sensitivity to weak-teacher corruption. Teacher and PTA student accuracy under inference-time teacher dropout, with the student setup fixed. Points reproduce reported benchmark averages; per-task results are in Appendix A.8 .
Variant
Avg. gain (pp)
SFT (ceiling)
–
CN-only (App. A.1.1 )
+1.076
{\color[rgb]{0.35,0.35,0.35}\pm 0.061}
Asymmetric (t → i: CE)
+1.147
{\color[rgb]{0.35,0.35,0.35}\pm 0.035}
PTA (full)
{\color[rgb]{0.1094,0.418,0.3945}\mathbf{+1.524}}
{\color[rgb]{0.35,0.35,0.35}\pm 0.108}
Table 3: PTA mechanism decomposition. Independent ablations of Eq. 13 : paired accuracy gains (pp) over matched SFT controls on 13 benchmarks, mean ± SD over five seeds (three for asymmetric).
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
# permuted scores
Across-seed range of Q
Std of Q
1,000
8.50 – 9.00
0.25
5,000 (default)
8.44 – 8.69
0.13
10,000
8.49 – 8.62
0.07
20,000
8.51 – 8.57
0.03
Appendix
Table 4: Threshold stability. Range and standard deviation of Q across seeds {41,42,43} for different numbers of permuted scores in Sperm . Threshold units are cosine similarity ⋅exp(logit_scale) . The default is 5,000 scores from R=5 random permutations of 1,000 pairs.
Method
Avg. accuracy (%)
Δ vs. SFT
SFT
62.42
0.00
FT
63.08
0.66
Naive percentile
64.59
2.17
PTA
65.60
+3.18
Appendix
Table 5: Thresholding ablation. Average accuracy (%) on a legacy benchmark suite at B=4096 with the last four vision blocks unfrozen (single seed); included to mirror prior protocol.
α
R
Q
Reference exceedance (%)
Auxiliary selection (%)
0.005
1
28.262±0.359
0.462±0.067
0.451±0.077
0.005
10
28.152±0.142
0.473±0.024
0.471±0.031
0.005
100
28.151±0.048
0.468±0.008
0.471±0.011
0.010
1
26.520±0.294
0.969±0.094
0.932±0.106
0.010
10
26.384±0.126
1.007±0.036
0.977±0.046
0.010
100
26.387±0.036
1.008±0.011
0.975±0.013
Appendix
Table 6: Global-permutation calibration on fixed features ( n=5,000 ). Values are mean ± sample standard deviation across 30 permutation seeds. Q is in scaled-similarity units. Rates are percentages, with their standard deviations in percentage points. Variability is conditional on the fixed records and features; reference exceedance measures random-pair selection, not semantic error.
α
M
COCO ViT-B/32
COCO ViT-B/16
Flickr30k ViT-B/32
Flickr30k ViT-B/16
0.005
100
1.258±1.166
1.170±0.869
1.384±1.248
1.176±0.945
0.005
200
0.909±0.742
0.904±0.684
0.951±0.817
0.942±0.707
0.005
500
0.643±0.330
0.631±0.342
0.595±0.321
0.605±0.309
0.005
1,000
0.560±0.250
0.548±0.222
0.581±0.293
0.572±0.220
0.005
5,000
0.518±0.109
0.509±0.101
0.509±0.136
0.525±0.121
0.005
10,000
0.515±0.089
0.508±0.094
0.496±0.108
0.519±0.094
Appendix
Table 7: Held-out random-pairing exceedance rate (%) for the linear-quantile calibration rule. Entries are mean ± sample standard deviation over 100 resampling seeds; standard deviations are in percentage points. M is the number of retained permuted calibration scores. These rates concern random pairing, rather than semantic false positives.
α
Dropout
Q
Auxiliary selection (%)
Known-positive recall (%)
0.005
0.0
27.698±0.623
0.589±0.145
80.72±5.44
0.005
0.1
27.509±0.537
0.581±0.139
74.49±4.96
0.005
0.3
26.827±0.376
0.540±0.122
46.70±5.12
0.005
0.5
27.756±0.157
0.510±0.127
4.88±0.88
0.005
0.8
25.418±0.178
0.633±0.146
0.65±0.16
0.010
0.0
25.990±0.599
1.143±0.255
91.51±2.49
Appendix
Table 8: Calibration-only weak-teacher corruption diagnostic. Entries are mean ± sample standard deviation across five paired seeds. Auxiliary selection and known-positive recall are percentages; their standard deviations are in percentage points. No student is trained in this experiment.
Figure 7: PTA data efficiency on CC3M.
Dropout
Caltech101
CIFAR10
CIFAR100
DTD
Flowers102
Food101
SUN397
ImageNet
Average
0.0
59.58
58.38
42.66
36.52
60.60
88.71
80.55
61.00
61.00
0.1
52.77
50.22
36.06
39.74
51.66
89.26
77.36
56.72
56.72
0.3
28.31
21.74
26.17
26.36
38.93
81.08
50.43
39.00
39.00
0.5
7.32
4.44
10.96
12.51
10.02
33.37
13.09
13.10
13.10
0.8
0.11
0.50
2.07
9.77
1.06
10.32
0.97
3.54
3.54
Appendix
Table 9: Zero-shot classification accuracy (%) of CLIP-B/32 with different dropout rates during inference. Higher dropout rates simulate progressively weaker models.
Weak Dropout
Caltech101
CIFAR10
CIFAR100
DTD
Flowers102
Food101
SUN397
ImageNet
Average
0.0
64.14
60.40
44.57
50.51
66.45
89.38
83.77
65.60
65.60
0.1
64.13
59.47
44.52
50.94
66.58
89.36
83.98
65.57
65.57
0.3
62.80
56.06
44.31
51.17
65.56
89.47
83.04
64.63
64.63
0.5
58.74
47.10
44.73
49.75
62.43
88.72
77.12
61.23
61.23
0.8
56.84
47.78
40.53
46.51
60.81
87.74
78.30
59.79
59.79
Appendix
Table 10: Classification accuracy (%) after fine-tuning CLIP-B/16 using PTA with CLIP-B/32 weak models at different dropout rates. All use identical hyperparameters on COCO Captions.
Method
Runs
GPU mix across runs
Elapsed/run (h)
GPU-hours
SA
5
L40S (5)
1.08–1.35
5.99
VPL
5
L40S (5)
1.34–1.98
8.15
FT
5
L40S (4), H100 (1)
0.99–1.62
6.81
MD
5
L40S (5)
1.43–1.74
7.99
WSA
5
L40S (5)
1.44–1.67
7.78
PTA
5
A100 (2), H100 (3)
1.00–2.14
7.24
Appendix
Table 11: Recorded training allocations for Table 1 . Each job uses one GPU. Counts in parentheses indicate how many of the five seeds ran on each GPU model. Elapsed time is the per-job range; GPU-hours sum the five allocations, using unrounded times.
Figure 8: Qualitative retrieval. Three selected examples. Within each top-3 strip, ranks run left to right; green outlines mark the matching image, ranked third by SA and first by PTA.
Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.
School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China · Li Auto Inc. · Institute of AI for Industries, Chinese Academy of Sciences