Organizations: School of Information Engineering, Minzu University of China · School of Artificial Intelligence, Beijing University of Posts and Telecommunications · National Library of China, Beijing, China · School of Cyberspace Security, Beijing University of Posts and Telecommunications
Vision-language models like CLIP have achieved remarkable progress in cross-modal representation learning, yet suffer from systematic misclassifications among visually and semantically similar categories. We observe that such confusion patterns are not random but persistently occur between specific category pairs, revealing the model's intrinsic bias and limited fine-grained discriminative ability. To address this, we propose CAPT, a Confusion-Aware Prompt Tuning framework that enables models to learn from their own misalignment. Specifically, we construct a Confusion Bank to explicitly model stable confusion relationships across categories and misclassified samples. On this basis, we introduce a Semantic Confusion Miner (SEM) to capture global inter-class confusion through semantic difference and commonality prompts, and a Sample Confusion Miner (SAM) to retrieve representative misclassified instances from the bank and capture sample-level cues through a Diff-Manner Adapter that integrates global and local contexts. To further unify confusion information across different granularities, a Multi-Granularity Difference Expert (MGDE) module is designed to jointly leverage semantic- and sample-level experts for more robust confusion-aware reasoning. Extensive experiments on 11 benchmark datasets demonstrate that our method significantly reduces confusion-induced errors while enhancing the discriminability and generalization of both base and novel classes, successfully resolving 50.72 percent of confusable sample pairs. Code will be released at https://github.com/greatest-gourmet/CAPT.
Figures & tables
Figure 1: Heatmap of model misclassifications shows certain categories are consistently and frequently mispredicted as specific others, e.g. , in OxfordPets, terrier is misclassified as bulldog 30 times while rarely being mistaken for other classes. CAPT significantly reduces such confusion rates, thereby improving overall accuracy.
Figure 2: Overview of CAPT. By matching feature representations, we first employ a Semantic Confusion Miner (SEM) that, together with statistics from the Confusion Bank, identifies Semantic Confusion Pairs and generates both commonality and difference prompts. Subsequently, the Sample Confusion Miner (SAM) locates the most representative confusing samples based on these pairs and extracts their Sample Confusion Feature via the Diff-Manner Adapter. Finally, the Multi-Granularity Discrepancy Expert (MGDE) module integrates semantic and sample level confusion information for unified representation refinement. Framework of MGDE is shown in Figure 3 .
Figure 3: Overview of Multi-Granularity Discrepancy Expert. MGDE integrates semantic and sample level experts to fuse dual confusion cues, reinforced by random vectors and specific initialization to enhance confusion learning in prompt tuning.
Method
Average
ImageNet
Caltech101
OxfordPets
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
CoOp (IJCV 22)
82.69
63.22
71.66
76.47
68.78
71.92
98.00
89.81
93.73
93.67
95.29
94.47
MaPLe (CVPR 23)
82.28
75.14
78.55
76.66
70.54
73.47
97.74
94.36
96.02
95.43
97.76
96.58
PromptKD (CVPR 24)
86.96
80.73
83.73
80.83
74.66
77.62
98.91
96.65
97.77
96.30
98.01
97.15
Spotlighter (EMNLP 25)
85.65
80.46
82.89
77.62
71.71
74.55
98.86
96.74
97.79
96.48
97.75
97.11
2SFS (CVPR 25)
85.55
75.48
80.20
77.71
70.99
74.20
98.71
94.43
96.53
95.32
97.82
96.55
Table 1: Comparison with other methods on base-to-new generalization with 16-shot.
Source
Target
ImageNet
Caltech101
OxfordPets
StanfordCars
Flowers102
Food101
FGVCAircraft
SUN397
DTD
EuroSAT
UCF101
Average
CoOp
71.51
93.70
89.14
64.51
68.71
85.30
18.47
64.15
41.92
46.39
66.55
63.88
PromptKD
-
93.61
91.59
73.93
75.33
88.84
26.24
68.57
55.08
63.74
76.39
71.33
LwEIB
71.31
94.51
92.50
66.58
73.03
86.37
27.70
69.33
50.63
55.37
70.03
68.81
TAC
72.77
94.53
90.67
65.30
72.20
85.83
23.53
67.63
47.57
48.07
70.00
66.53
CAPT
72.69
94.62
92.53
74.37
75.41
88.75
27.23
69.43
57.21
64.74
75.19
71.95
Table 2: Comparative results in the cross-dataset transfer setting. Bold accuracies are the highest. Optimization is conducted on ImageNet and evaluation is performed on other datasets.
Method
Source
Target
ImageNet
-V2
-Sketch
-A
-R
CoOp
71.51
64.20
47.99
49.71
75.21
LwEIB
71.31
64.47
50.07
51.00
77.81
TAC
72.77
65.97
50.30
51.73
78.50
CAPT
72.69
69.71
50.46
57.73
83.61
Table 3: Comparison with other methods on cross-domain generalization with 16-shot.
Method
Shot
1
2
4
8
16
CoOP
68.09
70.13
73.59
76.45
79.01
MaPLe
61.79
65.28
70.66
73.82
78.55
PromptKD
72.47
75.19
78.46
79.56
86.96
2SFS
72.55
73.28
78.50
80.7
85.50
CAPT
73.11
75.83
78.69
81.91
87.41
Table 4: Comparison with other methods on the few-shot learning setting with average accuracy.
SEM
SAM
MGDE
Base
Novel
HM
✓
80.69
64.39
71.62
✓
83.28
63.41
72.00
✓
✓
85.23
65.17
73.86
✓
✓
✓
87.41
80.90
83.90
Table 5: Ablation study of components in CAPT on base-to-new tasks. SEM: Semantic Confusion Miner. SAM: Sample Confusion Miner. MGDE: Multi-Granularity Discrepancy Expert.
Figure 4: Correction Rate of misclassified samples stored in Confusion Bank.
Method
Base
Novel
HM
CAPT w/o Confusion Statistics.
86.77
78.84
82.62
CAPT w Confusion Statistics.
87.41
80.90
83.90
Table 6: Effects of the Confusion Statistics in Confusion Score.
Method
Base
Novel
HM
Real-GT
86.94
79.52
83.06
Pseudo-GT
87.41
80.90
83.90
Table 7: Confusion Pair Selection with Pseudo-GT and Real-GT.
Figure 5: Grad-Cam of the effects of Diff-Manner Adapter, only using the global part, the local part, and the whole.
Method
Base
Novel
HM
Random Choice
86.86
79.55
83.04
Representative Samples
87.41
80.90
83.90
Table 8: Effects of the Confusion Instance Sampling.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Method Choice
Base
Novel
HM
Correction Rate
CLIP
73.56
77.23
77.35
64.28
CoOp
84.19
71.88
77.55
42.71
MaPLe
85.64
79.14
82.26
39.39
TAC
84.41
78.67
81.44
49.20
PromptKD
87.41
80.90
83.90
50.72
Appendix
Table 9: Impact of Model Choice on Confusion Bank Construction.
Figure 6: Instance of selecting representative confusion samples.
Figure 7: Procedure of generating semantic commonality and difference prompts.
Method
Base
Novel
HM
global
81.22
76.23
78.65
local
85.19
72.88
78.56
global+local
87.41
80.90
83.90
Appendix
Table 10: Impact of global and local part in Diff-Manner Adapter.
Figure 8: Framework of Equal 9 ,Equal 10 in Diff-Manner Adapter.
Figure 9: Effects of Different Confusion Pair in SEM and Representative Sample Number in SAM.
Method
Base
Novel
HM
Random Add 5% Gaussian Noise
79.87
74.49
77.09
Random Add 10% Gaussian Noise
72.41
69.56
70.96
Appendix
Table 11: Effects of Noise Add.
Dataset
Classes
Train
Val
Test
Task
Hand-crafted Prompt
ImageNet
1,000
1.28M
N/A
50,000
General object recognition
“a photo of a [CLASS].”
Caltech101
100
4,128
1,649
2,465
General object recognition
“a photo of a [CLASS].”
EuroSAT
10
13,500
5,400
8,100
Remote sensing classification
“a satellite image of [CLASS].”
SUN397
397
15,880
3,970
19,850
Scene classification
“a photo of a [CLASS].”
DTD
47
2,820
1,128
1,692
Texture classification
“a [CLASS] texture.”
UCF101
101
7,639
1,808
3,783
Human action recognition
“a photo of a person doing [CLASS].”
Appendix
Table 12: Statistics of the datasets used in our experiments
Figure 10: Effects of different s and γ in choosing adaptive α .
Figure 11: Heatmaps of model misclassifications across more datasets prove the fixed confusion pattern.
University of Kurdistan, Sanandaj, Iran · Department of Computer Engineering, University of Kurdistan, Sanandaj, Iran · CHIRP, Child Health Research Centre, The University of Queensland, Brisbane, Australia