M-PFT: Angular Margin for Calibrated Prompt-Tuning of Fine-Tuned Vision-Language Models

Shihab Aaqil Ahamed; Mohamed Ansaff; Nang Khin Myat Lay Phyu; Udaya Sampath K. Perera Miriya Thanthrige; Chamira U. S. Edussooriya; Ranga Rodrigo; Muhammad Haris Khan

cs.CVOct 29, 2026

M-PFT: Angular Margin for Calibrated Prompt-Tuning of Fine-Tuned Vision-Language Models

Authors: Shihab Aaqil Ahamed, Mohamed Ansaff, Nang Khin Myat Lay Phyu, Udaya Sampath K. Perera Miriya Thanthrige, Chamira U. S. Edussooriya, Ranga Rodrigo, Muhammad Haris Khan

Organizations: Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Dept. of Electronic and Telecommunication Engineering, University of Moratuwa, Sri Lanka · Albukhary International University, Alor Setar, Malaysia

Abstract

Prompt-tuning has emerged as a parameter-efficient technique for adapting large-scale vision-language models (VLMs) to downstream tasks while keeping the pre-trained weights frozen. However, prompt-tuned VLMs often suffer from a base-to-novel dual miscalibration problem, where models typically show underconfidence on base classes and overconfidence on novel classes. Current VLM calibration approaches address this dual miscalibration either separately or jointly, but lack a geometric understanding of the hyperspherical embedding space and thus fail to exploit angular discriminative properties to adjust distorted decision boundaries. To address this, we propose M-PFT, a novel angular-margin approach for prompt tuning that introduces two complementary objectives to promote class discriminability and semantic anchoring. This enlarges the angular margin by 1) minimizing the maximum cosine similarity between tunable class embeddings to reduce embedding collapse and 2) minimizing the angular distance between each tunable embedding and its corresponding zero-shot embedding to preserve semantic geometry. Extensive experiments across 9 prompt-tuning methods with various backbones on 15 diverse datasets show that our approach reduces the average calibration error on novel classes, while remaining competitive on base classes and maintaining comparable accuracy. Further analyses reveal that M-PFT effectively adjusts distorted decision boundaries induced by prompt tuning while preserving the pre-trained embedding space, improving calibration with negligible computational overhead.

Figures & tables

Panel of Figure 1 — Figure 1: Comparison of calibration scores with TCPT [ 39 ] and DAC [ 45 ] on 10 fine-grained classification datasets with CoOp [ 52 ] for base (left) and novel (right) classes. Each axis reports $1-\frac{\text{ECE}}{\text{ECE}_{\text{worst}}}$ (percentage of the worst method’s ECE removed; larger is better). Ours maximizes the covered area (a higher score indicates a larger footprint), indicating better calibration and effective addressing of the dual miscalibration problem.

Figure 2: Hyperspherical visualization of the dual miscalibration problem.

\mathcal{L}_{\text{CE}}

distorts decision boundaries by inducing embedding collapse

\theta_{jk}\downarrow

and semantic drift

\theta_{j}\uparrow

\mathcal{L}_{\text{min-max}}

maximizes

\theta_{jk}

to mitigate collapse and improve class discriminability, while

\mathcal{L}_{\text{anchor}}

minimizes the drift

\theta_{j}

by anchoring tunable embeddings to zero-shot embeddings to preserve semantic geometry. This dual design creates complementary objectives that act synergistically to enlarge the angular margin

\theta_{jk}-\theta_{j}

, yielding well-separated, discriminative decision boundaries while preserving pre-trained semantics and addressing the dual miscalibration.

Panel of Figure 3 — Figure 3: Analyses of dual miscalibration of prompt-tuned VLMs. Base classes show underconfidence mitigated by our complementary objective, whereas novel classes show overconfidence alleviated by the proposed method. The cross-dataset cosine similarity analyses reveal a negative correlation between angular margin ( $\theta_{jk}-\theta_{j}$ ) and ECE, where enlarging the margin lowers calibration error. This shows prompt tuning distorts decision boundaries with embedding collapse ( $\theta_{jk}\downarrow$ ) and semantic drift ( $\theta_{j}\uparrow$ ), resulting in base-to-novel dual miscalibration. These conflicting trends motivate the angular margin framework’s class discriminability and semantic anchoring objectives.

Class	Zero-Shot	Calibration	Model	Metric	CoOp	CoCoOp	KgCoOp	MaPLe	ProDA	ProGrad	PromptSRC	HiCroPL	MMRL
Class	Zero-Shot	Calibration	Model	Metric	CoOp	CoCoOp	KgCoOp	MaPLe	ProDA	ProGrad	PromptSRC	HiCroPL	MMRL	Average
Base	69.50 Acc $\uparrow$ 3.58 ECE $\downarrow$ CLIP	Baseline	CLIP	Acc. $\uparrow$	81.54	80.57	81.72	82.41	78.24	76.61	84.72	86.20	86.37	82.04
			CLIP	ECE $\downarrow$	6.82	4.52	6.37	3.29	4.02	5.15	4.26	3.81	2.66	4.54
			SigLIP	Acc. $\uparrow$	85.28	85.39	85.96	86.11	85.82	84.07	88.61	88.90	88.51	86.52
			SigLIP	ECE $\downarrow$	2.54	2.51	3.33	2.56	4.20	4.12	2.11	1.82	1.60	2.75
		TCPT	CLIP	Acc. $\uparrow$	83.19	80.16	81.34	82.75	78.51	76.98	85.31	86.66	86.39	82.37

Table 1: Comparison of calibration performance on base and novel classes averaged across 10 fine-grained classification datasets for top@1 Accuracy (Acc.) and Expected Calibration Error (ECE) with multiple prompt-tuning methods across various calibration baselines, where higher Acc. and lower ECE indicate better performance and calibration, respectively

Class	Metric	Zero-Shot	Baseline	TCPT	M-PFT
Class	Metric	Zero-Shot	Baseline	TCPT	(Ours)
Base	Acc. $\uparrow$	60.52	70.19	70.04	71.20
Base	ECE $\downarrow$	3.92	3.00	2.73	2.03
Novel	Acc. $\uparrow$	60.52	68.95	68.77	69.92
Novel	ECE $\downarrow$	4.03	2.90	2.55	1.95

Table 2: Calibration performance under natural distribution shifts.

Class	Backbone	Metric	Baseline	TCPT	M-PFT
Class	Backbone	Metric	Baseline	TCPT	(Ours)
Base	ResNet-50	Acc. $\uparrow$	77.66	78.57	78.14
	ResNet-50	ECE $\downarrow$	4.04	3.46	3.78
	ViT-B/32	Acc. $\uparrow$	80.56	80.90	80.16
	ViT-B/32	ECE $\downarrow$	3.15	2.87	2.98
Novel	ResNet-50	Acc. $\uparrow$	59.57	60.68	65.66

Table 3: Calibration performance with different backbones.

Loss Component	Base		Novel
Loss Component	Acc. $\uparrow$	ECE $\downarrow$	Acc. $\uparrow$	ECE $\downarrow$
$\mathcal{L}_{\text{CE}}$	82.41	3.29	75.64	6.18
$\mathcal{L}_{\text{CE}}+\mathcal{L}_{\text{min-max}}$	81.74	2.49	72.06	6.80
$\mathcal{L}_{\text{CE}}+\mathcal{L}_{\text{anchor}}$	82.64	3.25	75.36	4.28
$\mathcal{L}_{\text{CE}}+\mathcal{L}_{\text{min-max}}+\mathcal{L}_{\text{anchor}}$	82.43	3.12	75.88	4.11
(w/o gradient blocking)	82.43	3.12	75.88	4.11

Table 4: Loss component analyses on the effect of min-max regularization

\mathcal{L}_{\text{min-max}}

and anchoring regularization

\mathcal{L}_{\text{anchor}}

with (w/) and without (w/o) gradient blocking on both base and novel classes.

Figure 4: Cosine similarity comparision between tunable class embeddings. With gradient blocking w.r.t. zero-shot embeddings, cosine similarity is lower, corresponding to increased $\theta_{jk}\uparrow$ , showing higher class discriminability.

Class	Metric	Standard deviations (Std.))		Anchor-based Methods
		Baseline	M-PFT	Baseline		M-PFT (Ours)
		Baseline	(Ours)	CLIP	SigLIP	CLIP	SigLIP
Base	Acc. $\uparrow$	$\pm$ 0.36	$\pm$ 0.35	81.02	86.21	81.43	85.95
Base	ECE $\downarrow$	$\pm$ 0.29	$\pm$ 0.26	5.26	3.17	4.85	2.91
Novel	Acc. $\uparrow$	$\pm$ 0.51	$\pm$ 0.52	74.63	80.14	74.72	80.99
Novel	ECE $\downarrow$	$\pm$ 0.47	$\pm$ 0.42	4.33	3.49	3.66	3.57

Table 5: Standard deviation and Anchor-based method analyses.

Panel of Figure 5 — Figure 5: Varying shots.

Panel of Figure 6 — Figure 6: Different prompt initialization.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Class Counts			Train Samples
Dataset	Base	Novel	Total	Base (16-shot)
Generic Objects
Caltech101	50	50	100	800
ImageNet	500	500	1,000	8,000
Fine-Grained
OxfordPets	19	18	37	304

Appendix

Table 1: The split strategy of downstream datasets categorized by domain. Prompt tuning utilizes few-shot

(16)

training samples only from the base classes (

\mathcal{C}_{\text{base}}

), while evaluated across separate base and novel (

\mathcal{C}_{\text{novel}}

) test splits.

Methods	BS	LR	CTX
CoOp	32	0.002	16
CoCoOp	1	0.002	4
ProDA	4	0.001	16
KgCoOp	32	0.002	16
MaPLe	4	0.0026	2
ProGrad	32	0.002	16

Appendix

Table 2: Hyperparameters for prompt tuning methods. “BS” denotes the batch size. “LR” denotes the learning rate. “CTX” is the context length of the learnable prompt.

Method	Calibration	Metric	DTD	Flowers102	Food101	SUN397	Aircrafts	OxfordPets	Caltech101	UCF101	EuroSAT	Stanford Cars	Average
Baseline	Zero-Shot	Acc. $\uparrow$	53.00	71.80	90.10	69.40	27.70	91.30	97.20	71.00	57.00	63.60	69.21
Baseline	Zero-Shot	ECE $\downarrow$	4.53	3.11	1.57	1.59	3.03	2.25	6.49	3.24	8.35	3.74	3.79
CoOp	Baseline	Acc. $\uparrow$	76.27	90.00	90.20	81.14	35.23	94.77	97.98	83.32	90.24	76.22	81.54
	Baseline	ECE $\downarrow$	12.17	4.93	3.66	8.11	25.70	1.00	0.66	6.44	1.75	3.73	6.82
	TCPT	Acc. $\uparrow$	80.44	97.21	90.38	81.68	38.62	94.95	98.06	84.68	88.56	77.32	83.19
	TCPT	ECE $\downarrow$	2.42	4.80	0.30	1.22	4.96	1.94	1.01	1.11	4.90	7.10	2.98

Appendix

Table 3: Comparison of calibration performance on base classes across 10 fine-grained classification datasets using CLIP backbone for top@1 Accuracy (Acc.) and Expected Calibration Error (ECE) with multiple prompt-tuning methods across various calibration baselines, where higher Acc and lower ECE indicate better performance and calibration, respectively

Method	Calibration	Metric	DTD	Flowers102	Food101	SUN397	Aircrafts	OxfordPets	Caltech101	UCF101	EuroSAT	Stanford Cars	Average
Baseline	Zero-Shot	Acc. $\uparrow$	60.60	77.50	91.10	75.50	35.90	97.10	94.10	78.60	63.80	75.00	74.92
Baseline	Zero-Shot	ECE $\downarrow$	6.86	4.91	1.83	3.48	6.55	3.42	1.55	5.52	9.12	3.31	4.65
CoOp	Baseline	Acc. $\uparrow$	52.98	69.90	90.57	70.77	24.79	96.49	94.18	62.83	64.68	65.29	69.25
	Baseline	ECE $\downarrow$	20.82	12.13	3.03	13.70	30.44	1.67	2.16	18.74	11.88	11.73	12.63
	DAC	ECE $\downarrow$	10.48	10.19	1.78	4.05	17.38	1.82	3.17	8.67	8.62	5.16	7.13
	TCPT	Acc. $\uparrow$	48.07	71.63	90.84	70.03	30.83	97.36	93.56	66.49	56.70	69.49	69.50

Appendix

Table 16

Method	Calibration	Metric	DTD	Flowers102	Food101	SUN397	Aircrafts	OxfordPets	Caltech101	UCF101	EuroSAT	Stanford Cars	Average
HiCroPL	Baseline	Acc. $\uparrow$	65.14	76.15	91.88	78.16	39.47	97.49	95.52	82.03	74.45	74.92	77.52
	Baseline	ECE $\downarrow$	5.33	5.34	0.87	1.14	7.76	2.72	1.01	2.03	8.81	1.90	3.69
	DAC	ECE $\downarrow$	4.16	6.04	1.33	1.41	7.68	2.18	1.10	3.15	8.24	2.26	3.75
	TCPT	Acc. $\uparrow$	66.51	76.10	91.79	77.37	38.89	97.69	95.20	82.39	74.74	75.21	77.59
	TCPT	ECE $\downarrow$	4.95	7.53	0.50	2.07	7.99	1.49	1.12	1.73	10.83	2.11	4.03
		Acc. $\uparrow$	67.79	76.50	91.80	78.15	38.33	97.54	95.74	82.28	74.60	75.35	77.81

Appendix

Table 4: Comparison of calibration performance on novel classes across 10 fine-grained classification datasets using CLIP backbone for top@1 Accuracy (Acc.) and Expected Calibration Error (ECE) with multiple prompt-tuning methods across various calibration baselines, where higher Acc and lower ECE indicate better performance and calibration, respectively

Method	Calibration	Metric	DTD	Flowers102	Food101	SUN397	Aircrafts	OxfordPets	Caltech101	UCF101	EuroSAT	Stanford Cars	Average
Baseline	Zero-Shot	Acc. $\uparrow$	76.16	91.17	92.28	75.24	29.05	96.49	98.52	75.75	57.05	85.01	77.67
Baseline	Zero-Shot	ECE $\downarrow$	3.80	5.07	0.61	3.81	5.81	2.07	1.00	4.76	18.42	1.07	4.64
CoOp	Baseline	Acc. $\uparrow$	88.39	98.86	92.42	82.13	34.19	95.78	98.97	83.61	90.20	88.22	85.28
	Baseline	ECE $\downarrow$	2.36	2.02	1.45	0.92	4.38	1.79	1.25	2.26	4.97	3.99	2.54
	TCPT	Acc. $\uparrow$	88.62	99.05	92.48	82.12	35.20	95.94	98.95	83.63	90.14	88.67	85.48
	TCPT	ECE $\downarrow$	2.16	2.26	1.61	0.88	4.77	1.91	1.29	2.61	6.19	4.25	2.79

Appendix

Table 18

Method	Calibration	Metric	DTD	Flowers102	Food101	SUN397	Aircrafts	OxfordPets	Caltech101	UCF101	EuroSAT	Stanford Cars	Average
ProGrad	Baseline	Acc. $\uparrow$	85.73	96.96	92.84	81.63	39.10	96.69	98.79	82.69	78.16	88.11	84.07
	Baseline	ECE $\downarrow$	9.38	6.29	1.70	3.04	5.87	2.44	1.21	3.02	4.23	3.99	4.12
	TCPT	Acc. $\uparrow$	86.92	97.57	92.89	82.10	38.46	96.46	98.95	83.18	81.02	88.09	84.56
	TCPT	ECE $\downarrow$	9.14	5.05	0.74	2.20	6.45	1.61	0.74	2.43	4.49	3.31	3.62
		Acc. $\uparrow$	85.45	96.90	92.94	81.20	36.51	96.71	98.77	81.92	78.37	87.54	83.63
	M-PFT (Ours)	ECE $\downarrow$	9.34	6.33	1.08	1.66	4.50	1.90	0.78	2.27	4.27	2.95	3.51

Appendix

Table 5: Comparison of calibration performance on base classes across 10 fine-grained classification datasets using SigLIP backbone for top@1 Accuracy (Acc.) and Expected Calibration Error (ECE) with multiple prompt-tuning methods, where higher Acc and lower ECE indicate better performance and calibration, respectively.

Method	Calibration	Metric	DTD	Flowers102	Food101	SUN397	Aircrafts	OxfordPets	Caltech101	UCF101	EuroSAT	Stanford Cars	Average
Baseline	Zero-Shot	Acc. $\uparrow$	74.88	85.67	93.59	78.57	40.85	98.15	97.82	83.78	67.21	96.21	81.67
Baseline	Zero-Shot	ECE $\downarrow$	3.10	3.98	0.96	2.78	12.00	0.97	0.83	2.76	4.11	0.69	3.22
CoOp	Baseline	Acc. $\uparrow$	67.47	81.04	91.48	77.49	30.90	97.80	97.05	75.12	48.78	94.46	76.16
	Baseline	ECE $\downarrow$	13.28	8.48	0.98	3.82	17.82	1.24	1.14	3.77	16.90	1.93	6.94
	TCPT	Acc. $\uparrow$	67.75	81.09	91.85	77.46	33.44	97.50	97.71	74.94	42.16	94.26	75.82
	TCPT	ECE $\downarrow$	12.85	8.68	1.14	3.08	14.50	0.97	1.13	3.63	25.63	1.90	7.35

Appendix

Table 20

Method	Calibration	Metric	DTD	Flowers102	Food101	SUN397	Aircrafts	OxfordPets	Caltech101	UCF101	EuroSAT	Stanford Cars	Average
ProDA	Baseline	Acc. $\uparrow$	70.25	82.84	91.99	80.52	39.27	97.85	97.24	77.82	49.94	95.86	78.36
	Baseline	ECE $\downarrow$	7.91	6.34	1.63	1.68	8.48	1.45	0.79	2.17	13.87	1.95	4.63
	TCPT	Acc. $\uparrow$	74.76	83.78	93.33	80.67	40.89	98.30	98.07	81.68	46.88	96.31	79.47
	TCPT	ECE $\downarrow$	3.32	5.76	0.85	0.76	5.18	1.15	0.78	2.46	21.39	1.30	4.30
		Acc. $\uparrow$	70.49	83.60	92.98	80.91	39.33	98.21	98.00	77.47	49.79	96.23	78.70
	M-PFT (Ours)	ECE $\downarrow$	7.16	5.54	1.14	0.81	8.60	1.25	0.93	2.57	12.58	1.24	4.18

Appendix

Table 6: Comparison of calibration performance on novel classes across 10 fine-grained classification datasets using SigLIP backbone for top@1 Accuracy (Acc.) and Expected Calibration Error (ECE) with multiple prompt-tuning methods, where higher Acc and lower ECE indicate better performance and calibration, respectively.

Class	Calibration	Metric	ImageNet-1k	ImageNet-V2	ImageNet-S	ImageNet-A	ImageNet-R	Average
Base	Zero-Shot	Acc. $\uparrow$	72.40	60.87	46.09	47.81	75.41	60.52
	Zero-Shot	ECE $\downarrow$	1.51	2.43	4.88	8.34	2.42	3.92
	Baseline	Acc. $\uparrow$	76.71	67.35	53.35	68.31	85.22	70.19
	Baseline	ECE $\downarrow$	2.27	3.14	3.94	2.52	3.14	3.00
	TCPT	Acc. $\uparrow$	76.72	67.19	53.15	67.86	85.28	70.04
	TCPT	ECE $\downarrow$	2.39	3.09	3.91	2.21	2.05	2.73

Appendix

Table 7: Comparison of calibration performance on both base and novel classes across 4 natural distribution shift datasets for top@1 Accuracy (Acc.) and Expected Calibration Error (ECE) with MaPLe and various calibration baselines, where higher Acc and lower ECE indicate better performance and calibration, respectively.

Method	Calibration	Metric	DTD	Flowers102	Food101	SUN397	Aircrafts	OxfordPets	Caltech101	UCF101	EuroSAT	Stanford Cars	Average
MaPLe	Baseline	MCE $\downarrow$	0.70	0.71	0.46	0.39	0.89	0.51	0.62	0.71	1.96	0.86	0.78
	Baseline	ACE $\downarrow$	1.96	1.31	1.37	1.35	3.04	1.44	1.15	2.13	7.68	3.31	2.47
	TCPT	MCE $\downarrow$	0.51	0.29	0.12	0.28	0.81	0.18	0.21	0.55	1.84	0.55	0.53
	TCPT	ACE $\downarrow$	1.36	0.71	0.34	0.80	2.68	0.46	0.34	1.60	5.98	1.48	1.57
		MCE $\downarrow$	0.64	1.16	0.27	0.25	0.69	0.57	0.34	0.60	1.20	0.60	0.63
	M-PFT (Ours)	ACE $\downarrow$	2.29	2.14	0.80	1.09	2.41	0.99	0.53	1.92	3.83	2.08	1.81

Appendix

Table 8: Comparison of calibration performance on base classes across 10 fine-grained classification datasets using SigLIP backbone for Maximum Calibration Error (MCE) and Adaptive Calibration Error (ACE) with multiple prompt-tuning methods across various calibration baselines, where lower MCE and ACE indicate better performance and calibration.

Method	Calibration	Metric	DTD	Flowers102	Food101	SUN397	Aircrafts	OxfordPets	Caltech101	UCF101	EuroSAT	Stanford Cars	Average
MaPLe	Baseline	MCE $\downarrow$	2.50	1.50	0.48	0.21	2.3	0.50	0.38	0.75	6.72	0.48	1.58
	Baseline	ACE $\downarrow$	7.00	6.23	1.14	0.46	10.03	0.61	1.03	2.68	14.89	1.53	4.56
	TCPT	MCE $\downarrow$	1.57	1.46	0.16	0.38	1.68	0.31	0.28	0.73	1.76	0.25	0.86
	TCPT	ACE $\downarrow$	6.23	6.09	0.37	1.54	7.19	0.38	0.53	2.04	7.11	1.12	3.26
		MCE $\downarrow$	1.65	0.85	0.41	0.30	2.28	0.50	0.22	0.83	5.28	0.29	1.26
	M-PFT (Ours)	ACE $\downarrow$	6.08	3.70	0.81	0.97	9.06	0.65	0.35	2.66	9.06	1.04	3.44

Appendix

Table 9: Comparison of calibration performance on novel classes across 10 fine-grained classification datasets using the SigLIP backbone for Maximum Calibration Error (MCE) and Adaptive Calibration Error (ACE), with multiple prompt-tuning methods and across various calibration baselines, where lower MCE and ACE indicate better calibration performance.

Class	Loss Component	Metric	DTD	Flowers102	Food101	SUN397	Aircrafts	OxfordPets	Caltech101	UCF101	EuroSAT	Standford Cars	Average
Base	$\mathcal{L}_{\text{CE}}$	Acc. $\uparrow$	79.63	96.00	90.80	80.55	36.33	95.53	97.97	83.20	91.13	72.93	82.41
	$\mathcal{L}_{\text{CE}}$	ECE $\downarrow$	4.18	4.28	0.78	1.27	4.77	2.68	1.54	2.68	3.42	7.25	3.29
	$\mathcal{L}_{\text{CE}}+\mathcal{L}_{\text{min-max}}$	Acc. $\uparrow$	78.47	96.01	90.07	80.12	33.13	95.64	97.61	83.00	90.67	72.70	81.74
	$\mathcal{L}_{\text{CE}}+\mathcal{L}_{\text{min-max}}$	ECE $\downarrow$	3.50	3.14	1.49	2.09	3.32	1.68	0.67	1.65	3.76	3.60	2.49
	$\mathcal{L}_{\text{CE}}+\mathcal{L}_{\text{min-max}}+\mathcal{L}_{\text{anchor}}$	Acc. $\uparrow$	79.55	96.07	90.42	80.62	36.81	95.30	97.55	84.25	91.55	72.17	82.43
	(w/o gradient blocking)	ECE $\downarrow$	2.15	4.50	0.80	1.54	3.73	2.38	1.77	2.91	6.83	4.54	3.12

Appendix

Table 10: Loss component ablation on the effect of min-max regularization

\mathcal{L}_{\text{min-max}}

and anchoring regularization

\mathcal{L}_{\text{anchor}}

with (w/) and without (w/o) gradient blocking on both base and novel classes.

Method	Calibration	Metric	DTD	Flowers102	Food101	SUN397	Aircrafts	OxfordPets	Caltech101	UCF101	EuroSAT	Stanford Cars	Average
CoOp-ResNet50	Baseline	Acc.	74.85	95.25	82.54	76.37	29.95	90.5	95.22	80.4	81.32	70.22	77.66
	Baseline	ECE	8.65	4.22	1.15	2.81	2.62	2.32	1.27	1.82	9.34	6.15	4.04
	TCPT	Acc.	75.42	95.22	82.77	77.07	29.01	91.08	95.44	79.96	89.79	69.90	78.57
	TCPT	ECE	5.87	4.74	0.84	1.23	3.32	2.21	1.37	2.21	4.12	8.65	3.46
		Acc.	75.12	94.02	84.18	77.41	28.45	91.94	95.16	79.25	87.29	68.55	78.14
	M-PFT (Ours)	ECE	2.57	5.19	1.21	3.03	3.89	3.09	2.29	3.64	5.43	7.42	3.78

Appendix

Table 11: Comparison of calibration performance on base classes across 10 fine-grained classification tasks using ResNet-50 and ViT-B/32. We report top-1 accuracy (Acc.) and Expected Calibration Error (ECE) for CoOp [ 52 ] , a prompt-tuning technique, evaluated with different backbones.

Method	Calibration	Metric	DTD	Flowers102	Food101	SUN397	Aircrafts	OxfordPets	Caltech101	UCF101	EuroSAT	Stanford Cars	Average
CoOp-ResNet50	Baseline	Acc.	41.67	61.87	82.55	64.46	18.72	92.11	87.27	55.07	34.15	57.84	59.57
	Baseline	ECE	5.47	10.24	0.86	9.07	18.75	2.10	3.57	13.47	22.18	8.08	9.38
	TCPT	Acc.	39.05	61.47	82.50	64.94	21.66	93.08	87.52	53.94	42.75	59.88	60.68
	TCPT	ECE	8.83	4.13	1.72	4.53	5.36	2.68	2.82	10.16	10.93	3.43	5.46
		Acc.	46.46	71.07	84.68	70.71	21.72	94.93	91.38	65.66	45.76	64.22	65.66
	M-PFT (Ours)	ECE	10.91	4.61	1.98	1.1	10.41	3.63	1.56	3.77	9.26	2.34	4.96

Appendix

Table 12: Comparison of calibration performance on novel classes across 10 fine-grained classification tasks using ResNet-50 and ViT-B/32. We report top-1 accuracy (Acc.) and Expected Calibration Error (ECE) for CoOp [ 52 ] , a prompt-tuning technique, evaluated with different backbones.

Table 28

Method	Calibration	Metric	DTD	Flowers102	Food101	Aircrafts	OxfordPets	Caltech101	UCF101	EuroSAT	Stanford Cars	Average
Base	Baseline	Std. Acc	0.44	0.18	0.14	0.78	0.27	0.11	0.32	0.96	0.42	0.36
	Baseline	Std. ECE	0.24	0.08	0.20	0.60	0.20	0.17	0.55	0.64	0.19	0.29
		Std. Acc	0.45	0.18	0.14	0.79	0.28	0.12	0.25	0.84	0.43	0.35
	M-PFT (Ours)	Std. ECE	0.19	0.08	0.20	0.46	0.22	0.17	0.56	0.53	0.18	0.26
Novel	Baseline	Std. Acc	1.33	0.24	0.05	1.61	0.27	0.16	0.47	0.50	0.46	0.51
	Baseline	Std. ECE	0.12	0.58	0.01	2.08	0.15	0.23	0.39	1.03	0.15	0.47

Appendix

Table 15: Standard deviation (Std.) across

3

random seeds for base and novel classes across

10

fine-grained classification benchmarks.

VLM Calibration

Vision-language model calibration makes confidence scores better reflect when multimodal predictions are correct and grounded in visual evidence.

See all 84 papers

Feb 22, 2026

Towards Calibrating Prompt Tuning of Vision-Language Models

Prompt tuning of large-scale vision-language models such as CLIP enables efficient task adaptation without updating model weights. However, it often leads to poor confidence calibration and unreliable predictive uncertainty. We address this problem by proposing a calibration framework that enhances predictive reliability while preserving the geometry of the pretrained CLIP embedding space, which is required for robust generalization. Our approach extends the standard cross-entropy loss with two complementary regularizers: (1) a mean-variance margin penalty that stabilizes inter-class logit margins by maximizing their average while minimizing dispersion, mitigating underconfidence and overconfidence spikes; and (2) a text moment-matching loss that aligns the first and second moments of tuned text embeddings with their frozen CLIP counterparts, preserving semantic dispersion crucial for generalization. Through extensive experiments across 7 prompt-tuning methods and 11 diverse datasets, we demonstrate that our approach significantly reduces the Expected Calibration Error (ECE) compared to competitive calibration techniques on both base and novel classes

Ashshak Sharifdeen, Fahad Shamshad, Muhammad Akhtar Munir +6

Mohamed bin Zayed University of AI · University of Colombo · Michigan State University

VLM Calibration Prompt Learning for VLMs Confidence Estimation for VLMs

Ask PDF

Apr 30, 2026

Improving Calibration in Test-Time Prompt Tuning for Vision-Language Models via Data-Free Flatness-Aware Prompt Pretraining

Test-time prompt tuning (TPT) has emerged as a promising technique for enhancing the adaptability of vision-language models by optimizing textual prompts using unlabeled test data. However, prior studies have observed that TPT often produces poorly calibrated models, raising concerns about the reliability of their predictions. Recent works address this issue by incorporating additional regularization terms that constrain model outputs, which improve calibration but often degrade performance. In this work, we reveal that these regularization strategies implicitly encourage optimization toward flatter minima, and that the sharpness of the loss landscape around adapted prompts is a key factor governing calibration quality. Motivated by this observation, we introduce Flatness-aware Prompt Pretraining (FPP), a simple yet effective pretraining framework for TPT that initializes prompts within flatter regions of the loss landscape prior to adaptation. We show that simply replacing the initialization in existing TPT pipelines--without modifying any other components--is sufficient to improve both calibration and performance. Notably, FPP requires no labeled data and incurs no additional computational costs during test-time tuning, making it highly practical for real-world deployment. The code is available at: https://github.com/YonseiML/fpp.

Hyeonseo Jang, Jaebyeong Jeon, Joong-Won Hwang +1

1Yonsei University · 2ETRI

Deep Learning Optimization Vision-Language Models VLM Adaptation VLM Calibration Test-Time Prompt Tuning

Ask PDF

Mar 9, 2026

FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models

CLIP-based prompt tuning enables pretrained Vision-Language Models (VLMs) to efficiently adapt to downstream tasks. Although existing studies have made significant progress, they pay limited attention to changes in the internal attention representations of VLMs during the tuning process. In this paper, we attribute the failure modes of prompt tuning predictions to shifts in foreground attention of the visual encoder, and propose Foreground View-Guided Prompt Tuning (FVG-PT), an adaptive plug-and-play foreground attention guidance module, to alleviate the shifts. Concretely, FVG-PT introduces a learnable Foreground Reliability Gate to automatically enhance the foreground view quality, applies a Foreground Distillation Compensation module to guide visual attention toward the foreground, and further introduces a Prior Calibration module to mitigate generalization degradation caused by excessive focus on the foreground. Experiments on multiple backbone models and datasets show the effectiveness and compatibility of FVG-PT. Codes are available at: https://github.com/JREion/FVG-PT

Haoyang Li, Liang Wang, Siyu Zhou +5

University of Technology Sydney, NSW 2007, Australia · Shanghai University, Shanghai 200444, China

Attention-Guided Distillation VLM Adaptation Prompt Learning for VLMs Prompt Tuning Attention Mechanisms

Ask PDF

Abstract

VLM Calibration

Explore similar work

Towards Calibrating Prompt Tuning of Vision-Language Models

Improving Calibration in Test-Time Prompt Tuning for Vision-Language Models via Data-Free Flatness-Aware Prompt Pretraining

FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models