Fairness-Aware Test-Time Prompt Tuning

Yoann Launay; Parameswaran Kamalaruban; Tom Kempton; Stuart Burrell; David Sutton

cs.LGAug 26, 2026

Fairness-Aware Test-Time Prompt Tuning

Authors: Yoann Launay, Parameswaran Kamalaruban, Tom Kempton, Stuart Burrell, David Sutton

Organizations: University of Cambridge · Risk and Security AI Lab, Visa Inc. · University of Manchester

Abstract

Vision-language models have displayed remarkable capabilities in multi-modal understanding and are increasingly used in critical applications where economic and practical deployment constraints prohibit re-training or fine-tuning. However, these models can also exhibit systematic biases that disproportionately affect protected demographic groups and existing approaches to addressing these biases require extensive model retraining and access to demographic attributes. There is a clear need to develop test-time adaptation (TTA) approaches that improve the fairness characteristics of pretrained models under distributional shift. In this paper, we evaluate how episodic TTA affects fairness in CLIP classification under subpopulation shifts and develop FairTPT, a novel fairness-aware episodic TTA method that jointly minimizes target marginal entropy while maximizing spurious marginal entropy through soft-prompt tuning. We find that standard episodic TTA generally exacerbates disparities between majority and minority groups, that blinding a model to spurious attributes without degrading target performance is inherently challenging, and that excessive blinding can lead to catastrophic forgetting. This model collapse can be prevented by monitoring test-time changes in target loss within the linear regime, while still achieving fairness improvements on reactive data and preserving overall performance. FairTPT outperforms all state-of-the-art episodic test-time debiasing methods and establishes a foundation for robust TTA, which is essential for achieving fairness in practice.

Figures & tables

Figure 1: FairTPT pipeline using a single unlabeled image at a time (black frame). The soft prompt ${\bm{t}}_{\textnormal{ctx}}$ is the only parameter tuned at test time, requiring text processor to be re-written. Pink and grey blocks have frozen weights; gradients are computed only for pink blocks. This example illustrates the objective in Eq. ( 3 ). For the alternative objective in Eq. ( 4 ), the spurious prompts (orange) and their corresponding soft prompts would also include target labels, resulting in four spurious inputs in this example.

Method	FairFace				CelebA								WaterBirds				UTKFace								Average Results
	$\text{Gender}\times\text{Race}$				$\text{Hair color}\times\text{Gender}$				$\text{Smiling}\times\text{Gender}$				$\text{Type}\times\text{Background}$				$\text{Age}\times\text{Race}$				$\text{Gender}\times\text{Race}$
	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD
Zero-Shot	95.7	90.0	5.7	9.4	86.4	67.8	18.6	22.3	75.8	53.7	22.1	8.0	83.8	40.2	43.7	25.0	80.3	45.7	34.5	23.2	97.1	90.1	7.0	9.1	86.5	64.6	21.9	16.2
Episodic test-time adaptation methods
TPT	93.6	85.0	8.6	12.4	92.0	42.6	49.4	37.6	59.4	20.0	39.4	5.0	84.0	40.8	43.1	37.1	87.5	57.7	29.8	26.3	94.3	86.4	7.9	8.0	85.1	55.4	29.7	21.1
Zero	91.2	78.6	12.6	11.9	90.4	56.3	34.0	23.8	69.9	38.0	31.9	11.1	83.1	40.0	43.2	29.0	86.2	53.8	32.4	28.7	93.5	77.8	15.7	13.4	85.7	57.4	28.3	19.6

Table 1: Overall (Accuracy) and subgroup-level performance (Worst-Group Accuracy, Bias, and Equalized Odds Difference) evaluation of all the methods considered (ours and baselines). We report mean over 5 random seeds and mean aggregation over all equally sized datasets. Best results are in bold , second best underlined . The values that improve upon Zero-Shot by more than

2.0

percentage points are highlighted in green , while degradations greater than

2.0

percentage points are shown in red (arbitrary threshold). We set

\lambda_{\textnormal{fair}}=100

and

\lambda_{\textnormal{fair (mo)}}=100

for FairTPT and FairTPT (MO) , respectively. Hyperparameter values of all methods are presented in Table 3 .

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Explicit debiasing	Unsupervised	Episodic	Difference from our setup
OrthCali [ 11 ]	✓	✓	✓	—
SEraser [ 30 ]	✓	✓	✓	—
BendVLM [ 13 ]	✓	$\times$	$\times$	Requires a reference dataset of images with spurious labels.
RoboSHOT [ 1 ]	$\times$	✓	✓	Spurious attributes suggested by an LLM.
TIE [ 29 ]	✓	$\times$	$\times$	Supervised with respect to the spurious attribute for each batch.
TIE ∗ [ 29 ]	✓	✓	$\times$	Scale coefficient computed as the batch average.

Appendix

Table 2: Comparison of state-of-the-art test-time debiasing methods for VLMs [ 11 , 30 , 13 , 1 , 29 , 4 , 23 ] . Explicit debiasing refers to debiasing concepts chosen by the user or predefined policies. Unsupervised indicates access only to unlabeled images at test time. Episodic denotes processing a single image at a time, as opposed to batch processing in an online setting. Given the absence of differences from our setup, OrthCali serves as our fairness baseline.

P^{*}~\leftarrow~\argmin_{P}\left\lVert P-P_{0}\right\rVert^{2}+\frac{\lambda_{\textnormal{orth}}}{\left|{\mathcal{D}}\right|}\sum_{({\bm{z}}_{i},{\bm{z}}_{j})\in{\mathcal{D}}}\left\lVert P{\bm{z}}_{i}-P{\bm{z}}_{j}\right\rVert^{2},

Appendix

Algorithm 3 Fairness-Aware Test-Time Adaptive classification with VLMs

Method	$A$	$\rho$	Optimizer	$\eta_{\textnormal{init}}$	$\eta$	$\beta$	$n_{\textnormal{epochs}}$	$\lambda$
TPT	$64$	$0.1$	AdamW		$5e-3$		$1$
Zero	$64$	$0.1$
OrthCali								$\lambda_{\textnormal{orth}}=1000$
FairTPT	$64$	$0.75$	SGD	Any	$\eta_{\textnormal{ELRA}}$	0.01	1	$\lambda_{\textnormal{fair}}=100$
FairTPT (MO)	$64$	$0.75$	UPGrad	Any	$\eta_{\textnormal{ELRA}}$	0.01	1	$\lambda_{\textnormal{fair (mo)}}=100$
FairTPT w/o ELRA	$64$	$0.75$	AdamW		$5e-3$		1	$\lambda_{\textnormal{fair}}=100$

Appendix

Table 3: Hyperparameters of all the methods evaluated in our experiments.

Dataset	Loss Term	Initial Prompt Template
FairFace [ 20 ]	$\ell_{\textnormal{ent}}(x,\left\{{{\bm{t}}_{\textnormal{ctx}};{\mathcal{Y}}}\right\})$	A photo of a [T-CLS] person.
	$\ell_{\textnormal{ent}}(x,\left\{{{\bm{t}}_{\textnormal{ctx}};{\mathcal{S}}}\right\})$	A photo of a [S-CLS] person.
	$\ell_{\textnormal{ent}}(x,\left\{{{\bm{t}}_{\textnormal{ctx}},{\bm{t}}_{y};{\mathcal{S}}}\right\})$	A photo of a [S-CLS] [T-CLS] person.
CelebA [ 27 ]	$\ell_{\textnormal{ent}}(x,\left\{{{\bm{t}}_{\textnormal{ctx}};{\mathcal{Y}}}\right\})$	A photo of a celebrity [T-CLS].
	$\ell_{\textnormal{ent}}(x,\left\{{{\bm{t}}_{\textnormal{ctx}};{\mathcal{S}}}\right\})$	A photo of a [S-CLS] celebrity.
	$\ell_{\textnormal{ent}}(x,\left\{{{\bm{t}}_{\textnormal{ctx}},{\bm{t}}_{y};{\mathcal{S}}}\right\})$	A photo of a [S-CLS] celebrity [T-CLS].

Appendix

Table 4: Initial prompt templates for each dataset. The placeholders [T-CLS] and [S-CLS] denote the target and spurious attributes, respectively, as defined in Table 5 .

Dataset	Target Attribute ( [T-CLS] String)	Spurious Attribute ( [S-CLS] String)
FairFace	Gender ( male , female )	Race ( White , Southeast Asian , Middle Eastern ,
		Black , Indian , Latino Hispanic , East Asian )
CelebA	Hair color ( with dark hair , with blond hair )	Gender ( male , female )
	Smile ( smiling , not smiling )
WaterBirds	Bird type ( of a water bird , of a land bird )	Background ( on water , on land )
UTKFace	Age ( young , old )	Race ( White , Black , Asian , Indian )

Appendix

Table 5: Placeholder values for the prompt templates shown in Table 4 .

Method	FairFace				CelebA								WaterBirds				UTKFace								Average Results
	$\text{Gender}\times\text{Race}$				$\text{Hair color}\times\text{Gender}$				$\text{Smiling}\times\text{Gender}$				$\text{Type}\times\text{Background}$				$\text{Age}\times\text{Race}$				$\text{Gender}\times\text{Race}$
	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD
Zero Shot	95.7	90.0	5.7	9.4	86.4	67.8	18.6	22.3	75.8	53.7	22.1	8.0	83.8	40.2	43.7	25.0	80.3	45.7	34.5	23.2	97.1	90.1	7.0	9.1	86.5	64.6	21.9	16.2
$\eta=1e-5$
$\rho=0.1$	95.7	90.5	5.2	8.9	86.5	67.8	18.8	22.3	75.5	52.9	22.6	8.3	83.8	40.2	43.7	25.0	80.3	45.6	34.6	23.3	97.1	90.3	6.8	8.9	86.5	64.5	21.9	16.1
$\rho=0.25$	95.7	90.3	5.4	9.1	86.5	67.8	18.8	22.3	75.5	52.9	22.6	8.3	83.9	40.2	43.7	25.2	80.3	45.6	34.6	23.3	97.1	90.3	6.8	8.9	86.5	64.5	22.0	16.2

Appendix

Table 6: Hyperparameter benchmark of TPT. We report mean over 5 random seeds and mean aggregation over all equally sized datasets. Best results are in bold , second best underlined . The values that improve upon Zero-Shot by more than 2.0 percentage points are highlighted in green , while degradations greater than 2.0 percentage points are shown in red .

Method	FairFace				CelebA								WaterBirds				UTKFace								Average Results
	$\text{Gender}\times\text{Race}$				$\text{Hair color}\times\text{Gender}$				$\text{Smiling}\times\text{Gender}$				$\text{Type}\times\text{Background}$				$\text{Age}\times\text{Race}$				$\text{Gender}\times\text{Race}$
	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD
Zero Shot	95.7	90.0	5.7	9.4	86.4	67.8	18.6	22.3	75.8	53.7	22.1	8.0	83.8	40.2	43.7	25.0	80.3	45.7	34.5	23.2	97.1	90.1	7.0	9.1	86.5	64.6	21.9	16.2
$\rho=0.1$	91.2	78.6	12.6	11.9	90.4	56.3	34.0	23.8	69.9	38.0	31.9	11.1	83.1	40.0	43.2	29.0	86.2	53.8	32.4	28.7	93.5	77.8	15.7	13.4	85.7	57.4	28.3	19.6
$\rho=0.25$	93.4	83.3	10.1	11.4	90.4	56.3	34.1	27.1	68.7	34.3	34.4	12.6	83.9	41.9	42.0	31.0	86.6	56.0	30.5	30.3	94.8	81.4	13.4	12.9	86.3	58.9	27.4	20.9
$\rho=0.5$	94.1	86.2	7.8	9.3	90.3	56.3	34.0	28.0	68.2	33.1	35.1	13.2	84.1	42.1	42.0	31.1	87.0	58.3	28.7	28.1	94.9	80.2	14.7	14.2	86.4	59.4	27.1	20.6

Appendix

Table 7: Hyperparameter benchmark of Zero . We report mean over 5 random seeds and mean aggregation over all equally sized datasets. Best results are in bold , second best underlined . The values that improve upon Zero-Shot by more than 2.0 percentage points are highlighted in green , while degradations greater than 2.0 percentage points are shown in red .

Method	FairFace				CelebA								WaterBirds				UTKFace								Average Results
	$\text{Gender}\times\text{Race}$				$\text{Hair color}\times\text{Gender}$				$\text{Smiling}\times\text{Gender}$				$\text{Type}\times\text{Background}$				$\text{Age}\times\text{Race}$				$\text{Gender}\times\text{Race}$
	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD
Zero Shot	95.7	90.0	5.7	9.4	86.4	67.8	18.6	22.3	75.8	53.7	22.1	8.0	83.8	40.2	43.7	25.0	80.3	45.7	34.5	23.2	97.1	90.1	7.0	9.1	86.5	64.6	21.9	16.2
$\lambda_{\textnormal{orth}}=0$	96.2	90.9	5.3	8.5	86.2	70.7	15.5	22.7	71.8	40.1	31.7	24.4	85.5	55.7	29.9	4.3	81.1	45.2	36.0	18.9	96.9	89.6	7.3	9.4	86.3	65.4	20.9	14.7
$\lambda_{\textnormal{orth}}=1$	96.2	90.9	5.3	8.5	86.2	70.7	15.5	22.7	71.7	39.9	31.8	24.5	86.0	58.5	27.5	3.6	81.1	45.2	36.0	18.2	96.9	89.6	7.3	9.4	86.4	65.8	20.6	14.5
$\lambda_{\textnormal{orth}}=10$	96.1	90.6	5.5	8.8	86.1	70.7	15.4	22.5	71.5	38.3	33.2	27.6	86.5	63.1	23.4	8.5	81.1	45.2	36.0	18.6	97.0	89.6	7.3	9.4	86.4	66.3	20.1	15.9

Appendix

Table 8: Hyperparameter benchmark of OrthCali . We report mean over 5 random seeds and mean aggregation over all equally sized datasets. Best results are in bold , second best underlined . The values that improve upon Zero-Shot by more than 2.0 percentage points are highlighted in green , while degradations greater than 2.0 percentage points are shown in red .

Method	FairFace				CelebA								WaterBirds				UTKFace								Average Results
	$\text{Gender}\times\text{Race}$				$\text{Hair color}\times\text{Gender}$				$\text{Smiling}\times\text{Gender}$				$\text{Type}\times\text{Background}$				$\text{Age}\times\text{Race}$				$\text{Gender}\times\text{Race}$
	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD
Zero Shot	95.7	90.0	5.7	9.4	86.4	67.8	18.6	22.3	75.8	53.7	22.1	8.0	83.8	40.2	43.7	25.0	80.3	45.7	34.5	23.2	97.1	90.1	7.0	9.1	86.5	64.6	21.9	16.2
FairTPT
$\lambda_{\textnormal{fair}}=1$	95.5	90.5	5.0	8.8	86.2	67.7	18.5	22.0	75.3	52.6	22.7	8.1	83.9	41.0	42.8	26.2	80.3	46.5	33.8	22.5	97.0	90.6	6.5	8.7	86.4	64.8	21.6	16.1
$\lambda_{\textnormal{fair}}=100$	95.5	90.7	4.8	8.1	85.6	67.7	17.9	21.9	75.9	56.1	19.8	6.2	83.2	42.4	40.8	23.5	81.0	51.5	29.5	23.4	96.7	90.6	6.1	8.6	86.3	66.5	19.8	15.3

Appendix

Table 9: Overall and subgroup-level performance evaluation of FairTPT and FairTPT (MO) (with and without ELRA). We report mean over 5 random seeds and mean aggregation over all equally sized datasets. Best results are in bold , second best underlined . The values that improve upon Zero-Shot by more than 2.0 percentage points are highlighted in green , while degradations greater than 2.0 percentage points are shown in red .

Method	FairFace				CelebA								WaterBirds				UTKFace								Average Results
	$\text{Gender}\times\text{Race}$				$\text{Hair color}\times\text{Gender}$				$\text{Smiling}\times\text{Gender}$				$\text{Type}\times\text{Background}$				$\text{Age}\times\text{Race}$				$\text{Gender}\times\text{Race}$
	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD
Zero Shot	95.7	90.0	5.7	9.4	86.4	67.8	18.6	22.3	75.8	53.7	22.1	8.0	83.8	40.2	43.7	25.0	80.3	45.7	34.5	23.2	97.1	90.1	7.0	9.1	86.5	64.6	21.9	16.2
FairTPT and FairTPT (MO) with S loss
$\lambda_{\textnormal{fair}}=1$	95.5	90.5	5.0	8.8	86.2	67.7	18.5	22.0	75.3	52.6	22.7	8.1	83.9	41.0	42.8	26.2	80.3	46.5	33.8	22.5	97.0	90.6	6.5	8.7	86.4	64.8	21.6	16.1
$\lambda_{\textnormal{fair}}=100$	95.5	90.7	4.8	8.1	85.6	67.7	17.9	21.9	75.9	56.1	19.8	6.2	83.2	42.4	40.8	23.5	81.0	51.5	29.5	23.4	96.7	90.6	6.1	8.6	86.3	66.5	19.8	15.3

Appendix

Table 10: Impact of the loss on the experiment. We report mean over 5 data random seeds and mean aggregation over all equally sized datasets. Best results are in bold , second best underlined . The values that improve upon Zero-Shot by more than 2.0 percentage points are highlighted in green , while degradations greater than 2.0 percentage points are shown in red .

Method	FairFace				CelebA								WaterBirds				UTKFace								Average Results
	$\text{Gender}\times\text{Race}$				$\text{Hair color}\times\text{Gender}$				$\text{Smiling}\times\text{Gender}$				$\text{Type}\times\text{Background}$				$\text{Age}\times\text{Race}$				$\text{Gender}\times\text{Race}$
	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD
Zero-Shot	95.7	90.0	5.7	9.4	86.4	67.8	18.6	22.3	75.8	53.7	22.1	8.0	83.8	40.2	43.7	25.0	80.3	45.7	34.5	23.2	97.1	90.1	7.0	9.1	86.5	64.6	21.9	16.2
Episodic test-time debiasing methods
OrthCali	95.9	87.8	8.1	11.3	85.1	71.6	13.5	22.0	71.4	36.8	34.6	30.6	83.3	62.0	21.4	16.2	79.8	47.0	32.8	14.4	96.8	88.9	7.9	10.1	85.4	65.7	19.7	17.4
Our methods

Appendix

Table 11: Overall and subgroup-level performance evaluation of FairTPT followed by the orthogonal projection of OrthCali [ 11 ] ; we set

\lambda_{\textnormal{fair}}=100

and

\lambda_{\textnormal{orth}}=0

. The spurious embeddings used for projection are those obtained after FairTPT . We report mean over 5 random seeds and mean aggregation over all equally sized datasets. Best results are in bold , second best underlined . The values that improve upon Zero-Shot by more than 2.0 percentage points are highlighted in green , while degradations greater than 2.0 percentage points are shown in red .

Method	FairFace												CelebA								UTKFace								Average Results
	$\text{Age}\times\text{Race}$				$\text{Age}\times\text{Gender}$				$\text{Race}\times\text{Gender}$				$\text{Makeup}\times\text{Gender}$				$\text{Glasses}\times\text{Gender}$				$\text{Age}\times\text{Gender}$				$\text{Race}\times\text{Gender}$
	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD
Zero Shot	83.3	40.1	43.2	34.2	83.3	52.8	30.5	8.0	57.6	2.5	55.1	7.1	66.1	26.7	39.4	38.6	94.5	78.8	15.7	7.9	80.3	45.3	35.0	12.3	62.3	14.3	47.9	14.0	75.3	37.2	38.1	17.4
Episodic test-time adaptation methods
TPT	86.6	54.9	31.7	27.8	86.6	61.3	25.3	12.7	64.9	27.3	37.6	6.5	51.3	13.9	37.4	26.6	95.0	23.9	71.1	14.4	87.5	60.7	26.8	20.2	73.9	42.9	31.0	10.5	78.0	40.7	37.3	17.0
TPT with ELRA	83.5	41.8	41.7	33.5	83.5	53.5	30.0	8.4	57.8	2.7	55.1	7.2	65.8	26.3	39.5	38.1	95.0	75.0	20.0	6.3	80.3	45.2	35.1	12.6	62.5	14.6	47.9	14.3	75.5	37.0	38.5	17.2

Appendix

Table 12: Overall and subgroup-level performance evaluation of all considered methods on additional dataset-attribute configurations. We report mean over 5 random seeds and mean aggregation over all equally sized datasets. Best results are in bold , second best underlined . The values that improve upon Zero-Shot by more than 2.0 percentage points are highlighted in green , while degradations greater than 2.0 percentage points are shown in red .

Dataset	Before Update	FairTPT	FairTPT (MO)	TPT
FairFace (Gender $\times$ Race)	42.1	53.9	57.4	39.5
CelebA (Hair color $\times$ Gender)	15.3	41.4	46.4	14.3
CelebA (Smiling $\times$ Gender)	15.3	40.1	50.9	11.9
WaterBirds (Type $\times$ Background)	56.0	70.9	71.1	51.2
UTKFace (Age $\times$ Race)	45.4	53.3	56.1	35.3
UTKFace (Gender $\times$ Race)	45.4	55.6	57.2	29.9

Appendix

Table 13: ASI evaluation across datasets. Mean over 5 seeds and aggregated results.

Dataset	Before Update	FairTPT	FairTPT (MO)	TPT
FairFace (Gender $\times$ Race)	66.4	65.0	64.4	88.4
CelebA (Hair color $\times$ Gender)	57.5	53.9	55.1	96.0
CelebA (Smiling $\times$ Gender)	45.2	44.2	43.4	80.2
WaterBirds (Type $\times$ Background)	62.1	59.7	60.7	80.0
UTKFace (Age $\times$ Race)	46.1	45.2	44.8	85.0
UTKFace (Gender $\times$ Race)	71.6	70.6	69.3	91.4

Appendix

Table 14: ATC evaluation across datasets. Mean over 5 seeds and aggregated results.

Method	TPT	Zero	FairTPT	FairTPT (MO)
Runtime (sec)	0.43	0.04	1.03	1.71

Appendix

Table 15: Inference-time runtime per image (sec) on FairFace (Gender

\times

Race).

Method	FairFace				CelebA								WaterBirds				UTKFace								Average Results
	$\text{Gender}\times\text{Race}$				$\text{Hair color}\times\text{Gender}$				$\text{Smiling}\times\text{Gender}$				$\text{Type}\times\text{Background}$				$\text{Age}\times\text{Race}$				$\text{Gender}\times\text{Race}$
	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD	A	WGA	B	EOD
Zero-Shot	93.3	82.0	11.4	17.3	80.3	72.1	8.2	17.6	85.4	75.8	9.6	6.6	73.6	50.9	22.7	40.1	81.5	62.6	18.9	32.6	95.3	85.0	10.3	12.5	84.9	71.4	13.5	21.1
Episodic test-time adaptation methods
TPT	93.1	84.8	8.3	13.9	77.5	69.9	7.6	9.5	86.7	79.2	7.5	3.8	71.2	44.1	27.0	48.8	61.9	33.6	28.2	20.5	95.4	88.8	6.6	9.0	81.0	66.8	14.2	17.6
Zero	88.5	74.3	14.2	19.2	80.8	72.4	8.3	13.9	83.7	71.6	12.1	5.6	75.3	54.2	21.1	41.5	72.8	52.9	19.9	20.3	92.5	85.7	6.8	11.6	82.3	68.5	13.7	18.7

Appendix

Table 16: Overall (Accuracy) and subgroup-level performance (Worst-Group Accuracy, Bias, and Equalized Odds Difference) evaluation of all the methods considered (ours and baselines) with CLIP-ViT-B/32 as base model. We report mean over 5 random seeds and mean aggregation over all equally sized datasets. Best results are in bold , second best underlined . The values that improve upon Zero-Shot by more than

2.0

percentage points are highlighted in green , while degradations greater than

2.0

percentage points are shown in red (arbitrary threshold). We set

\lambda_{\textnormal{fair}}=100

and

\lambda_{\textnormal{fair (mo)}}=100

for FairTPT and FairTPT (MO) , respectively. Hyperparameter values of all methods are presented in Table 3 .

Test-Time Adaptation of VLMs

Test-time adaptation of vision-language models uses unlabeled deployment data to improve predictions under domain shifts without retraining.

See all 79 papers

Selective Test-Time Debiasing for CLIP via Reward Gating
Jul 1, 2026Jaeho Han, Jisoo Yang, Hyeondong Woo +3Debiased ML VLM Adaptation
Department of Artificial Intelligence, Chung-Ang University
SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models
Jun 5, 2026Sunoh Kim, Daeho UmVLM Robustness VLM Adaptation
Dankook University, Yongin, South Korea · University of Seoul, Seoul, South Korea
USE: A Unified Self-Ensembling Framework for Test-Time Prompt Tuning
Jul 4, 2026Siru Jiang, Jian Liang, Ran He +1Ensemble Learning VLM Adaptation
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences +1

Jul 1, 2026J/K move · Enter open · S save

Selective Test-Time Debiasing for CLIP via Reward Gating

Jaeho Han, Jisoo Yang, Hyeondong Woo, Mingyu Jeon+2

Department of Artificial Intelligence, Chung-Ang University

Debiased ML VLM Adaptation Test-Time Adaptation of VLMs Cultural Bias in VLMs

Open paper AskPDF arXiv

Vision language models (VLMs) demonstrate strong zero-shot performance, but often perpetuate social stereotypes in person-centric queries, yielding skewed demographic distributions. Current debiasing methods apply uniform bias corrections across all input queries regardless of their bias sensitivity, creating a fundamental fairness--utility trade-off. Strong debiasing distorts semantically meaningful information in bias-insensitive queries, while weak debiasing fails to mitigate stereotypes in bias-sensitive ones. This one-size-fits-all approach hampers simultaneously achieving high utility on bias-insensitive queries and fairness on bias-sensitive queries. We introduce Reward-Gated Test-Time Adaptation (RG-TTA), a reinforcement learning-based test-time adaptation framework that selectively applies debiasing based on input sensitivity. RG-TTA adaptively triggers fairness regularization based on the bias sensitivity of each input during test-time policy adaptation, while focusing exclusively on optimizing cross-modal alignment for bias-insensitive inputs. Experiments on fairness benchmarks (e.g., FairFace, UTKFace) demonstrate substantial bias reduction while simultaneously improving zero-shot utility, resolving the trade-off of uniform debiasing.

Abstract

Test-Time Adaptation of VLMs

Explore similar work

Selective Test-Time Debiasing for CLIP via Reward Gating

SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models

USE: A Unified Self-Ensembling Framework for Test-Time Prompt Tuning