Rethinking Post-hoc Debiasing of Large Pretrained Models

Divyaansh Mertia; Chirag Bhutra; Himanshu Shekhar; Shubham Bagwari; Pratik Mazumder

cs.CVOct 21, 2026

Rethinking Post-hoc Debiasing of Large Pretrained Models

Authors: Divyaansh Mertia, Chirag Bhutra, Himanshu Shekhar, Shubham Bagwari, Pratik Mazumder

Organizations: Indian Institute of Technology Jodhpur

Abstract

Large pretrained models such as CLIP are widely used as frozen feature extractors. However, their embeddings inherit spurious correlations from training data, introducing bias into the features. A growing body of research has sought to address this issue through post hoc debiasing, leaving the backbone untouched. We discover that despite differing mechanisms, these methods require the bias feature to be identified and expressed in text. This dependency is not innocuous. In this work, we experimentally demonstrate that this requirement incurs an unforeseen penalty in certain scenarios. For example, on the UrbanCars benchmark, having two co-occurring bias attributes, the recent methods underperform a vanilla classifier on biased frozenbackbone features, and even providing ground-truth bias identities does not close the gap. We show that the pattern that such methods seek is readily observable in the frozen geometry. Specifically, cross-class training pairs placed closest together by the frozen encoder almost always share a spurious attribute, regardless of the bias type, without requiring access to the bias labels. Based on this, we propose a novel approach to identify such pairs and learn a single linear map that transforms the features to perform post-hoc debiasing. The proposed approach achieves competitive performance across all datasets without access to bias attribute annotations or any LLMs.

Figures & tables

Method	Venue	Bias Info	Text-Agnostic
Post-Hoc Debiasing (Frozen Backbone)
FairerCLIP [ 2 ]	ICLR’24	named	✗
RoboShot [ 1 ]	ICLR’24	LLM	✗
PRISM [ 10 ]	ICCV’25	LLM	✗
PRISM-mini [ 10 ]	ICCV’25	LLM	✗
$\text{SEM}_{i}$ [ 3 ]	CVPR’26	none	✗

Table 1: Requirements across debiasing methods. Bias Info denotes the spurious attribute supervision required (per-sample labels, attributes named by the practitioner, LLM proposals, or none ). Text-Agnostic indicates methods that do not depend on a text encoder or prompt generation.

Figure 1: On UrbanCars, most debiasing methods are worse than no debiasing. Worst-group accuracy of each method, measured against a linear classifier trained on the same frozen CLIP features with no debiasing. Bars to the left of the reference mark methods that reduce accuracy below it. The dashed bar denotes PRISM with the ground-truth attribute identities ( Sec. 4.2 ).

Cosine similarity to prompt embeddings
Method	Bias info	Ovr.	W-BG	W-Co	W-Joint
CLIP-cosine	none	50.00	4.03	13.71	0.00
PRISM	LLM	53.43	6.45	16.13	1.61
$+$ oracle	oracle	59.48	33.06	47.58	30.65
PRISM-mini	LLM	53.63	24.19	27.42	6.45
$+$ oracle	oracle	55.65	26.61	20.97	9.68

Table 2: Correct specification is necessary but not sufficient. Replacing LLM-generated attribute descriptions with ground-truth prompts improves PRISM and PRISM-mini, but both remain below the undebiased reference.

Cross-class pair shares…	all pairs	selected
neither shortcut	81.9	5.2
background only	8.6	35.2
co-occurring object only	8.6	12.1
both shortcuts	0.9	47.6
at least one shortcut	18.1	94.8
number of pairs	16.0M	986

Table 3: What a similarity threshold selects. Composition of cross-class training pairs on UrbanCars (%) in the frozen embedding space, that share each shortcut. Selected denotes pairs whose cosine similarity exceeds the threshold used by our method ( Sec. 5 ). Attribute labels are used only for this analysis.

Figure 2: Bias concentration scales with feature similarity. The percentage of cross-class training pairs sharing a spurious background or co-occurring object increases monotonically as a function of the cosine similarity threshold.

Figure 3: Method overview – Left: we select two sets of pairs in the frozen space using class labels and cosine similarity alone. $\mathcal{P}$ takes cross-class pairs above $\tau_{\mathrm{push}}$ , $94.8\%$ of which are bias-sharing, and $\mathcal{Q}$ takes same-class pairs below $\tau_{\mathrm{pull}}$ . Right: a selected pair contributes gradients until it reaches its margin and none after. Thresholds decide which pairs move, margins how far. We only train $\mathbf{W}$ .

Cosine similarity to prompt embeddings
Method	Venue	Ovr.	W-BG	W-Co	W-Joint
CLIP-cos	–	50.00	4.03	13.71	0.00
RoboShot	ICLR’24	51.81	13.71	20.16	0.00
FairerCLIP	ICLR’24	81.25	64.52	69.36	58.06
PRISM	ICCV’25	53.43	6.45	16.13	1.61
PRISM-m	ICCV’25	53.63	24.19	27.42	6.45

Table 4: UrbanCars: two simultaneous shortcuts. Methods are grouped by the rule used to classify, so that comparison is made between methods sharing a decision procedure. Shaded grey rows are the undebiased reference for each rule. Best method: bold , second: underlined . Results are averaged over 3 seeds. For our method, we report the mean and standard deviation. †

\text{SEM}_{i}

transforms only the text embeddings leaving image features unchanged, so a head fit on its features reproduces CLIP-linear exactly.

Cosine similarity to prompt embeddings
			BAR				BFFHQ								Waterbirds ‡
			$1\%$		$5\%$		$0.5\%$		$1\%$		$2\%$		$5\%$		$5\%$
Method	Venue	Bias info	Ovr.	Worst	Ovr.	Worst	Ovr.	Worst	Ovr.	Worst	Ovr.	Worst	Ovr.	Worst	Ovr.	Worst
CLIP-cosine	–	none	96.49	74.65	96.49	74.65	70.70	37.60	70.70	37.60	70.70	37.60	70.70	37.60	86.20	39.60
RoboShot [ 1 ]	ICLR’24	LLM	24.08	0.00	24.08	0.00	47.90	2.80	47.90	2.80	47.90	2.80	47.90	2.80	72.10	41.90
FairerCLIP [ 2 ]	ICLR’24	named	94.03	59.16	82.95	21.13	73.50	38.00	83.40	56.80	94.00	83.60	94.40	83.60	89.12	77.41

Table 5: Results on the BAR, BFFHQ, and Waterbirds datasets. For our method, we report the mean and standard deviation. Note: ‡ For Waterbirds, we follow the evaluation protocol of [ 3 ] , including the ViT-L/14@336px backbone. Further details, including standard deviations, are provided in the supplementary materials.

Objective			UrbanCars
$\mathcal{L}_{\text{push}}$	$\mathcal{L}_{\text{pull}}$	margins	Ovr.	W-BG	W-Co	W-Joint
✓	–	✓	85.48	73.39	70.97	48.39
–	✓	✓	71.77	47.58	49.19	6.45
✓	–	–	87.10	78.23	75.81	58.06
–	✓	–	87.70	78.23	75.81	61.29
✓	✓	–	90.73	86.29	84.68	79.03

Table 6: Ablation on loss terms

Figure 4: Threshold stability on UrbanCars. Each cell reports the mean and spread across the four margin settings at that threshold pair. The red border box marks the selected configuration.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Class	Background	Co‑obj	Prompt
urban car	urban	urban	"A sleek black SUV cruises down a neon‑lit city avenue, passing a bright orange traffic cone near the curb."
urban car	urban	country	"A sleek urban car is parked on a bustling downtown street while a curious cow grazes on a patch of grass beside the curb."
urban car	country	urban	"A glossy urban car is parked on a dusty dirt road beside a bright red fire hydrant, with golden wheat fields stretching behind it."
urban car	country	country	"A sleek urban car is parked on a grassy farm lane, while a curious cow grazes beside a weathered wooden barn in the background."
country car	urban	urban	"A country car is parked on a bustling city street lined with brick buildings, next to a bright red fire hydrant."
country car	urban	country	"A bright red country car is stopped at a busy city intersection, with a lone cow standing on the sidewalk beside it."

Appendix

Table 1: Oracle prompts used for UrbanCars. Each row shows one prompt per combination of class, background, and co‑occurring object. These prompts replace the LLM‑generated descriptions in PRISM.

Dataset	Ratio	Cosine		Linear head
		Ovr.	Worst	Ovr.	Worst
BAR	$1\%$	$97.31_{\pm 0.30}$	$87.79_{\pm 0.66}$	$97.54_{\pm 0.18}$	$83.10_{\pm 1.41}$
BAR	$5\%$	$97.83_{\pm 0.22}$	$87.79_{\pm 0.66}$	$97.89_{\pm 0.00}$	$84.51_{\pm 0.00}$
BFFHQ	$0.5\%$	$95.03_{\pm 0.17}$	$88.00_{\pm 0.57}$	$92.67_{\pm 0.25}$	$84.00_{\pm 0.40}$
BFFHQ	$1\%$	$95.50_{\pm 0.51}$	$90.80_{\pm 0.86}$	$94.30_{\pm 0.17}$	$86.67_{\pm 0.23}$
BFFHQ	$2\%$	$96.60_{\pm 0.08}$	$92.67_{\pm 0.19}$	$95.60_{\pm 0.26}$	$90.67_{\pm 0.46}$

Appendix

Table 2: Seed sweep on BAR, BFFHQ and Waterbirds datasets. Mean and standard deviation over three seeds.

Figure 1: Threshold stability on BAR at $1\%$ conflicting ratio. Each cell reports the mean and spread across the four margin settings at that threshold pair. The red border box marks the selected configuration.

Figure 2: Threshold stability on BAR at $5\%$ conflicting ratio. Each cell reports the mean and spread across the four margin settings at that threshold pair. The red border box marks the selected configuration.

Figure 3: Threshold stability on BFFHQ at $0.5\%$ conflicting ratio. Each cell reports the mean and spread across the four margin settings at that threshold pair. The red border box marks the selected configuration.

Figure 4: Threshold stability on BFFHQ at $1\%$ conflicting ratio. Each cell reports the mean and spread across the four margin settings at that threshold pair. The red border box marks the selected configuration.

Figure 5: Threshold stability on BFFHQ at $2\%$ conflicting ratio. Each cell reports the mean and spread across the four margin settings at that threshold pair. The red border box marks the selected configuration.

Figure 6: Threshold stability on BFFHQ at $5\%$ conflicting ratio. Each cell reports the mean and spread across the four margin settings at that threshold pair. The red border box marks the selected configuration.

Figure 7: Threshold stability on Waterbirds. Each cell reports the mean and spread across the four margin settings at that threshold pair. The red border box marks the selected configuration.

Backbone		Ovr.	W-BG	W-Co	W-Joint
CLIP ViT-L/14	no debiasing	83.87	68.55	65.32	38.71
	$+$ ours	90.93	86.29	85.48	83.87
SigLIP ViT-L/16	no debiasing	87.70	79.03	78.23	62.90
	$+$ ours	92.54	87.90	88.71	87.10
DINOv3 ViT-L/16 †	no debiasing	92.34	86.29	87.90	80.65
	$+$ ours	93.15	89.52	89.52	83.87

Appendix

Table 3: The method transfers across frozen encoders. UrbanCars, with and without our projection, evaluated with the linear head. Accuracy improves on all four metrics for every backbone, by a margin that narrows as the undebiased representation improves. Rows are single runs at a fixed seed. † DINOv3 is a vision-only encoder and produces no text embeddings, so none of the prompt-based methods we compare against can be applied to it and our method does not use the text side.

Class ratio		Ovr.	W-BG	W-Co	W-Joint
CLIP-cosine		50.00	4.03	13.71	0.00
CLIP-linear		83.87	68.55	65.32	38.71
balanced	cosine	89.11	76.61	76.61	66.13
	head	90.93	86.29	85.48	83.87
$70{:}30$	cosine	92.90	88.70	88.70	87.10
	head	92.54	86.29	83.87	80.65

Appendix

Table 4: Robustness to class imbalance on UrbanCars. The training split is resampled so that the two classes appear in the stated ratio, leaving the shortcut correlations unchanged. Worst Joint accuracy never falls below twice the undebiased reference of

38.71

. Both classification rules are reported, and rows are single runs at a fixed seed.

Figure 8: Class separation before and after the projection. Pairwise cosine similarity over UrbanCars training pairs, split by whether the pair shares a class, with the overlap between the two shaded. AUC is the probability that a randomly drawn same-class pair is more similar than a randomly drawn cross-class pair and at

0.5

similarity would carry no information about class membership. Curves are normalized to equal height, as same-class pairs are the more numerous.

Figure 9: Where conflicting samples sit. UrbanCars training embeddings projected onto the plane spanned by the class direction and the background direction, before and after our projection. Filled points are bias-aligned samples and open points are bias-conflicting. Each arrow runs from the aligned mean of a class to its conflicting mean, and the dashed line marks the midpoint between the two class means. Attribute labels are used for this measurement only and never during training.

Debiased ML

Debiased machine learning develops methods to reduce unfair or spurious model behavior and statistical bias in learned estimates.

See all 56 papers

SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Models
Mar 19, 2026Quentin Guimard, Federico Bartsch, Simone Caldarella +3Cultural Bias in VLMs Vision-Language Models
University of Trento · Toyota Motor Europe · Fondazione Bruno Kessler
Selective Test-Time Debiasing for CLIP via Reward Gating
Jul 1, 2026Jaeho Han, Jisoo Yang, Hyeondong Woo +3Debiased ML VLM Adaptation
Department of Artificial Intelligence, Chung-Ang University
Bias Leaves a Gradient Trail: Label-Free Bias Identification via Gradient Probes on Concept Decompositions
May 27, 2026Thomas Vitry, Kieran Edgeworth, Stefan Wermter +1Debiased ML Spurious Correlation Robustness
University of Hamburg, Hamburg, Germany · Ecole Normale Superieure de Rennes, Rennes, France

Mar 19, 2026J/K move · Enter open · S save

SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Models

Quentin Guimard, Federico Bartsch, Simone Caldarella, Rahaf Aljundi+2

University of Trento · Toyota Motor Europe · Fondazione Bruno Kessler

Cultural Bias in VLMs Vision-Language Models

Open paper AskPDF arXiv

Models that bridge vision and language, such as CLIP, are key components of multimodal AI, yet their large-scale, uncurated training data introduce severe social and spurious biases. Existing post-hoc debiasing methods often operate directly in the dense CLIP embedding space, where bias and task-relevant information are highly entangled. This entanglement limits their ability to remove bias without degrading semantic fidelity. In this work, we propose Sparse Embedding Modulation (SEM), a post-hoc, zero-shot debiasing framework that operates in a Sparse Autoencoder (SAE) latent space. By decomposing CLIP text embeddings into disentangled features, SEM identifies and modulates bias-relevant neurons while preserving query-relevant ones. This enables more precise, non-linear interventions. Across four benchmark datasets and two CLIP backbones, SEM achieves substantial fairness gains in retrieval and zero-shot classification. Our results demonstrate that sparse latent representations provide an effective foundation for post-hoc debiasing of vision-language models.

Abstract

Debiased ML

Explore similar work

SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Models

Selective Test-Time Debiasing for CLIP via Reward Gating

Bias Leaves a Gradient Trail: Label-Free Bias Identification via Gradient Probes on Concept Decompositions