SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Models

Quentin Guimard; Federico Bartsch; Simone Caldarella; Rahaf Aljundi; Elisa Ricci; Massimiliano Mancini

cs.CVMar 19, 2026

SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Models

Authors: Quentin Guimard, Federico Bartsch, Simone Caldarella, Rahaf Aljundi, Elisa Ricci, Massimiliano Mancini

Organizations: University of Trento · Toyota Motor Europe · Fondazione Bruno Kessler

Abstract

Models that bridge vision and language, such as CLIP, are key components of multimodal AI, yet their large-scale, uncurated training data introduce severe social and spurious biases. Existing post-hoc debiasing methods often operate directly in the dense CLIP embedding space, where bias and task-relevant information are highly entangled. This entanglement limits their ability to remove bias without degrading semantic fidelity. In this work, we propose Sparse Embedding Modulation (SEM), a post-hoc, zero-shot debiasing framework that operates in a Sparse Autoencoder (SAE) latent space. By decomposing CLIP text embeddings into disentangled features, SEM identifies and modulates bias-relevant neurons while preserving query-relevant ones. This enables more precise, non-linear interventions. Across four benchmark datasets and two CLIP backbones, SEM achieves substantial fairness gains in retrieval and zero-shot classification. Our results demonstrate that sparse latent representations provide an effective foundation for post-hoc debiasing of vision-language models.

Figures & tables

Panel of Figure 1 — Figure 1: SAEs decompose entangled embeddings for precise intervention. (a) Standard methods operate directly on the dense, entangled CLIP embedding space. (b) Our SEM first projects the embedding into a sparse, disentangled latent space via an SAE. This enables a precise intervention on specific features, resolving the limitations of dense-space manipulation.

Figure 2: SAEs Significantly Improve Feature Disentanglement. We plot our Disentanglement Score (higher is better) , which measures a profession probe’s ability to avoid capturing bias. Standard CLIP embeddings (blue) show low disentanglement, while our SAE latent space (orange) consistently increases the score.

Panel of Figure 3 — Figure 3: Overview of the SEM framework. Our method operates in two stages: (a) Scoring: The CLIP embedding is projected into the SAE latent space. Neurons are then scored for content relevance ( Sec. 3.3 ) and bias sensitivity ( Sec. 3.4 ) by comparing their activations to pre-computed prompt sets. (b) Steering: The scores are combined into a modulation coefficient $M$ that attenuates bias neurons and boosts content neurons ( Sec. 3.5 ). The final, debiased embedding is reconstructed from this modulated latent vector.

Figure 4: Visualizing Debiasing on Entangled Concepts. (a) A 2D PCA of original CLIP embeddings for 100 professions. Gender clusters (‘female’, ‘male’) are clearly separated, but the ‘neutral’ and ‘male’ ones incorrectly overlap. (b) Orth-Proj achieves a partial overlap between ‘male’ and ‘female’ clusters, but fails to merge the ‘neutral’ cluster and appears to disrupt the data’s underlying structure. (c) SEM b successfully merges all three clusters (‘male’, ‘female’, and ‘neutral’) into a cohesive distribution with a consistent structure.

Method	Content Preservation $(\uparrow)$	Bias Neutralization $(\uparrow)$
Orth-Proj	0.415	0.916
SEM b	0.878	0.974

Table 1: Quantitative analysis of debiasing methods. Ideally, methods should have high Content Preservation and high Bias Neutralization. Orth-Proj fails at content preservation.

Bias-agnostic + input-specific prompts
Method	FairFace								UTKFace
	ViT-B/16				ViT-L/14@336px				ViT-B/16				ViT-L/14@336px
	Race		Gender		Race		Gender		Race		Gender		Race		Gender
	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$
Base CLIP	0.237	0.795	0.139	0.346	0.244	0.798	0.114	0.326	0.124	0.475	0.134	0.321	0.124	0.461	0.040	0.185
RoboShot	0.327	0.891	0.349	0.508	0.304	0.926	0.324	0.519	0.220	0.681	0.247	0.396	0.236	0.742	0.269	0.467

Table 2: Measuring race and gender bias for Stereotype queries on FairFace and UTKFace. Bold : Best in setting (row group) and better than Base CLIP . Underline : Best in setting, but not improving over Base CLIP . Gray : Method is not zero-shot.

Bias-agnostic + input-specific prompts
Method	CelebA (Gender)						Waterbirds (Background)
	ViT-B/16			ViT-L/14@336px			ViT-B/16			ViT-L/14@336px
	Acc. $(\uparrow)$	WG $(\uparrow)$	Gap $(\downarrow)$	Acc. $(\uparrow)$	WG $(\uparrow)$	Gap $(\downarrow)$	Acc. $(\uparrow)$	WG $(\uparrow)$	Gap $(\downarrow)$	Acc. $(\uparrow)$	WG $(\uparrow)$	Gap $(\downarrow)$
Base CLIP	0.748	0.612	0.136	0.869	0.780	0.090	0.829	0.250	0.579	0.862	0.396	0.466
RoboShot	0.788	0.693	0.095	0.848	0.812	0.036	0.806	0.262	0.545	0.862	0.485	0.377
SEM i	0.736	0.610	0.125	0.791	0.744	0.047	0.801	0.496	0.305	0.832	0.523	0.309

Table 3: Measuring zero-shot classification fairness on CelebA and Waterbirds. Bold : Best in setting (row group) and better than Base CLIP . Underline : Best in setting, but not improving over Base CLIP . Gray : Method is not zero-shot.

SEM i Variants (Bias-Agnostic)
Method Variant	CelebA (Gender)			Waterbirds (Background)
Method Variant	Acc. $(\uparrow)$	WG $(\uparrow)$	Gap $(\downarrow)$	Acc. $(\uparrow)$	WG $(\uparrow)$	Gap $(\downarrow)$
SEM i (Full)	0.791	0.745	0.046	0.832	0.523	0.309
– $M(j)=1$	0.729	0.640	0.089	0.872	0.357	0.515
– median CLIP	0.687	0.558	0.129	0.879	0.400	0.479
SEM b Variants (Bias-Aware)

Table 4: Ablation study on zero-shot classification tasks (ViT-L/14@336px). Our full methods ( SEM i and SEM b ) provide the most robust, balanced performance. Bold : Best in setting.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Accountant	Actor	Architect	Astronaut	Audiologist	Author	Baker	Barber
Biologist	Blacksmith	Bricklayer	Bus driver	Butcher	Carpenter	Chef	Chemist
Civil engineer	Cleaner	Clerk	Coach	Comedian	Computer programmer	Construction worker	Consultant
Counselor	Dancer	Dentist	Designer	Detective	Dietitian	DJ	Doctor
Driver	Economist	Editor	Electrician	Engineer	Entrepreneur	Farmer	Firefighter
Florist	Graphic designer	Hairdresser	Historian	Interpreter	Journalist	Judge	Lawyer
Librarian	Magician	Makeup artist	Manager	Marine biologist	Mathematician	Mechanic	Model

Appendix

Table 5: Complete list of 100 professions used in both the disentanglement and qualitative studies.

Templates with {bias} and {profession} placeholders
A photo of a {bias} {profession}.	A {bias} {profession} at work.
An image of a {bias} {profession}.	An illustration of a {bias} {profession}.
A {bias} {profession}.	A studio shot of a {bias} {profession}.
A portrait of a {bias} {profession}.	A {bias} professional who works as a {profession}.
This is a {bias} {profession}.	A close-up of a {bias} {profession}.
Here is a {bias} {profession}.	A {bias} {profession} on the job.

Appendix

Table 6: Prompt templates used for the disentanglement study.

Method	ViT-B/16						ViT-L/14@336px
Method	Race			Gender			Race			Gender
	$acc_{p}$ $(\uparrow)$	$acc_{b}$ $(\uparrow)$	$acc_{b\leftarrow p}$ $(\downarrow)$	$acc_{p}$ $(\uparrow)$	$acc_{b}$ $(\uparrow)$	$acc_{b\leftarrow p}$ $(\downarrow)$	$acc_{p}$ $(\uparrow)$	$acc_{b}$ $(\uparrow)$	$acc_{b\leftarrow p}$ $(\downarrow)$	$acc_{p}$ $(\uparrow)$	$acc_{b}$ $(\uparrow)$	$acc_{b\leftarrow p}$ $(\downarrow)$
Base CLIP	1.000	1.000	0.957	1.000	1.000	0.923	1.000	1.000	0.949	1.000	1.000	0.852
SAE	0.996	1.000	0.755	0.995	0.997	0.800	0.994	0.998	0.710	0.993	0.996	0.748

Appendix

Table 7: Full Probing Results. Mean accuracies for profession prediction (

acc_{p}

), direct bias prediction (

acc_{b}

), and sequential entanglement probe (

acc_{b\leftarrow p}

) across Race and Gender settings. Lower entanglement (

acc_{b\leftarrow p}

) indicates better disentanglement.

Prompt
A firefighter in full gear holding a water hose.
A musician playing a guitar on a dimly lit stage.
A group of puppies tumbling and playing together.
A modern skyscraper made of glass and steel.
A golden retriever fetching a stick in a park.
A panoramic skyline of a modern city at night.

Appendix

Table 8: Examples of diverse prompts used to establish a baseline activation distribution.

Bias-agnostic + input-specific prompts
Method	ViT-B/16					ViT-L/14@336px
	Stereotype		Hair Color			Stereotype		Hair Color
	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$	Prec. $(\uparrow)$	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$	Prec. $(\uparrow)$
Base CLIP	0.314	0.555	0.179	0.409	0.629	0.237	0.536	0.148	0.359	0.622
RoboShot	0.189	0.355	0.144	0.244	0.633	0.195	0.394	0.276	0.429	0.675
SEM i	0.173	0.443	0.191	0.345	0.678	0.153	0.413	0.237	0.458	0.698

Appendix

Table 9: Measuring gender bias for Stereotype and Hair Color queries on CelebA. Bold : Best in setting (row group) and better than Base CLIP . Underline : Best in setting, but not improving over Base CLIP . Gray : Method is not zero-shot.

SEM i Variants (Bias-Agnostic)
Method Variant	FairFace								UTKFace
	ViT-B/16				ViT-L/14@336px				ViT-B/16				ViT-L/14@336px
	Race		Gender		Race		Gender		Race		Gender		Race		Gender
	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$
SEM i (Full)	0.170	0.691	0.088	0.269	0.147	0.625	0.123	0.328	0.096	0.407	0.065	0.245	0.059	0.442	0.032	0.185
– $M(j)=1$	0.139	0.659	0.078	0.243	0.124	0.573	0.093	0.288	0.075	0.397	0.088	0.278	0.061	0.368	0.038	0.196

Appendix

Table 10: Extended ablation study for retrieval on FairFace and UTKFace. Bold : Best in setting.

SEM i Variants (Bias-Agnostic)
Method Variant	CelebA (Gender)						Waterbirds (Background)
	ViT-B/16			ViT-L/14@336px			ViT-B/16			ViT-L/14@336px
	Acc. $(\uparrow)$	WG $(\uparrow)$	Gap $(\downarrow)$	Acc. $(\uparrow)$	WG $(\uparrow)$	Gap $(\downarrow)$	Acc. $(\uparrow)$	WG $(\uparrow)$	Gap $(\downarrow)$	Acc. $(\uparrow)$	WG $(\uparrow)$	Gap $(\downarrow)$
SEM i (Full)	0.736	0.611	0.125	0.791	0.745	0.046	0.801	0.498	0.303	0.832	0.523	0.309
– $M(j)=1$	0.734	0.609	0.125	0.729	0.640	0.089	0.834	0.210	0.624	0.872	0.357	0.515
– median CLIP	0.728	0.601	0.127	0.687	0.558	0.129	0.840	0.563	0.277	0.879	0.400	0.479

Appendix

Table 11: Extended ablation study for zero-shot classification on CelebA and Waterbirds. Bold : Best in setting.

SEM i Variants (Bias-Agnostic)
Method Variant	ViT-B/16					ViT-L/14@336px
	Stereotype		Hair Color			Stereotype		Hair Color
	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$	Prec. $(\uparrow)$	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$	Prec. $(\uparrow)$
SEM i (Full)	0.173	0.443	0.191	0.344	0.679	0.153	0.413	0.236	0.458	0.698
– $M(j)=1$	0.185	0.456	0.193	0.371	0.672	0.195	0.490	0.274	0.484	0.709
– median CLIP	0.250	0.503	0.181	0.382	0.689	0.155	0.411	0.299	0.500	0.708

Appendix

Table 12: Extended ablation study for retrieval on CelebA. Bold : Best in setting.

Bias-agnostic + input-specific prompts
Method	FairFace								UTKFace
	ResNet-50				ResNet-101				ResNet-50				ResNet-101
	Race		Gender		Race		Gender		Race		Gender		Race		Gender
	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$
Base CLIP	0.215	0.735	0.170	0.351	0.203	0.744	0.144	0.335	0.127	0.477	0.153	0.340	0.152	0.496	0.136	0.333
RoboShot	0.215	0.706	0.299	0.445	0.222	0.798	0.338	0.494	0.152	0.586	0.258	0.414	0.206	0.652	0.323	0.492

Appendix

Table 13: Measuring race and gender bias for Stereotype queries on FairFace and UTKFace (ResNet backbones). Bold : Best in setting (row group) and better than Base CLIP . Underline : Best in setting, but not improving over Base CLIP . Gray : Method is not zero-shot.

Bias-agnostic + input-specific prompts
Method	CelebA (Gender)						Waterbirds (Background)
	ResNet-50			ResNet-101			ResNet-50			ResNet-101
	Acc. $(\uparrow)$	WG $(\uparrow)$	Gap $(\downarrow)$	Acc. $(\uparrow)$	WG $(\uparrow)$	Gap $(\downarrow)$	Acc. $(\uparrow)$	WG $(\uparrow)$	Gap $(\downarrow)$	Acc. $(\uparrow)$	WG $(\uparrow)$	Gap $(\downarrow)$
Base CLIP	0.820	0.768	0.053	0.689	0.502	0.188	0.837	0.394	0.442	0.801	0.499	0.301
RoboShot	0.841	0.806	0.035	0.737	0.596	0.140	0.762	0.458	0.304	0.761	0.450	0.310
SEM i	0.835	0.799	0.036	0.811	0.758	0.052	0.851	0.557	0.295	0.843	0.581	0.262

Appendix

Table 14: Measuring zero-shot classification fairness on CelebA and Waterbirds (ResNet Backbones). Bold : Best in setting (row group) and better than Base CLIP . Underline : Best in setting, but not improving over Base CLIP . Gray : Method is not zero-shot.

Bias-agnostic + input-specific prompts
Method	ResNet-50					ResNet-101
	Stereotype		Hair Color			Stereotype		Hair Color
	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$	Prec. $(\uparrow)$	KL $(\downarrow)$	MS $(\downarrow)$	KL $(\downarrow)$	MS $(\downarrow)$	Prec. $(\uparrow)$
Base CLIP	0.389	0.622	0.187	0.367	0.735	0.300	0.560	0.205	0.414	0.718
RoboShot	0.190	0.337	0.364	0.550	0.762	0.294	0.454	0.274	0.459	0.723
SEM i	0.050	0.193	0.246	0.369	0.508	0.041	0.185	0.301	0.508	0.688

Appendix

Table 15: Measuring gender bias for Stereotype and Hair Color queries on CelebA (ResNet Backbones). Bold : Best in setting (row group) and better than Base CLIP . Underline : Best in setting, but not improving over Base CLIP . Gray : Method is not zero-shot.

Cultural Bias in VLMs

See all 14 papers

Interpretable Debiasing of Vision-Language Models for Social Fairness
Feb 27, 2026Na Min An, Yoonna Jang, Yusuke Hirota +3Cultural Bias in VLMs VLM Interpretability
KAIST AI · University of Copenhagen · NVIDIA
Selective Test-Time Debiasing for CLIP via Reward Gating
Jul 1, 2026Jaeho Han, Jisoo Yang, Hyeondong Woo +3Debiased ML VLM Adaptation
Department of Artificial Intelligence, Chung-Ang University
A Closed-Form Solution for Debiasing Vision-Language Models with Utility Guarantees Across Modalities and Tasks
Mar 13, 2026Tangzheng Lian, Guanyu Hu, Yijing Ren +2Cultural Bias in VLMs Algorithmic Fairness
King’s College London · Queen Mary University of London

Feb 27, 2026J/K move · Enter open · S save

Interpretable Debiasing of Vision-Language Models for Social Fairness

Na Min An, Yoonna Jang, Yusuke Hirota, Ryo Hachiuma+2

KAIST AI · University of Copenhagen · NVIDIA

Cultural Bias in VLMs VLM Interpretability

Open paper AskPDF arXiv

The rapid advancement of Vision-Language models (VLMs) has raised growing concerns that their black-box reasoning processes could lead to unintended forms of social bias. Current debiasing approaches focus on mitigating surface-level bias signals through post-hoc learning or test-time algorithms, while leaving the internal dynamics of the model largely unexplored. In this work, we introduce an interpretable, model-agnostic bias mitigation framework, DeBiasLens, that localizes social attribute neurons in VLMs through sparse autoencoders (SAEs) applied to multimodal encoders. Building upon the disentanglement ability of SAEs, we train them on facial image or caption datasets without corresponding social attribute labels to uncover neurons highly responsive to specific demographics, including those that are underrepresented. By selectively deactivating the social neurons most strongly tied to bias for each group, we effectively mitigate socially biased behaviors of VLMs without degrading their semantic knowledge. Our research lays the groundwork for future auditing tools, prioritizing social fairness in emerging real-world AI systems.

Abstract

Cultural Bias in VLMs

Explore similar work

Interpretable Debiasing of Vision-Language Models for Social Fairness

Selective Test-Time Debiasing for CLIP via Reward Gating

A Closed-Form Solution for Debiasing Vision-Language Models with Utility Guarantees Across Modalities and Tasks