Interpretable Debiasing of Vision-Language Models for Social Fairness

Na Min An; Yoonna Jang; Yusuke Hirota; Ryo Hachiuma; Isabelle Augenstein; Hyunjung Shim

cs.CVFeb 27, 2026

Interpretable Debiasing of Vision-Language Models for Social Fairness

Authors: Na Min An, Yoonna Jang, Yusuke Hirota, Ryo Hachiuma, Isabelle Augenstein, Hyunjung Shim

Organizations: KAIST AI · University of Copenhagen · NVIDIA

Abstract

The rapid advancement of Vision-Language models (VLMs) has raised growing concerns that their black-box reasoning processes could lead to unintended forms of social bias. Current debiasing approaches focus on mitigating surface-level bias signals through post-hoc learning or test-time algorithms, while leaving the internal dynamics of the model largely unexplored. In this work, we introduce an interpretable, model-agnostic bias mitigation framework, DeBiasLens, that localizes social attribute neurons in VLMs through sparse autoencoders (SAEs) applied to multimodal encoders. Building upon the disentanglement ability of SAEs, we train them on facial image or caption datasets without corresponding social attribute labels to uncover neurons highly responsive to specific demographics, including those that are underrepresented. By selectively deactivating the social neurons most strongly tied to bias for each group, we effectively mitigate socially biased behaviors of VLMs without degrading their semantic knowledge. Our research lays the groundwork for future auditing tools, prioritizing social fairness in emerging real-world AI systems.

Figures & tables

Figure 2: Overview of our interpretable VLM debiasing framework. DeBiasLens consists of three stages: (1) SAE is trained on top of the last layer of the VLM image/text encoder (Section 3.1 ). (2) The social neurons are identified based on the consistency and specificity of SAE activations across data (Section 3.2 ). (3) The selected neurons are activated to generate debiased features, weighted summed with original features for further usage across downstream tasks (Section 3.3 ).

Figure 3: Comparison between similarity trend of facial image pairs for original and SAE-attached CLIP. The difference between the cosine similarity of random and social attribute-overlapping image pairs (G: gender, R: race, A: age) becomes more pronounced when our SAE is attached, indicating that the SAE can capture latent bias-sensitive features.

Panel of Table 1 — Table 1: Max Skew@1000 (scaled by 100) results on FairFace dataset using diverse prompts for gender bias evaluation . Note ${\dagger}$ represents reproduced results (T and I indicate SAE attached to text and image encoder). Our DeBiasLens attains comparable performance with SoTA VLM debiasing methods without using labels during training (but required during probing), with interpretable inference components.

Figure 4: Comparison between bias mitigation vs . general performance of LVLMs. Our method achieves the best trade-off among existing approaches (

\leftarrow

\uparrow

, the better).

Figure 5: Neuron-specific results for CLIP text encoder. Modulating gender neurons mitigates only gender bias, indicating high neuron specificity.

Figure 6: Top activating images per social neuron. Each social neuron corresponds to a human-interpretable concept of a social bias attribute.

Figure 7: Proportion of effective social neurons (top) and corresponding Max Skew scores (bottom) of CLIP (ViT-B/16) image encoder. The expansion factor 8 shows the most similar results across training datasets, and the FairFace dataset achieves the most stable, consistent bias scores as the expansion factor increases. (0: Male, 1: Female).

Figure 8: Proportion of effective social neurons (top) and corresponding Max Skew scores (bottom) of CLIP (ViT-B/16) text encoder. The expansion factor 8 shows the most similar results across training datasets, and the Cocogendertxt dataset achieves the most stable, consistent bias scores as the expansion factor increases. (0: Male, 1: Female).

Figure 9: Newly generated SBBench synthetic datasets. We synthesize images to observe varying training and social neuron probing datasets when training SAE for bias mitigation.

Panel of Table 3 — Table 3: SBBench (categories: age and gender) accuracy of DeBiasLens applied to LVLM. The best performance is achieved when SAE is trained and gender neurons are selected using the FairFace datasets, measured using a rule-based and model-based evaluation.

Figure 10: Qualitative results of top activating images per gender attribute neuron across different training datasets. FairFace emerges as the most suitable dataset for training SAEs aimed at bias mitigation, as it produces neurons that strongly align with social attributes.

$\boldsymbol{\alpha}$	CLIP ViT-B/16				LLaVA-1.5-7b-hf
	ImgNette $\uparrow$ [ 29 ]		FairFace $\downarrow$ [ 41 ]		MME $\uparrow$ [ 12 ]	VLA $\downarrow$ [ 21 ]
	Image	Text	Image	Text	Image	Text
0.0	99.5	99.1	18.8	16.7	1440.10	0.62
0.2	99.3	99.0	17.8	12.8	1479.28	0.57
0.4	99.0	99.1	16.2	8.7	1496.10	0.53
0.5	98.5	98.9	15.2	7.4	1483.89	0.50

Table 4: General performance and bias mitigation score results of DeBiasLens applied to CLIP and LLaVA across varying weighted proportions. There exists a trade-off between these two aspects for both modalities and model types.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Type	Image Encoder	Text Encoder
CLIP (ViT-B/32)	VLM	ViT-B/32	Transformer (512-d)
CLIP (ViT-L/14@336)	VLM	ViT-L/14@336	Transformer (768-d)
LLaVA-1.5-7B	LVLM	ViT-L/14@336	–
LLaVAOneVision	LVLM	SigLIP-so400m/14@384	–
InternVL2-8B	LVLM	InternViT-300M@448px	–

Appendix

Table 5: Overview of multimodal models. The table lists the image and text encoders used in VLMs and LVLMs considered in this work.

Figure 11: Age distribution of FairFace evaluation dataset across genders. Although the gender distribution is balanced, there is a skewed gender distribution per age group.

Figure 12: Age neuron activating images.

Prompts	Adj	Occup	Act	Ster	Adj	Occup	Act	Ster	Avg
Datasets	PATA				Pairs
CLIP (ViT-B/16)	14.1	19.9	10.2	20.3	13.1	22.4	26.8	18.5	18.16
Prompt	7.3	16.9	10.0	16.0	11.9	23.9	15.2	15.0	14.53
DeBiasLens	7.1	13.3	11.9	5.5	10.5	19.5	17.7	11.6	12.14

Appendix

Table 6: Bias Score Results on Non-Overlapping Datasets.

Figure 13: Difference between male and female ’yes’ probability across skill-related prompts for InternVL2-8b and DeBiasLens -Intern. Our method shows most of the skills having no statistically different probability across genders (*: p

<

0.1, **: p

<

0.01, ***: p

<

0.001).

Figure 14: Difference between male and female ’yes’ probability across occupation-related prompts for InternVL2-8b and DeBiasLens -Intern. Our method shows most of the occupations having no statistically different probability across genders (*: p

<

0.1, **: p

<

0.01, ***: p

<

0.001).

Figure 15: Difference between male and female ’yes’ probability across sentiment-related prompts for InternVL2-8b and DeBiasLens -Intern. Our method shows most of the sentiments having no statistically different probability across genders (*: p

<

0.1, **: p

<

0.01, ***: p

<

0.001).

Method	Par (M)	GPU hrs	Overhead (ms)	FLOPs	Trade-off
Full FT	6979.58	0.02	310.19	1.14e+13	1.29
LoRA FT	301.99	0.32	310.89	1.14e+13	1.30
Pruning (0.05)	-	0.00	355.31	1.11e+13	1.53
Pruning (0.5)	-	0.00	268.35	7.90e+12	1.18
Prompt Tuning	0.08	1.72	361.15	1.52e+14	0.92
Prompt Engin.	-	0.00	316.74	1.22e+13	1.35

Appendix

Table 7: Computational Cost Results. The trade-off score (

\uparrow

the better) is proportional to

\Delta\text{BiasScore}-\Delta\text{VLAPerf}

Targeted Attributes	Gender Skew $\Delta$	Age Skew $\Delta$	Race Skew $\Delta$
Gender only	-8.0%	-5.6%	-1.3%
Age only	-8.1%	-18.0%	-3.1%
Race only	-7.9%	-6.3%	-6.4%
Gender $\times$ Age	-11.3%	-18.7%	-4.1%
Gender $\times$ Race	-10.7%	-7.4%	-6.4%
Age $\times$ Race	-11.2%	-19.5%	-11.7%

Appendix

Table 8: Intersectional Fairness (MaxSkew) Results.

Panel of Table 9 — Table 9: SBBench (categories: age and gender) accuracy of DeBiasLens applied to LVLM. The best performance is achieved when SAE is trained and age neurons are selected using the FairFace datasets, measured using a rule-based and model-based evaluation.

LLaVA-1.5-7b-hf [ 51 ] ( Fairface $-$ Top neurons )
$\alpha$	Percep [ 12 ]	Reason [ 12 ]	MMMU [ 87 ]	Seed2 [ 50 ]
0.0	1205.10	235.00	0.30	0.59
0.2	1252.50	226.78	0.29	0.59
0.4	1240.75	255.35	0.29	0.58
0.5	1209.25	274.64	0.30	0.58
0.6	1187.84	266.42	0.30	0.57

Appendix

Table 10: General performance (

\uparrow

) on varying weighted proportion for LVLMs. The general VLM performance overall decreases as the weight proportion of the SAE decoded embedding increases.

$\alpha$	Occupations	Trait	Trait (gendered)	Skills
0.0	0.3500	0.7000	0.7500	0.7143
0.2	0.3250	0.6000	0.7083	0.6667
0.4	0.3250	0.6000	0.5833	0.6190
0.5	0.3250	0.5500	0.5417	0.6190
0.6	0.3250	0.5500	0.5417	0.6190
0.8	0.2750	0.5500	0.5417	0.5714

Appendix

Table 11: Gender disproportion rate (

\downarrow

) across varying weighted proportions for LLaVA-1.5-7b-hf. The disproportion rate decreases as the weight proportion of the SAE decoded embedding increases.

Representation	$\alpha$	Gender Acc	Age Acc	Race Acc
$\mathbf{v}$ (original)	0.0	95.9	55.6	71.0
$\mathbf{v^{\prime}}$ (mixed)	0.6	95.2	56.2	70.8
$\hat{\mathbf{v}}$ (SAE recon)	1.0	92.7	51.4	62.6

Appendix

Table 12: Social Attribute Predictability Results.

Figure 16: Additional newly generated SBBench synthetic datasets. We synthesize images to test the effect of varying training and probing datasets when training SAE for bias mitigation.

Figure 17: Qualitative results of top retrieved images given neutral prompts. Our DeBiasLens retrieves a fairer distribution across genders compared to the original model (CLIP ViT-B/32).

Figure 18: Qualitative results on responses to ambiguous visual questions. Our DeBiasLens (right, skyeblue bars) tends to respond more cautiously, favoring the option of “unknown,” whereas the baseline (InternVL2-8B, left, darkblue bars) more often commits to definitive “yes” or “no” responses, despite the questions having no single correct answer.

Figure 19: Additional top activating images per two social neurons across categories. Each social neuron corresponds to a human-interpretable concept of a social bias attribute.

Figure 20: Interpretable social neurons. We visualize the top activating training (top row) and evaluation (bottom row) images for each social neuron labeled with two human-interpretable concepts.

Figure 21: Proportion of effective age neurons (top) and corresponding Max Skew scores (bottom) of CLIP (ViT-B/16) image encoder. The expansion factor 8 shows the lowest bias scores across thresholds (0: 3-9, 1: 10-19, 2: 20-29, 3: 30-39, 4: 40-49, 5: 50-59, 6: 60-69).

Figure 22: Proportion of effective race neurons (top) and corresponding Max Skew scores (bottom) of CLIP (ViT-B/16) image encoder. The expansion factors 2 and 8 show the lowest and the most stable bias scores, respectively, across thresholds (0: White, 1: Southeast Asian, 2: Middle Eastern, 3: Black, 4: Indian, 5: Latino Hispanic, 6: East Asian).

Figure 23: Proportion of effective age neurons (top) and corresponding Max Skew scores (bottom) of CLIP (ViT-L/14@336) image encoder. The expansion factors 4 and 8 show the lowest bias scores across thresholds (0: Male, 1: Female).

Figure 24: Proportion of effective race neurons (top) and corresponding Max Skew scores (bottom) of CLIP (ViT-L/14@336) text encoder. The expansion factors 1 and 8 overall show the lowest bias scores across thresholds (0: Male, 1: Female).

Figure 25: Proportion of effective age neurons (top) and corresponding Max Skew scores (bottom) of CLIP (ViT-L/14@336) image encoder. The expansion factor 8 shows the lowest bias scores across thresholds (0: 3-9, 1: 10-19, 2: 20-29, 3: 30-39, 4: 40-49, 5: 50-59, 6: 60-69).

Figure 26: Proportion of effective race neurons (top) and corresponding Max Skew scores (bottom) of CLIP (ViT-L/14@336) image encoder. The expansion factors 2 and 8 show the lowest and the most stable bias scores, respectively, across thresholds (0: White, 1: Southeast Asian, 2: Middle Eastern, 3: Black, 4: Indian, 5: Latino Hispanic, 6: East Asian).

Figure 27: Proportion of effective gender (top), age (middle), and race (bottom) neurons of InternVL2-8B image encoder. There is a similar trend of effective neuron proportions across expansion factors for different social attributes ( Gender –0: Male, 1: Female; Age –0: 3-9, 1: 10-19, 2: 20-29, 3: 30-39, 4: 40-49, 5: 50-59, 6: 60-69; Race –0: White, 1: Southeast Asian, 2: Middle Eastern, 3: Black, 4: Indian, 5: Latino Hispanic, 6: East Asian).

Figure 28: Proportion of effective gender (top), age (middle), and race (bottom) neurons of LLaVA-1.5-7b-hf image encoder. There is a similar trend of effective neuron proportions across expansion factors for different social attributes ( Gender –0: Male, 1: Female; Age –0: 3-9, 1: 10-19, 2: 20-29, 3: 30-39, 4: 40-49, 5: 50-59, 6: 60-69; Race –0: White, 1: Southeast Asian, 2: Middle Eastern, 3: Black, 4: Indian, 5: Latino Hispanic, 6: East Asian).

Figure 29: Proportion of effective gender neurons of LLaVA-1.5-7b-hf (top), InternVL2-8B (middle), and LLaVAOneVision (bottom) image encoder. There is a similar trend of effective neuron proportions across expansion factors for different models, even when trained and probed with synthetic datasets (0: Male, 1: Female).

Figure 30: Proportion of effective age neurons of LLaVA-1.5-7b-hf (top), InternVL2-8B (middle), and LLaVAOneVision (bottom) image encoder. There is a similar trend of effective neuron proportions across expansion factors for different models, even when trained and probed with synthetic datasets (0: Old, 1: Young).

Cultural Bias in VLMs

See all 14 papers

A Closed-Form Solution for Debiasing Vision-Language Models with Utility Guarantees Across Modalities and Tasks
Mar 13, 2026Tangzheng Lian, Guanyu Hu, Yijing Ren +2Cultural Bias in VLMs Algorithmic Fairness
King’s College London · Queen Mary University of London
SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Models
Mar 19, 2026Quentin Guimard, Federico Bartsch, Simone Caldarella +3Cultural Bias in VLMs Vision-Language Models
University of Trento · Toyota Motor Europe · Fondazione Bruno Kessler
Mitigating Bias in Large Vision-Language Models via Counterfactual Ensemble Decoding
Aug 12, 2026Yisong Xiao, Aishan Liu, Yongxin Huang +6Cultural Bias in VLMs Vision-Language Models
State Key Lab of Software Development Environment, Beihang University, Beijing 100191, China

Mar 13, 2026J/K move · Enter open · S save

A Closed-Form Solution for Debiasing Vision-Language Models with Utility Guarantees Across Modalities and Tasks

Tangzheng Lian, Guanyu Hu, Yijing Ren, Dimitrios Kollias+1

King’s College London · Queen Mary University of London

Cultural Bias in VLMs Algorithmic Fairness Intersectional Fairness

Open paper AskPDF arXiv

While Vision-Language Models (VLMs) have achieved remarkable performance across diverse downstream tasks, recent studies have shown that they can inherit social biases from the training data and further propagate them into downstream applications. To address this issue, various debiasing approaches have been proposed, yet most of them aim to improve fairness without having a theoretical guarantee that the utility of the model is preserved. In this paper, we introduce a debiasing method that yields a closed-form solution in the cross-modal space, achieving Pareto-optimal fairness with bounded utility losses. Our method is training-free, requires no annotated data, and can jointly debias both visual and textual modalities across downstream tasks. Extensive experiments show that our method outperforms existing methods in debiasing VLMs across diverse fairness metrics and datasets for both group and intersectional fairness in downstream tasks such as zero-shot image classification, text-to-image retrieval, and text-to-image generation while preserving task performance.

Abstract

Cultural Bias in VLMs

Explore similar work

A Closed-Form Solution for Debiasing Vision-Language Models with Utility Guarantees Across Modalities and Tasks

SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Models

Mitigating Bias in Large Vision-Language Models via Counterfactual Ensemble Decoding