Mitigating Bias in Large Vision-Language Models via Counterfactual Ensemble Decoding

Yisong Xiao; Aishan Liu; Yongxin Huang; Zonghao Ying; Shiji Zhao; Tianlin Li; Yong Han; Jian Yang

cs.CLAug 12, 2026

Mitigating Bias in Large Vision-Language Models via Counterfactual Ensemble Decoding

Authors: Yisong Xiao, Aishan Liu, Yongxin Huang, Zonghao Ying, Shiji Zhao, Tianlin Li, Yong Han, Jian Yang,

Organizations: State Key Lab of Software Development Environment, Beihang University, Beijing 100191, China

Abstract

Large Vision-Language Models (LVLMs) have achieved remarkable performance across a wide range of tasks; however, they often inherit social biases from their training data, resulting in biased behavior when processing portraits from different social groups. Existing debiasing approaches typically compare token probabilities between the original and biased generations during decoding, but they are fundamentally limited by their reliance on a single, stereotyped viewpoint and fail to account for the diversity of social perspectives. Inspired by the social science principle that diversity fosters fairness, we propose Counterfactual Ensemble Decoding (CED), a novel framework that constructs multi-group counterfactual perspectives within the visual representation space and integrates them during decoding to promote equitable model behavior. CED first performs counterfactual steering in the visual space by identifying semantic directions associated with each social group and generating counterfactual representations along these directions, thereby offering diverse perspectives that disrupt stereotypical narratives. During decoding, CED locates the decoder layer exhibiting the greatest divergence among these perspectives and ensembles their token distributions using uncertainty-aware weights, prioritizing high-confidence tokens from different groups to yield a more balanced probability distribution that guides fairer generation. Extensive experiments on three social bias evaluation benchmarks demonstrate that \tool achieves substantial improvements over leading baselines, reducing bias by up to 47.97% across scenarios involving occupations, descriptors, and persona traits. Moreover, CED also preserves the core capabilities of the original model with minimal degradation.

Figures & tables

Figure 1: Illustration of gender bias in LVLMs within the occupation scenario. We introduce CED for debiasing, which ensembles diverse perspectives to disrupt the dominance of stereotypical narratives.

Figure 2: Overview of CED. CED identifies directions for each social group and applies steering to generate counterfactual visual representations, introducing diverse perspectives. Then, CED locates the layer with the greatest conflict among these perspectives and ensembles their token distributions, disrupting the dominance of stereotypical narratives and promoting balanced token probabilities in the predictions.

Model	Method	original	reprompt	explanation	GenProj	SelfDebias	CED
LLaVA-1.5	$Ipss{\uparrow}$	53.21	52.56	\ul 55.05	53.86	53.56	63.02
	$B_{ovl}{\downarrow}$	23.24	24.37	\ul 13.61	22.38	23.33	7.90
	$Acc{\uparrow}$	68.53	68.59	62.05	68.67	69.08	\ul 68.89
Qwen3-VL	$Ipss{\uparrow}$	66.23	\ul 67.54	63.12	64.56	62.69	72.28
	$B_{ovl}{\downarrow}$	15.70	13.84	\ul 7.37	15.45	14.82	4.10
	$Acc{\uparrow}$	77.65	\ul 76.81	69.87	76.65	74.43	75.99

TABLE I: Results of different methods on the GenderBias-VL.

Figure 3: Illustration of debiasing performance on an example from GenderBias-VL. The left shows the base image with the occupation Legal secretary , while the right is its gender counterfactual version. Bias is measured as the absolute difference in the selection probability of the Legal secretary option between the original and counterfactual questions.

Model	Method	pair1	pair2	pair3	pair4	pair5	pair6	pair7	pair8	pair9	pair10
LLaVA-1.5	original	25.69	37.09	4.52	43.84	5.32	21.18	27.56	32.39	15.47	19.35
	reprompt	22.14	38.27	\ul 4.20	46.61	6.43	19.12	28.56	37.83	16.61	23.88
	explanation	\ul 12.80	\ul 11.32	18.76	\ul 23.64	5.86	\ul 14.15	\ul 13.68	18.75	\ul 8.20	\ul 8.91
	GenProj	23.26	36.57	4.21	43.17	\ul 5.09	20.56	26.56	30.61	15.07	18.73
	SelfDebias	25.65	37.78	4.57	43.97	5.46	21.34	27.55	32.18	15.38	19.37
	CED	8.29	2.01	3.46	21.66	3.70	8.00	6.88	\ul 19.19	0.93	4.84

TABLE II: Bias results (

B_{pair}{\downarrow}

) of different methods on each occupation pair from the GenderBias-VL dataset. The occupation pairs are as follows: pair1 ( Dentist , Dental hygienist ), pair2 ( CEO , Executive secretary ), pair3 ( Surgeon , Surgical technologist ), pair4 ( Lawyer , Legal secretary ), pair5 ( Refractory mechanic , Filling operator ), pair6 ( Aircraft pilot , Flight attendant ), pair7 ( Computer Systems Manager , Receptionist ), pair8 ( EMT , Licensed practical nurse ), pair9 ( Network Architect , Billing Clerk ), and pair10 ( Financial analyst , HR manager ).

Model	Method	original	reprompt	explanation	GenProj	SelfDebias	CED
LLaVA-1.5	occupation	5.52	5.59	5.29	5.41	\ul 4.74	2.77
	descriptor	6.38	6.65	\ul 4.91	6.32	5.86	2.36
	persona	5.98	6.00	\ul 4.09	5.83	5.59	3.01
Qwen3-VL	occupation	5.26	5.07	\ul 4.19	5.02	4.34	2.37
	descriptor	6.48	6.04	\ul 4.54	6.43	4.89	3.08
	persona	4.48	4.24	\ul 3.20	4.31	3.95	2.11

TABLE III: Race bias results (

S_{bias}\downarrow

) on ModSCAN across occupation, descriptor, and persona trait scenarios.

Figure 4: Distribution of LLaVA-1.5’s stereotypical racial bias ( $S_{bias}$ ) on ModSCAN across occupation (top), descriptor (middle), and persona trait (bottom) scenarios.

Metric	Method	LLaVA-1.5	Qwen3-VL	InternVL2
$stereostype$	original	9.34	11.68	9.28
	reprompt	8.92	10.06	\ul 9.04
	GenProj	9.68	11.33	9.05
	SelfDebias	\ul 7.73	\ul 9.56	9.14
	CED	5.27	6.50	6.80
$sentiment$	original	1.74%	2.40%	1.66%

TABLE IV: Results on VisBias. The stereotype word bias score (

stereotype\downarrow

) is scaled by a factor of 1000, while the sentiment bias score (

sentiment\downarrow

) is reported as a percentage.

Dataset	Method	LLaVA-1.5	Qwen3-VL	InternVL2
MMBench	original	62.89	83.93	82.56
	reprompt	62.20	84.45	82.56
	explanation	48.80	78.10	75.95
	GenProj	62.80	\ul 84.02	81.79
	SelfDebias	59.02	83.93	78.52
	CED	\ul 62.54	81.68	\ul 82.38

TABLE V: Capability evaluation results ( accuracy

\uparrow

) of different methods on three LVLMs.

Figure 5: Bias distribution of LLaVA-1.5 on VisBias: stereotype bias (top) and sentiment bias (bottom).

Figure 6: Illustration of bias mitigation using our CED in the description of the female CEO image. The stereotypical words are highlighted in red .

Figure 7: Bias ( $B_{ovl}$ ) across varying counterfactual manipulation magnitude $\alpha$ .

Method	original	reprompt	GenProj	SelfDebias	CED
Time (s)	5.49	10.20	6.14	12.09	7.80

TABLE VI: Time (seconds) consumed by different bias mitigation methods to generate 128 tokens on the LLaVA-1.5 model.

Cultural Bias in VLMs

See all 14 papers

Interpretable Debiasing of Vision-Language Models for Social Fairness
Feb 27, 2026Na Min An, Yoonna Jang, Yusuke Hirota +3Cultural Bias in VLMs VLM Interpretability
KAIST AI · University of Copenhagen · NVIDIA
A Closed-Form Solution for Debiasing Vision-Language Models with Utility Guarantees Across Modalities and Tasks
Mar 13, 2026Tangzheng Lian, Guanyu Hu, Yijing Ren +2Cultural Bias in VLMs Algorithmic Fairness
King’s College London · Queen Mary University of London
ViD: Vision-Dominant Gender Bias Mitigation for Large Vision-Language Models
Sep 15, 2026Zhipeng Zhao, Zhaoqiang Wei, Peishun Liu +2Vision-Language Models Large Vision-Language Models
Ocean University of China

Feb 27, 2026J/K move · Enter open · S save

Interpretable Debiasing of Vision-Language Models for Social Fairness

Na Min An, Yoonna Jang, Yusuke Hirota, Ryo Hachiuma+2

KAIST AI · University of Copenhagen · NVIDIA

Cultural Bias in VLMs VLM Interpretability

Open paper AskPDF arXiv

The rapid advancement of Vision-Language models (VLMs) has raised growing concerns that their black-box reasoning processes could lead to unintended forms of social bias. Current debiasing approaches focus on mitigating surface-level bias signals through post-hoc learning or test-time algorithms, while leaving the internal dynamics of the model largely unexplored. In this work, we introduce an interpretable, model-agnostic bias mitigation framework, DeBiasLens, that localizes social attribute neurons in VLMs through sparse autoencoders (SAEs) applied to multimodal encoders. Building upon the disentanglement ability of SAEs, we train them on facial image or caption datasets without corresponding social attribute labels to uncover neurons highly responsive to specific demographics, including those that are underrepresented. By selectively deactivating the social neurons most strongly tied to bias for each group, we effectively mitigate socially biased behaviors of VLMs without degrading their semantic knowledge. Our research lays the groundwork for future auditing tools, prioritizing social fairness in emerging real-world AI systems.

Abstract

Cultural Bias in VLMs

Explore similar work

Interpretable Debiasing of Vision-Language Models for Social Fairness

A Closed-Form Solution for Debiasing Vision-Language Models with Utility Guarantees Across Modalities and Tasks

ViD: Vision-Dominant Gender Bias Mitigation for Large Vision-Language Models