Large Vision-Language Models (LVLMs) have achieved remarkable performance across a wide range of tasks; however, they often inherit social biases from their training data, resulting in biased behavior when processing portraits from different social groups. Existing debiasing approaches typically compare token probabilities between the original and biased generations during decoding, but they are fundamentally limited by their reliance on a single, stereotyped viewpoint and fail to account for the diversity of social perspectives. Inspired by the social science principle that diversity fosters fairness, we propose Counterfactual Ensemble Decoding (CED), a novel framework that constructs multi-group counterfactual perspectives within the visual representation space and integrates them during decoding to promote equitable model behavior. CED first performs counterfactual steering in the visual space by identifying semantic directions associated with each social group and generating counterfactual representations along these directions, thereby offering diverse perspectives that disrupt stereotypical narratives. During decoding, CED locates the decoder layer exhibiting the greatest divergence among these perspectives and ensembles their token distributions using uncertainty-aware weights, prioritizing high-confidence tokens from different groups to yield a more balanced probability distribution that guides fairer generation. Extensive experiments on three social bias evaluation benchmarks demonstrate that \tool achieves substantial improvements over leading baselines, reducing bias by up to 47.97% across scenarios involving occupations, descriptors, and persona traits. Moreover, CED also preserves the core capabilities of the original model with minimal degradation.
Figures & tables
Figure 1: Illustration of gender bias in LVLMs within the occupation scenario. We introduce CED for debiasing, which ensembles diverse perspectives to disrupt the dominance of stereotypical narratives.
Figure 2: Overview of CED. CED identifies directions for each social group and applies steering to generate counterfactual visual representations, introducing diverse perspectives. Then, CED locates the layer with the greatest conflict among these perspectives and ensembles their token distributions, disrupting the dominance of stereotypical narratives and promoting balanced token probabilities in the predictions.
Model
Method
original
reprompt
explanation
GenProj
SelfDebias
CED
LLaVA-1.5
Ipss↑
53.21
52.56
\ul 55.05
53.86
53.56
63.02
Bovl↓
23.24
24.37
\ul 13.61
22.38
23.33
7.90
Acc↑
68.53
68.59
62.05
68.67
69.08
\ul 68.89
Qwen3-VL
Ipss↑
66.23
\ul 67.54
63.12
64.56
62.69
72.28
Bovl↓
15.70
13.84
\ul 7.37
15.45
14.82
4.10
Acc↑
77.65
\ul 76.81
69.87
76.65
74.43
75.99
TABLE I: Results of different methods on the GenderBias-VL.
Figure 3: Illustration of debiasing performance on an example from GenderBias-VL. The left shows the base image with the occupation Legal secretary , while the right is its gender counterfactual version. Bias is measured as the absolute difference in the selection probability of the Legal secretary option between the original and counterfactual questions.
Model
Method
pair1
pair2
pair3
pair4
pair5
pair6
pair7
pair8
pair9
pair10
LLaVA-1.5
original
25.69
37.09
4.52
43.84
5.32
21.18
27.56
32.39
15.47
19.35
reprompt
22.14
38.27
\ul 4.20
46.61
6.43
19.12
28.56
37.83
16.61
23.88
explanation
\ul 12.80
\ul 11.32
18.76
\ul 23.64
5.86
\ul 14.15
\ul 13.68
18.75
\ul 8.20
\ul 8.91
GenProj
23.26
36.57
4.21
43.17
\ul 5.09
20.56
26.56
30.61
15.07
18.73
SelfDebias
25.65
37.78
4.57
43.97
5.46
21.34
27.55
32.18
15.38
19.37
CED
8.29
2.01
3.46
21.66
3.70
8.00
6.88
\ul 19.19
0.93
4.84
TABLE II: Bias results ( Bpair↓ ) of different methods on each occupation pair from the GenderBias-VL dataset. The occupation pairs are as follows: pair1 ( Dentist , Dental hygienist ), pair2 ( CEO , Executive secretary ), pair3 ( Surgeon , Surgical technologist ), pair4 ( Lawyer , Legal secretary ), pair5 ( Refractory mechanic , Filling operator ), pair6 ( Aircraft pilot , Flight attendant ), pair7 ( Computer Systems Manager , Receptionist ), pair8 ( EMT , Licensed practical nurse ), pair9 ( Network Architect , Billing Clerk ), and pair10 ( Financial analyst , HR manager ).
Model
Method
original
reprompt
explanation
GenProj
SelfDebias
CED
LLaVA-1.5
occupation
5.52
5.59
5.29
5.41
\ul 4.74
2.77
descriptor
6.38
6.65
\ul 4.91
6.32
5.86
2.36
persona
5.98
6.00
\ul 4.09
5.83
5.59
3.01
Qwen3-VL
occupation
5.26
5.07
\ul 4.19
5.02
4.34
2.37
descriptor
6.48
6.04
\ul 4.54
6.43
4.89
3.08
persona
4.48
4.24
\ul 3.20
4.31
3.95
2.11
TABLE III: Race bias results ( Sbias↓ ) on ModSCAN across occupation, descriptor, and persona trait scenarios.
Figure 4: Distribution of LLaVA-1.5’s stereotypical racial bias ( Sbias ) on ModSCAN across occupation (top), descriptor (middle), and persona trait (bottom) scenarios.
Metric
Method
LLaVA-1.5
Qwen3-VL
InternVL2
stereostype
original
9.34
11.68
9.28
reprompt
8.92
10.06
\ul 9.04
GenProj
9.68
11.33
9.05
SelfDebias
\ul 7.73
\ul 9.56
9.14
CED
5.27
6.50
6.80
sentiment
original
1.74%
2.40%
1.66%
TABLE IV: Results on VisBias. The stereotype word bias score ( stereotype↓ ) is scaled by a factor of 1000, while the sentiment bias score ( sentiment↓ ) is reported as a percentage.
Dataset
Method
LLaVA-1.5
Qwen3-VL
InternVL2
MMBench
original
62.89
83.93
82.56
reprompt
62.20
84.45
82.56
explanation
48.80
78.10
75.95
GenProj
62.80
\ul 84.02
81.79
SelfDebias
59.02
83.93
78.52
CED
\ul 62.54
81.68
\ul 82.38
TABLE V: Capability evaluation results ( accuracy ↑ ) of different methods on three LVLMs.
Figure 5: Bias distribution of LLaVA-1.5 on VisBias: stereotype bias (top) and sentiment bias (bottom).
Figure 6: Illustration of bias mitigation using our CED in the description of the female CEO image. The stereotypical words are highlighted in red .
The rapid advancement of Vision-Language models (VLMs) has raised growing concerns that their black-box reasoning processes could lead to unintended forms of social bias. Current debiasing approaches focus on mitigating surface-level bias signals through post-hoc learning or test-time algorithms, while leaving the internal dynamics of the model largely unexplored. In this work, we introduce an interpretable, model-agnostic bias mitigation framework, DeBiasLens, that localizes social attribute neurons in VLMs through sparse autoencoders (SAEs) applied to multimodal encoders. Building upon the disentanglement ability of SAEs, we train them on facial image or caption datasets without corresponding social attribute labels to uncover neurons highly responsive to specific demographics, including those that are underrepresented. By selectively deactivating the social neurons most strongly tied to bias for each group, we effectively mitigate socially biased behaviors of VLMs without degrading their semantic knowledge. Our research lays the groundwork for future auditing tools, prioritizing social fairness in emerging real-world AI systems.
While Vision-Language Models (VLMs) have achieved remarkable performance across diverse downstream tasks, recent studies have shown that they can inherit social biases from the training data and further propagate them into downstream applications. To address this issue, various debiasing approaches have been proposed, yet most of them aim to improve fairness without having a theoretical guarantee that the utility of the model is preserved. In this paper, we introduce a debiasing method that yields a closed-form solution in the cross-modal space, achieving Pareto-optimal fairness with bounded utility losses. Our method is training-free, requires no annotated data, and can jointly debias both visual and textual modalities across downstream tasks. Extensive experiments show that our method outperforms existing methods in debiasing VLMs across diverse fairness metrics and datasets for both group and intersectional fairness in downstream tasks such as zero-shot image classification, text-to-image retrieval, and text-to-image generation while preserving task performance.
Tangzheng Lian, Guanyu Hu, Yijing Ren +2
King’s College London · Queen Mary University of London
Gender bias in large vision-language models (LVLMs) undermines their fairness and reliability, compromising output trustworthiness. Current mitigation methods rely on training-phase adjustments or post-hoc calibration, but face limitations in dynamic visual bias mitigation. These include inability to capture real-time visual-textual incongruence, dependence on predefined gender bias taxonomies, and degraded cross-modal alignment with emergent bias patterns. To address these challenges, we propose ViD, a causally-inspired framework that analyzes attention mechanisms across five distinct patterns, revealing confounding effects from strong language priors. ViD demonstrates that visual-to-language cross-attention effectively suppresses bias while preserving general reasoning capabilities and text generation quality. ViD incorporates dual mechanisms: backdoor adjustment counters strong language priors, while refined token selection in decoding layers optimizes processing. This enhances model robustness and inference efficiency. Our integrated approach significantly mitigates gender bias across multidimensional social attributes in LVLMs, improving visual grounding and output fairness. Cross-benchmark validation shows ViD reduces gender bias by 14.7% on single-attribute evaluations (FACET) and achieves significant improvements on image captioning tasks (MS COCO), with gender bias score improving from 0.6708 to 0.9978 for LLaVA. Crucially, these improvements require no additional training overhead, making ViD a scalable and practical solution for bias mitigation in LVLMs.