This paper studies the problem of test-time robustness scaling for large vision-language models (LVLMs), which aims to mitigate language bias during inference. Existing approaches typically adopt test-time scaling techniques such as contrastive decoding to improve inference performance. Despite the progress, the performance of these approaches is far from satisfactory due to intentional evidence removal during perturbations and redundant inference cost. Towards this end, we propose a novel approach named Margin-gated Contrastive Debiasing with Evidence Restoration (MAGIC). The core of our MAGIC is to evaluate the decision margins of LVLMs to selectively activate textual and visual perturbation branches for efficient debiased inference. In particular, our MAGIC first measures the prediction margins of crucial tokens, and applies a gating function to identify low-confidence responses that are susceptible to language biases. For selected samples, we integrate the token logits from multiple textual and image perturbation branches with the original ones for debiased decision. More importantly, we selectively restore the most informative corrupted views with sharp prediction distributions for ambiguous samples after debiasing, enabling evidence recovery for discriminative decision-making. We further utilize a calibration dataset to actively identify and prune ineffective branches during contrastive debiasing for cost reduction. Extensive experiments on benchmark datasets validate the effectiveness of the proposed MAGIC in comparison with extensive baselines.
Figures & tables
Figure 1: Overview of MAGIC . Each step fires only when the margin m still leaves the decision contested, so a sample that clears a gate exits with the scores it already has. Thresholds are fitted offline on the validation split.
Figure 2: Utility of the ungated debiasing step against the base decision margin, on biased and natural samples. Bands are 95% intervals and shading marks the margins over which the two curves separate.
Figure 3: Error of the corrector of Eq. ( 1 ) as h^ averages k branches, with an a+b/k fit and its asymptote a dashed.
Subset Size
B Subset
S Subset
BS Subset
Overlap
LLaVA-NeXT (MCQ)
1810
1005
2476
339
LLaVA-NeXT (Others)
345
582
794
133
LLaVA-NeXT (Overall)
2155
1587
3270
472
Qwen2-VL (MCQ)
1080
252
1243
89
Qwen2-VL (Others)
327
311
513
125
Table 1: The size of each subset in constructed DRBench. The overall number of samples across all six datasets is 13,251 , with the MCQ and Others categories being 10,632 and 2,619 , respectively.
Bias Subset
Sensitivity Subset
BS Subset
Model
Method
MCQ
Others
All
MCQ
Others
All
MCQ
Others
All
Passes
Base
5.37
8.56
6.11
38.10
34.41
36.06
10.78
23.59
14.52
1
TIE
16.20
16.82
16.35
45.63
36.66
40.67
20.27
27.29
22.32
2
VCD
15.74
21.71
17.13
46.83
40.84
43.52
20.11
30.41
23.12
2
M3ID
19.81
21.71
20.26
47.22
41.16
43.87
23.65
30.60
25.68
2
SCI 3
21.67
26.30
22.74
44.05
42.44
43.16
24.54
32.75
26.94
3
Table 2: Experiments on the B(ias) Subset, S(ensitivity) Subset, and BS Subset of DRBench. Baseline results are from Tang et al. (2026a) . Passes is the mean per-sample forward count. We bold the best result in each column and underline the second best.
Single Dataset
Question Type
Method
MMB-C
MMB-E
MME
CCB
MMS
ViLP
MCQ
Others
Overall
Qwen2-VL
85.26
86.36
87.89
73.22
59.50
56.53
80.91
79.27
80.58
Qwen2-VL-TIE
86.00 ↑ 0.74
86.59 ↑ 0.23
86.52 ↓ 1.37
73.84 ↑ 0.62
59.00 ↓ 0.50
57.08 ↑ 0.55
81.30 ↑ 0.39
78.43 ↓ 0.84
80.73 ↑ 0.15
Qwen2-VL-VCD
86.05 ↑ 0.79
86.56 ↑ 0.20
86.41 ↓ 1.48
73.77 ↑ 0.55
60.08 ↑ 0.58
57.92 ↑ 1.39
81.42 ↑ 0.51
78.58 ↓ 0.69
80.86 ↑ 0.28
Qwen2-VL-M3ID
85.69 ↑ 0.43
86.46 ↑ 0.10
86.10 ↓ 1.79
73.96 ↑ 0.74
59.75 ↑ 0.25
57.78 ↑ 1.25
81.25 ↑ 0.34
78.31 ↓ 0.96
80.67 ↑ 0.09
Qwen2-VL-SCI 5
85.97 ↑ 0.71
86.67 ↑ 0.31
87.36 ↓ 0.53
73.59 ↑ 0.37
59.92 ↑ 0.42
58.06 ↑ 1.53
81.39 ↑ 0.48
79.31 ↑ 0.04
80.98 ↑ 0.40
Table 3: Experiments on the six real-world LVLM datasets, MMB(ench-Dev)-C/E(N-V11), MME, CCB(ench), MMS(tar), and ViLP (using 80% test splits). Baseline results are from Tang et al. (2026a) .
Figure 4: Accuracy against compute on the biased subset of Qwen2-VL as t1 is swept. Each baseline sits at one fixed budget, while MAGIC traces a frontier. The star marks the deployed setting.
Figure 5: Two biased-subset samples MAGIC repairs on Qwen2-VL, at the fitted thresholds. Each note keeps its stage’s color from Figure 1 and is dashed when that gate never opened. ✓ and ✗ flag the answer a stage leaves, m is the top-two margin, and the committed answer is bold.
Model
Method
B
S
BS
Passes
Base
0.49
34.65
12.56
1
TIE
5.71
32.44
13.47
2
VCD
7.19
32.60
14.27
2
M3ID
8.74
33.54
15.18
2
SCI 3
21.37
41.42
24.96
3
SCI 5
22.43
47.56
26.44
5
Table 4: B/S/BS Overall accuracy on two more recent backbones. We bold the best result in each column and underline the second best.
Model
Configuration
MCQ
Others
Overall
Passes
MAGIC w/o CD
13.17
25.39
16.70
2.2
MAGIC w/o BC
31.59
34.77
32.51
4.0
MAGIC w/o AF
31.11
32.42
31.49
3.2
MAGIC w/o SR
30.40
33.59
31.32
3.8
Qwen2-VL
MAGIC
32.30
35.16
33.13
4.9
MAGIC w/o CD
17.07
38.44
22.28
1.9
Table 5: Ablation on the BS Subset of DRBench. CD: Contrastive Debiasing, BC: Blank Contrast, AF: Answer-Format Contrast, SR: Selective Restoration. Passes is the mean per-sample forward count.
Model
Value
MCQ
Others
Overall
Passes
Debiasing gate t1
0.5
18.81
29.30
21.84
3.0
1.0
24.68
31.64
26.69
3.7
2.0
31.03
34.38
32.00
4.4
3.0
32.30
34.96
33.07
4.8
5.0
32.38
35.74
33.35
5.0
Table 6: Sensitivity to the two allocation thresholds on Qwen2-VL, BS Subset of DRBench. We bold the best result in each column within a block.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Four further Qwen2-VL samples, one per exit of Algorithm 1 .
Model
Value
MCQ
Others
Overall
Passes
Debiasing gate t1
0.5
18.81
29.30
21.84
3.0
1.0
24.68
31.64
26.69
3.7
2.0
31.03
34.38
32.00
4.4
3.0
32.30
34.96
33.07
4.8
5.0
32.38
35.74
33.35
5.0
Appendix
Table 7: Sensitivity to the two allocation thresholds on the BS Subset of DRBench, on both backbones. We bold the best result in each column within a block.
Biased (BS)
Natural
Method
MCQ
Others
MCQ
Others
SCI 5
5.0
5.0
5.0
5.0
SCI 7
7.0
7.0
7.0
7.0
MAGIC (Qwen2-VL)
5.0
4.5
1.5
2.8
MAGIC (LLaVA-NeXT)
4.9
4.2
2.4
2.5
Appendix
Table 8: Average inference passes per sample.
Figure 7: Random and keyword-matched replacements decide as a blank image does. The keyed-full condition answers the question and is the positive control.
Qwen2-VL
LLaVA-NeXT
Method
S
BS
S
BS
VACoDe-A (max contrast)
46.91
22.28
40.99
26.78
VACoDe-A (min contrast)
38.55
32.20
31.09
29.38
CHASD-A (uncertainty-gated)
48.73
30.53
40.30
30.47
gate only (blank contrast)
45.35
29.29
39.52
30.00
MAGIC
48.73
32.30
40.99
31.13
Appendix
Table 9: Our answer-level adaptations of the nearest adaptive-contrast methods, MCQ cells on the test split.
Instruction issued by the benchmark
Answer-format rephrasing
Please select the correct answer from the options above.
Return only the final option letter, with no explanation.
Please answer yes or no.
Return exactly one word: yes or no.
Answer the question using a single word or phrase.
Return only the final short answer, without a full sentence.
Answer the question directly.
Return only the final answer, without a full sentence.
Appendix
Table 10: The answer-format rephrasings the debiasing step contrasts against, one representative benchmark instruction per row. Variant wordings of the same instruction map to the same rephrasing.
Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.
The emergence of Large Language Models (LLMs) has driven rapid progress in multi-modal learning, particularly in the development of Large Vision-Language Models (LVLMs). However, existing LVLM training paradigms place excessive reliance on the LLM component, giving rise to two critical robustness challenges: language bias and language sensitivity. To address both issues simultaneously, we propose a novel Self-Critical Inference (SCI) framework that extends Visual Contrastive Decoding by conducting multi-round counterfactual reasoning through both textual and visual perturbations. This process further introduces a new strategy for improving robustness by scaling the number of counterfactual rounds. Moreover, we also observe that failure cases of LVLMs differ significantly across models, indicating that fixed robustness benchmarks may not be able to capture the true reliability of LVLMs. To this end, we propose the Dynamic Robustness Benchmark (DRBench), a model-specific evaluation framework targeting both language bias and sensitivity issues. Extensive experiments show that SCI consistently outperforms baseline methods on DRBench, and that increasing the number of inference rounds further boosts robustness beyond existing single-step counterfactual reasoning methods.
Kaihua Tang, Jiaxin Qi, Jinli Ou +2
Tongji University, China · Computer Network Information Center, CAS, China · University of Chinese Academy of Sciences, China +1
Large Vision-Language Models (LVLMs) have achieved remarkable performance across a wide range of tasks; however, they often inherit social biases from their training data, resulting in biased behavior when processing portraits from different social groups. Existing debiasing approaches typically compare token probabilities between the original and biased generations during decoding, but they are fundamentally limited by their reliance on a single, stereotyped viewpoint and fail to account for the diversity of social perspectives. Inspired by the social science principle that diversity fosters fairness, we propose Counterfactual Ensemble Decoding (CED), a novel framework that constructs multi-group counterfactual perspectives within the visual representation space and integrates them during decoding to promote equitable model behavior. CED first performs counterfactual steering in the visual space by identifying semantic directions associated with each social group and generating counterfactual representations along these directions, thereby offering diverse perspectives that disrupt stereotypical narratives. During decoding, CED locates the decoder layer exhibiting the greatest divergence among these perspectives and ensembles their token distributions using uncertainty-aware weights, prioritizing high-confidence tokens from different groups to yield a more balanced probability distribution that guides fairer generation. Extensive experiments on three social bias evaluation benchmarks demonstrate that \tool achieves substantial improvements over leading baselines, reducing bias by up to 47.97% across scenarios involving occupations, descriptors, and persona traits. Moreover, CED also preserves the core capabilities of the original model with minimal degradation.
Yisong Xiao, Aishan Liu, Yongxin Huang +6
State Key Lab of Software Development Environment, Beihang University, Beijing 100191, China
Large Vision-Language Models (LVLMs) extend large language models with visual understanding, but remain vulnerable to hallucination, where outputs are fluent yet inconsistent with images. Recent studies link this issue to language bias-the tendency of LVLMs to over-rely on text while neglecting visual inputs. Yet most analyses remain empirical without uncovering its underlying cause. In this paper, we provide a systematic study of language bias and identify its root in modality misalignment during training. Our analysis shows that both Visual Instruction Tuning (VIT) and Direct Preference Optimization (DPO) often prioritize textual improvements, which may cause LVLMs to overly lean toward language modeling rather than balanced multimodal understanding. To address this, we propose two simple yet effective methods: Language Bias Regularization (LBR) which mitigates language bias through regularization during instruction tuning, and Language Bias Penalty (LBP), which penalizes language bias in the DPO training process. Extensive experiments across diverse models and benchmarks demonstrate the effectiveness of our approach. LBR consistently improves performance on over ten general benchmarks, while LBP significantly reduces hallucination and improves trustworthiness. Together, these methods not only mitigate language bias but also advance the overall alignment of LVLMs, all without introducing any additional data or auxiliary models. Our code is publicly available at https://github.com/lab-klc/LVLM-Language-Bias.