This paper studies the problem of test-time robustness scaling for large vision-language models (LVLMs), which aims to mitigate language bias during inference. Existing approaches typically adopt test-time scaling techniques such as contrastive decoding to improve inference performance. Despite the progress, the performance of these approaches is far from satisfactory due to intentional evidence removal during perturbations and redundant inference cost. Towards this end, we propose a novel approach named Margin-gated Contrastive Debiasing with Evidence Restoration (MAGIC). The core of our MAGIC is to evaluate the decision margins of LVLMs to selectively activate textual and visual perturbation branches for efficient debiased inference. In particular, our MAGIC first measures the prediction margins of crucial tokens, and applies a gating function to identify low-confidence responses that are susceptible to language biases. For selected samples, we integrate the token logits from multiple textual and image perturbation branches with the original ones for debiased decision. More importantly, we selectively restore the most informative corrupted views with sharp prediction distributions for ambiguous samples after debiasing, enabling evidence recovery for discriminative decision-making. We further utilize a calibration dataset to actively identify and prune ineffective branches during contrastive debiasing for cost reduction. Extensive experiments on benchmark datasets validate the effectiveness of the proposed MAGIC in comparison with extensive baselines.
Figures & tables
Figure 1: Overview of MAGIC . Each step fires only when the margin m still leaves the decision contested, so a sample that clears a gate exits with the scores it already has. Thresholds are fitted offline on the validation split.
Figure 2: Utility of the ungated debiasing step against the base decision margin, on biased and natural samples. Bands are 95% intervals and shading marks the margins over which the two curves separate.
Figure 3: Error of the corrector of Eq. ( 1 ) as h^ averages k branches, with an a+b/k fit and its asymptote a dashed.
Subset Size
B Subset
S Subset
BS Subset
Overlap
LLaVA-NeXT (MCQ)
1810
1005
2476
339
LLaVA-NeXT (Others)
345
582
794
133
LLaVA-NeXT (Overall)
2155
1587
3270
472
Qwen2-VL (MCQ)
1080
252
1243
89
Qwen2-VL (Others)
327
311
513
125
Table 1: The size of each subset in constructed DRBench. The overall number of samples across all six datasets is 13,251 , with the MCQ and Others categories being 10,632 and 2,619 , respectively.
Bias Subset
Sensitivity Subset
BS Subset
Model
Method
MCQ
Others
All
MCQ
Others
All
MCQ
Others
All
Passes
Base
5.37
8.56
6.11
38.10
34.41
36.06
10.78
23.59
14.52
1
TIE
16.20
16.82
16.35
45.63
36.66
40.67
20.27
27.29
22.32
2
VCD
15.74
21.71
17.13
46.83
40.84
43.52
20.11
30.41
23.12
2
M3ID
19.81
21.71
20.26
47.22
41.16
43.87
23.65
30.60
25.68
2
SCI 3
21.67
26.30
22.74
44.05
42.44
43.16
24.54
32.75
26.94
3
Table 2: Experiments on the B(ias) Subset, S(ensitivity) Subset, and BS Subset of DRBench. Baseline results are from Tang et al. (2026a) . Passes is the mean per-sample forward count. We bold the best result in each column and underline the second best.
Single Dataset
Question Type
Method
MMB-C
MMB-E
MME
CCB
MMS
ViLP
MCQ
Others
Overall
Qwen2-VL
85.26
86.36
87.89
73.22
59.50
56.53
80.91
79.27
80.58
Qwen2-VL-TIE
86.00 ↑ 0.74
86.59 ↑ 0.23
86.52 ↓ 1.37
73.84 ↑ 0.62
59.00 ↓ 0.50
57.08 ↑ 0.55
81.30 ↑ 0.39
78.43 ↓ 0.84
80.73 ↑ 0.15
Qwen2-VL-VCD
86.05 ↑ 0.79
86.56 ↑ 0.20
86.41 ↓ 1.48
73.77 ↑ 0.55
60.08 ↑ 0.58
57.92 ↑ 1.39
81.42 ↑ 0.51
78.58 ↓ 0.69
80.86 ↑ 0.28
Qwen2-VL-M3ID
85.69 ↑ 0.43
86.46 ↑ 0.10
86.10 ↓ 1.79
73.96 ↑ 0.74
59.75 ↑ 0.25
57.78 ↑ 1.25
81.25 ↑ 0.34
78.31 ↓ 0.96
80.67 ↑ 0.09
Qwen2-VL-SCI 5
85.97 ↑ 0.71
86.67 ↑ 0.31
87.36 ↓ 0.53
73.59 ↑ 0.37
59.92 ↑ 0.42
58.06 ↑ 1.53
81.39 ↑ 0.48
79.31 ↑ 0.04
80.98 ↑ 0.40
Table 3: Experiments on the six real-world LVLM datasets, MMB(ench-Dev)-C/E(N-V11), MME, CCB(ench), MMS(tar), and ViLP (using 80% test splits). Baseline results are from Tang et al. (2026a) .
Figure 4: Accuracy against compute on the biased subset of Qwen2-VL as t1 is swept. Each baseline sits at one fixed budget, while MAGIC traces a frontier. The star marks the deployed setting.
Figure 5: Two biased-subset samples MAGIC repairs on Qwen2-VL, at the fitted thresholds. Each note keeps its stage’s color from Figure 1 and is dashed when that gate never opened. ✓ and ✗ flag the answer a stage leaves, m is the top-two margin, and the committed answer is bold.
Model
Method
B
S
BS
Passes
Base
0.49
34.65
12.56
1
TIE
5.71
32.44
13.47
2
VCD
7.19
32.60
14.27
2
M3ID
8.74
33.54
15.18
2
SCI 3
21.37
41.42
24.96
3
SCI 5
22.43
47.56
26.44
5
Table 4: B/S/BS Overall accuracy on two more recent backbones. We bold the best result in each column and underline the second best.
Model
Configuration
MCQ
Others
Overall
Passes
MAGIC w/o CD
13.17
25.39
16.70
2.2
MAGIC w/o BC
31.59
34.77
32.51
4.0
MAGIC w/o AF
31.11
32.42
31.49
3.2
MAGIC w/o SR
30.40
33.59
31.32
3.8
Qwen2-VL
MAGIC
32.30
35.16
33.13
4.9
MAGIC w/o CD
17.07
38.44
22.28
1.9
Table 5: Ablation on the BS Subset of DRBench. CD: Contrastive Debiasing, BC: Blank Contrast, AF: Answer-Format Contrast, SR: Selective Restoration. Passes is the mean per-sample forward count.
Model
Value
MCQ
Others
Overall
Passes
Debiasing gate t1
0.5
18.81
29.30
21.84
3.0
1.0
24.68
31.64
26.69
3.7
2.0
31.03
34.38
32.00
4.4
3.0
32.30
34.96
33.07
4.8
5.0
32.38
35.74
33.35
5.0
Table 6: Sensitivity to the two allocation thresholds on Qwen2-VL, BS Subset of DRBench. We bold the best result in each column within a block.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Four further Qwen2-VL samples, one per exit of Algorithm 1 .
Model
Value
MCQ
Others
Overall
Passes
Debiasing gate t1
0.5
18.81
29.30
21.84
3.0
1.0
24.68
31.64
26.69
3.7
2.0
31.03
34.38
32.00
4.4
3.0
32.30
34.96
33.07
4.8
5.0
32.38
35.74
33.35
5.0
Appendix
Table 7: Sensitivity to the two allocation thresholds on the BS Subset of DRBench, on both backbones. We bold the best result in each column within a block.
Biased (BS)
Natural
Method
MCQ
Others
MCQ
Others
SCI 5
5.0
5.0
5.0
5.0
SCI 7
7.0
7.0
7.0
7.0
MAGIC (Qwen2-VL)
5.0
4.5
1.5
2.8
MAGIC (LLaVA-NeXT)
4.9
4.2
2.4
2.5
Appendix
Table 8: Average inference passes per sample.
Figure 7: Random and keyword-matched replacements decide as a blank image does. The keyed-full condition answers the question and is the positive control.
Qwen2-VL
LLaVA-NeXT
Method
S
BS
S
BS
VACoDe-A (max contrast)
46.91
22.28
40.99
26.78
VACoDe-A (min contrast)
38.55
32.20
31.09
29.38
CHASD-A (uncertainty-gated)
48.73
30.53
40.30
30.47
gate only (blank contrast)
45.35
29.29
39.52
30.00
MAGIC
48.73
32.30
40.99
31.13
Appendix
Table 9: Our answer-level adaptations of the nearest adaptive-contrast methods, MCQ cells on the test split.
Instruction issued by the benchmark
Answer-format rephrasing
Please select the correct answer from the options above.
Return only the final option letter, with no explanation.
Please answer yes or no.
Return exactly one word: yes or no.
Answer the question using a single word or phrase.
Return only the final short answer, without a full sentence.
Answer the question directly.
Return only the final answer, without a full sentence.
Appendix
Table 10: The answer-format rephrasings the debiasing step contrasts against, one representative benchmark instruction per row. Variant wordings of the same instruction map to the same rephrasing.
Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.