Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework

Kaihua Tang; Jiaxin Qi; Jinli Ou; Yuhua Zheng; Jianqiang Huang

cs.CVMar 8, 2026

Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework

Authors: Kaihua Tang, Jiaxin Qi, Jinli Ou, Yuhua Zheng, Jianqiang Huang

Organizations: Tongji University, China · Computer Network Information Center, CAS, China · University of Chinese Academy of Sciences, China · HIAS, University of Chinese Academy of Sciences, China

Abstract

The emergence of Large Language Models (LLMs) has driven rapid progress in multi-modal learning, particularly in the development of Large Vision-Language Models (LVLMs). However, existing LVLM training paradigms place excessive reliance on the LLM component, giving rise to two critical robustness challenges: language bias and language sensitivity. To address both issues simultaneously, we propose a novel Self-Critical Inference (SCI) framework that extends Visual Contrastive Decoding by conducting multi-round counterfactual reasoning through both textual and visual perturbations. This process further introduces a new strategy for improving robustness by scaling the number of counterfactual rounds. Moreover, we also observe that failure cases of LVLMs differ significantly across models, indicating that fixed robustness benchmarks may not be able to capture the true reliability of LVLMs. To this end, we propose the Dynamic Robustness Benchmark (DRBench), a model-specific evaluation framework targeting both language bias and sensitivity issues. Extensive experiments show that SCI consistently outperforms baseline methods on DRBench, and that increasing the number of inference rounds further boosts robustness beyond existing single-step counterfactual reasoning methods.

Figures & tables

Figure 1: (a) and (b) are real DRBench examples suffering from language sensitivity and bias issues; (c) shows the overall proportion of different types of non-robust samples across all 6 datasets under two commonly used LVLMs; (d) demonstrates a novel test-time scaling strategy of robustness regarding the increased counterfactual rounds in the proposed SCI.

Subset Size	B Subset	S Subset	BS Subset	Overlap
LLaVA-NeXT (MCQ)	1810	1005	2476	339
LLaVA-NeXT (Others)	345	582	794	133
LLaVA-NeXT (Overall)	2155	1587	3270	472
Qwen2-VL (MCQ)	1080	252	1243	89
Qwen2-VL (Others)	327	311	513	125
Qwen2-VL (Overall)	1407	563	1756	214

Table 1: The size of each subset in constructed DRBench. The overall number of samples across all 6 datasets is 13251, with MCQ and Others categories being 10632 and 2619, respectively.

Method	B Subset			S Subset			BS Subset
Method	MCQ	Others	Overall	MCQ	Others	Overall	MCQ	Others	Overall
LLaVA-NeXT	0.0	0.0	0.0	39.2	37.63	38.63	15.91	27.58	18.75
LLaVA-NeXT-TIE	12.98	23.48	14.66	39.00	57.56	45.81	21.89	44.21	27.31
LLaVA-NeXT-VCD	12.65	25.51	14.71	40.50	56.53	46.38	22.54	44.58	27.89
LLaVA-NeXT-M3ID	16.91	25.22	18.24	39.90	56.36	45.94	24.15	44.33	29.05
LLaVA-NeXT-SCI 3 (ours)	21.22	35.36	23.48	39.60	60.31	47.20	27.14	50.13	32.72

Table 2: Experiments on B(ias) Subset, S(ensitivity) Subset, and BS Subset of the proposed DRBench. Bold texts indicate the best result of each column and underline texts indicate the second best result.

Construction Model	Methods	MCQ	Others	Overall
LLaVA-NeXT	LLaVA-NeXT-Original	15.91	27.58	18.75
	LLaVA-NeXT-SCI 5	28.80	51.01	34.19
	Qwen2-VL-Original	59.29	63.48	60.31
	Qwen2-VL-SCI 5	61.15	67.88	62.78
Qwen2-VL	Qwen2-VL-Original	10.78	23.59	14.52
	Qwen2-VL-SCI 5	28.00	33.14	29.50

Table 3: Ablation on cross-model BS Subset evaluation.

Method	Single Dataset						Question Type
Method	MMB-C	MMB-E	MME	CCB	MMS	ViLP	MCQ	Others	Overall
LLaVA-NeXT	78.0	79.72	79.57	47.0	44.75	51.53	70.12	71.86	70.46
LLaVA-NeXT-TIE	78.28	80.28	77.30	45.65	46.00	53.19	70.36	70.68	70.42
LLaVA-NeXT-VCD	78.38	80.28	78.09	46.63	45.00	54.31	70.44	71.55	70.66
LLaVA-NeXT-M3ID	78.31	80.18	78.62	45.89	45.92	54.03	70.36	71.86	70.66
LLaVA-NeXT-SCI 5 (ours)	78.21	80.08	80.15	46.20	45.75	53.06	70.32	72.70	70.79

Table 4: Experiments on MMB(ench-Dev)-C/E(N-V11), MME, CCB(ench), MMS(tar), and ViLP indicate that SCI has more consistent improvement than TDE/VCD/M3ID on those real-world LVLM benchmarks (using 80% test splits). Blue texts indicate an improvement over the baseline.

Base	VC-C	VC-N	TC-V1	TC-V2	MCQ	Others	Overall
✓					10.78	23.59	14.52
	✓				8.77	18.52	11.62
		✓			10.62	25.15	14.86
			✓		10.38	24.37	14.46
				✓	12.07	23.00	15.26
✓	✓				21.72	29.43	23.97

Table 5: Ablation experiments for different counterfactual logits combinations using Qwen2-VL on BS Subset.

Figure 2: Investigating the test-time scaling effect on robustness with respect to the number of inference rounds on B/S/BS subsets across different question types and LVLMs.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Methods	Constraint	Original	B Subset	S Subset	BS Subset
Qwen2-VL	NA	81.12	6.10	37.59	15.46
Qwen2-VL-SCI 5	✗	68.93	26.16	34.04	27.63
Qwen2-VL-SCI 5	✓	81.03	29.65	40.43	32.55

Appendix

Table 6: Ablation study for the adaptive plausibility constraint. To evaluate the effect of adaptive plausibility constraint, we conducted experiments on validation sets of original 6 datasets together with B(ias)/S(ensitive)/BS Subsets.

Method	Qwen2-VL	Qwen2-VL-SCI 3	Qwen2-VL-SCI 5	Qwen2-VL-SCI 7
Inference Time (w/o batch inference)	540.47ms	1599.65ms	2707.16ms	3611.18ms
Inference Time (w/ batch inference)	540.47ms	697.24ms	978.14ms	1342.86ms

Appendix

Table 7: We report the average inference time per sample on the MMStar dataset using one A800 GPU to illustrate the computational overhead introduced by SCI. Note that the baseline speed w/o batch inference sequentially conduct each counterfactual inference round, while w/ batch inference, all counterfactual inference rounds are conducted in one batch. Therefore, the later is significantly faster than the baseline speed.

Methods	Hyperparameters	MCQ	Others	Overall
Qwen2-VL	-	11.97	22.38	15.46
Qwen2-VL-SCI 5	$\tau_{1}=2.0$ $\tau_{2}=0.2$	33.45	30.77	32.55
Qwen2-VL-SCI 5	$\tau_{1}=2.0$ $\tau_{2}=2.0$	22.89	27.97	24.59
Qwen2-VL-SCI 5	$\tau_{1}=2.0$ $\tau_{2}=1.0$	26.06	29.37	27.17
Qwen2-VL-SCI 5	$\tau_{1}=2.0$ $\tau_{2}=0.5$	32.39	30.77	31.85
Qwen2-VL-SCI 5	$\tau_{1}=20$ $\tau_{2}=0.2$	3.87	18.88	8.89

Appendix

Table 8: Ablation study for temperature scaling hyperparameters

\tau_{1}

and

\tau_{2}

of SCI. Experiments are conducted under validation set of BS Subset.

Method	Bias Subset			Sensitivity Subset			BS Subset
Method	MCQ	Others	Overall	MCQ	Others	Overall	MCQ	Others	Overall
LLaVA-NeXT	0.0	0.0	0.0	39.2	37.63	38.63	15.91	27.58	18.75
LLaVA-NeXT-TCF-V1	3.20	6.38	3.71	36.62	26.80	33.02	14.86	19.65	16.02
LLaVA-NeXT-TCF-V2	5.80	8.70	6.26	24.08	33.51	27.54	9.77	24.56	13.36
LLaVA-NeXT-TCF-V3	3.09	3.19	3.11	38.61	34.54	37.11	15.99	25.31	18.26
LLaVA-NeXT-VCF-Color0	4.59	4.06	4.50	27.26	23.54	25.90	13.85	18.01	14.86

Appendix

Table 9: The complete experiments on Bias Subset, Sensitivity Subset, and BS Subset of the DRBench across two widely used base LVLMs demonstrate the effectiveness of the proposed SCI framework. Bold texts indicate the best result of each column.

Method	Single Dataset						Question Type
Method	MMB-C	MMB-E	MME	CCB	MMS	ViLP	MCQ	Others	Overall
LLaVA-NeXT	78.0	79.72	79.57	47.0	44.75	51.53	70.12	71.86	70.46
LLaVA-NeXT-TC-V1	77.46	79.95	76.20	46.75	43.92	51.53	69.87	69.42	69.78
LLaVA-NeXT-TC-V2	77.44	77.51	78.78	46.20	42.08	50.14	68.68	70.90	69.12
LLaVA-NeXT-VC-C0	29.97	31.85	50.29	27.02	25.08	28.47	29.66	44.29	32.55
LLaVA-NeXT-VC-N500	30.69	33.08	48.29	28.25	25.0	29.03	30.55	42.99	33.01

Appendix

Table 10: Experiments on MMB(ench-Dev)-C/E(N-V11), MME, CCB(ench), MMS(tar), and ViLP including all counterfactual inference results used by SCI 5 . Blue texts indicate an improvement over the baseline.

Figure 3: The list of all TC-V1 prompts that add an additional system prompt instructing the model to focus on image details.

Figure 4: The list of all TC-V2 prompts that further modify the system prompt’s language from English to Chinese or vice versa.

Figure 5: The list of all TC-V3 prompts that inject identity information by prompting the model to respond as a clever student.

VLM Robustness

Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.

See all 454 papers

Same Answer, Different Representations: Hidden instability in VLMs
Feb 6, 2026Farooq Ahmad Wani, Alessandro Suglia, Rohit Saxena +6VLM Evaluation VLM Robustness
Sapienza University of Rome · University of Edinburgh · University of Oxford +5
What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models
Mar 13, 2026Sen Nie, Jie Zhang, Zhongqi Wang +3VLM Robustness Adversarial Robustness
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · University of Chinese Academy of Sciences, Beijing 100049, China
MAGIC: Margin-gated Contrastive Debiasing with Evidence Restoration for VLM Test-time Robustness Scaling
Oct 23, 2026Shunshun Gu, Enze Zhang, Ming Pei +2VLM Robustness Test-Time Scaling for VLMs
University of Wisconsin–Madison · University of North Carolina at Chapel Hill

Feb 6, 2026J/K move · Enter open · S save

Same Answer, Different Representations: Hidden instability in VLMs

Farooq Ahmad Wani, Alessandro Suglia, Rohit Saxena, Aryo Pradipta Gema+5

Sapienza University of Rome · University of Edinburgh · University of Oxford+5

VLM Evaluation VLM Robustness Vision-Language Models Neural Network Robustness Adversarial Robustness Representation Geometry in VLMs

Open paper AskPDF arXiv

The robustness of Vision Language Models (VLMs) is commonly assessed through output-level invariance, implicitly assuming that stable predictions reflect stable multimodal processing. In this work, we argue that this assumption is insufficient. We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness (spatial consistency of vision tokens), alongside standard label-based metrics. Applying this framework to modern VLMs across the SEEDBench, MMMU, and POPE datasets reveals three distinct failure modes. First, models frequently preserve predicted answers while undergoing substantial internal representation drift; for perturbations such as text overlays, this drift approaches the magnitude of inter-image variability, indicating that representations move to regions typically occupied by unrelated inputs despite unchanged outputs. Second, robustness does not improve with scale; larger models achieve higher accuracy but exhibit equal or greater sensitivity, consistent with sharper yet more fragile decision boundaries. Third, we find that perturbations affect tasks differently: they harm reasoning when they disrupt how models combine coarse and fine visual cues, but on the hallucination benchmarks, they can reduce false positives by making models generate more conservative answers.

Abstract

VLM Robustness

Explore similar work

Same Answer, Different Representations: Hidden instability in VLMs

What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models

MAGIC: Margin-gated Contrastive Debiasing with Evidence Restoration for VLM Test-time Robustness Scaling