Evidence-Order Calibration for Selective Visual Reasoning under Progressive Loss of Question-Critical Evidence

Muhamathu Ameer Ali Aacaas Muhamath

cs.CVAug 29, 2026

Evidence-Order Calibration for Selective Visual Reasoning under Progressive Loss of Question-Critical Evidence

Authors: Muhamathu Ameer Ali Aacaas Muhamath

Organizations: Department of Electrical Engineering University of Moratuwa Moratuwa, Sri Lanka

Abstract

Vision-language model (VLM) confidence may change in aggregate when visual evidence is degraded while remaining structurally inconsistent within individual examples. We study answer-level reliability along five-step, question-conditioned evidence-loss trajectories. Using a frozen Qwen2.5-VL-3B-Instruct model, we construct 176 accepted GQA-derived trajectories (880 masking conditions) by progressively masking scene-graph-localized question-critical regions. Native sequence confidence has an evidence monotonicity violation rate (EMVR) of 0.436, and 92.0% of trajectories contain at least one adjacent violation. A matched non-critical-region control shows that full critical masking reduces accuracy by 28.2 percentage points, compared with 0.6 points for equally sized non-critical masks; the paired difference is 27.6 points (95% CI [20.0, 34.7]). We train a lightweight post-hoc reliability head on frozen hidden states, sequence confidence, and entropy. Adding evidence-order supervision to binary cross-entropy (BCE) reduces masking EMVR from 0.330 to 0.303 (paired difference -0.027, 95% CI [-0.044, -0.010]). The same mask-trained objective reduces EMVR from 0.449 to 0.402 on held-out question IDs under unseen local Gaussian blur (difference -0.0468, 95% CI [-0.0739, -0.0199]). AUROC, Brier, and AURC differences between the two learned heads are statistically inconclusive, and native confidence remains stronger for selective-risk ranking. The results separate evidence-order consistency from conventional correctness discrimination rather than establishing generic confidence superiority.

Figures & tables

Figure 1: Study design. Progressive, local, question-critical evidence loss produces supervision for a post-hoc reliability head while the answer-generating VLM remains frozen.

Figure 2: Aggregate behavior under progressive masking of question-critical evidence. Accuracy and sequence log-confidence decrease while entropy rises.

Figure 3: Example native-confidence violation. The answer is initially correct, changes after additional critical masking, and confidence is non-monotonic across the trajectory.

Intervention	Acc. clean	Acc. full	Acc. drop
Critical	0.641	0.359	0.282
Matched non-critical	0.641	0.635	0.006
Paired difference	critical $-$ non-critical		0.276
95% CI			[0.200, 0.347]

Table 1: Matched control on 170 valid paired trajectories. Changes are full-mask minus clean except accuracy drop, which is clean minus full.

Figure 4: Critical versus matched non-critical masking on 170 paired trajectories. Non-critical curves remain nearly flat in aggregate despite a high directional EMVR caused by small fluctuations.

Score/model	Accuracy	AUROC	AUPRC	Brier	EMVR	TVR	AURC
Native confidence	–	0.7689	–	–	0.4361	–	0.2654
BCE-only	0.6104	0.6743	0.7237	0.2268	0.3298	0.8127	0.3151
BCE + order	0.6036	0.6762	0.7222	0.2256	0.3027	0.7673	0.3158

Table 2: Frozen primary masking results. Lower is better for Brier, EMVR, trajectory violation rate (TVR), and AURC. Native confidence is a score, so learned-head accuracy and Brier are not applicable.

Score/model	Accuracy	AUROC	AUPRC	Brier	EMVR	TVR
Native confidence	–	0.7010	0.7110	–	0.4207	0.8921
BCE-only	0.5979	0.6049	0.6142	0.3521	0.4492	0.9089
BCE + order	0.6026	0.6205	0.6300	0.3383	0.4022	0.8863
Order $-$ BCE EMVR	$-0.04678$ ; 95% CI [ $-0.07386$ , $-0.01989$ ]

Table 3: Mask-train

\rightarrow

blur-test grouped-cross-validation means. Lower is better for Brier, EMVR, and TVR. The statistically supported paired result is the order-minus-BCE EMVR difference; other metric intervals cross zero.

Figure 5: EMVR under the primary masking experiment and held-out blur transfer. The order head improves on BCE in both settings; native confidence is shown as a behavioral reference, not an identically trained model.

Figure 6: Masking risk–coverage curves. Native confidence ranks correctness better than either learned head despite its higher evidence-order violation rate.

		Critical masking		Mask $\rightarrow$ blur
Features	Objective	AUROC	EMVR	AUROC	EMVR
Confidence	BCE	0.7614	0.4364	0.7010	0.4207
Entropy	BCE	0.7515	0.4136	0.6829	0.4135
Confidence + entropy	BCE	0.7612	0.4278	0.6959	0.4107
Hidden only	BCE	0.6152	0.4093	0.5917	0.4564
HS	BCE	0.6298	0.4037	0.6068	0.4421

Table 4: P20 feature/objective ablations (means). These are diagnostic ablations, not replacements for the frozen primary results. HS denotes hidden states plus native signals.

Selective Prediction for VLMs

See all 12 papers

Evidence-RL: Towards Evidence-intensive Visual Reasoning
Aug 8, 2026Haojie Huang, Xinlei Yu, Chengming Xu +6Vision-Language Grounding Visually Grounded Reasoning
National University of Singapore · Zhejiang University · Fudan University +2
The Mirage of Calibrated Confidence: Trajectory-Independence of Verbalized Confidence in Vision-Language Models
Sep 16, 2026Jisoo Yang, Jaeho Han, Trung X. Pham +1VLM Evaluation Confidence Estimation in Language Models
Department of Artificial Intelligence, Chung-Ang University, Republic of Korea · Van Lang University, Ho Chi Minh City, Vietnam
Look Where It Counts: A Free, Label-Free Visual Evidence Signal for Fine-Grained Vision-Language Reasoning
Sep 21, 2026Santi Ram Tiwari, Nihal Naik, Devbrat Pandey +1Vision-Language Grounding VLM Reasoning
KGraph AI Solutions Pvt. Ltd. Bangalore, India – 560016

Aug 8, 2026J/K move · Enter open · S save

Evidence-RL: Towards Evidence-intensive Visual Reasoning

Haojie Huang, Xinlei Yu, Chengming Xu, Zhangquan Chen+5

National University of Singapore · Zhejiang University · Fudan University+2

Vision-Language Grounding Visually Grounded Reasoning RL for VLMs

Open paper AskPDF arXiv

Vision-Language Models (VLMs) should answer from concrete image evidence rather than language priors, dataset shortcuts, or irrelevant visual context. Existing perception-aware post-training methods encourage image use through global perturbations or attention proxies, but they do not test whether a sampled answer causally depends on the local evidence that supports it. We propose Counterfactual Evidence Disentanglement (CED), a training-time evidence audit for VLM grounding. For each response, CED neutralizes an object-centric Evidence Region and compares the resulting support drop against matched non-evidence Regions. We combine this signal with answer correctness inside GRPO, rewarding correct answers that rely on the evidence path rather than shortcut or nuisance paths. CED uses weak object-level proposals, requires no question-specific evidence annotations, and adds no inference-time overhead. Across nine public benchmarks and four backbones, CED outperforms prior RL-based post-training methods, with targeted analyses verifying its object-centric signal.

Abstract

Selective Prediction for VLMs

Explore similar work

Evidence-RL: Towards Evidence-intensive Visual Reasoning

The Mirage of Calibrated Confidence: Trajectory-Independence of Verbalized Confidence in Vision-Language Models

Look Where It Counts: A Free, Label-Free Visual Evidence Signal for Fine-Grained Vision-Language Reasoning