The Evidence Anchoring Boundary: Where Visual Interventions Matter and When They Help in Multimodal LLMs

Qiji Zhou; Junyi Hu; Peiyu Wang; Yue Zhang

cs.CVOct 9, 2026

The Evidence Anchoring Boundary: Where Visual Interventions Matter and When They Help in Multimodal LLMs

Authors: Qiji Zhou, Junyi Hu, Peiyu Wang, Yue Zhang

Organizations: Westlake University · Duke University · Institute of Automation, Chinese Academy of Sciences

Abstract

Visual interventions provide a training-free way to reduce hallucinations in multimodal large language models. However, an intervention may have little leverage at some layers and may damage a correct answer even when it changes the output. We investigate where visual interventions retain leverage and when they improve predictions. First, we apply the same visual-residual edit at successive decoder layers, matching its RMS strength across depth. Across six checkpoints from three model families, we observe a persistent drop in intervention effect and define its operational onset for this edit as the Evidence Anchoring Boundary (EAB). With boundaries fixed on calibration queries, held-out effects on disjoint evaluation queries before EAB are $13.78\times$ -- $26.10\times$ larger than those after it. Comparison with intermediate readouts shows that the edit can lose leverage before answer preferences stabilize. Second, we compare counterfactual subtraction across presence, attribute, and relation questions. Subtraction can correct a misleading preference, leave a decision unchanged, or remove a preference supporting the correct answer. Based on these observations, we develop calibrated selective routing, which preserves the original prediction unless calibration supports an intervention for the question type. It improves full-set attribute accuracy by $1.56$ -- $1.81$ percentage points across three checkpoints, while universal subtraction is neutral or harmful.

Figures & tables

Figure 1: From intervention leverage to selective visual intervention. Top: training-free intervention sites. Middle (Q1): the Evidence Anchoring Boundary (EAB) marks where the tested visual-residual edit becomes persistently weak under equal-RMS perturbations across depth, potentially before answer stabilization. Bottom (Q2): an edit can be corrective, redundant, or destructive depending on the preference it removes, motivating calibrated selective intervention.

Question	Dataset / task	Use in this paper
Q1: layer-wise effect	Shared sampled queries	EAB and answer-readout comparison
Q2: presence	POPE; TVQA-Existence	Confidence-conditioned diagnostics
Q2: attributes	AMBER claim pairs	Outcome analysis and routing
Q2: relations	AMBER claim pairs	Outcome analysis

Table 1: Main evaluation map. Q1 uses 120 shared queries per checkpoint (48 calibration, 72 disjoint evaluation); Q2 coverage varies by checkpoint as recorded in Appendix A.1 .

Figure 2: EAB identifies where the tested direct visual edit loses leverage. (a) Checkpoint-level boundaries frozen on calibration queries identify a persistent low-effect regime on disjoint evaluation queries under the same equal-RMS strength control. (b) Answer-readout reliability can continue to increase after this measured drop; ACP marks answer-readout stabilization. (c) Fully held-out pre-EAB effects are

13.78\times

–

26.10\times

post-EAB effects across six checkpoints.

Figure 3: The removed preference explains three intervention outcomes. (a) Presence:

G

adds little consistent discrimination between correct and incorrect Yes predictions after conditioning on

M

. (b) Schematic yellow-minus-red score differences illustrate

\Delta G=\Delta M-\Delta M_{\mathrm{cf}}

: subtraction preserves, corrects, or damages the ranking according to the removed preference. The values are illustrative, not measured. (c) Observed pair-accuracy changes for hard attributes, inverse-spatial relations, and direct-contact relations; positive values favor

G

. Error bars show 95% image-cluster confidence intervals, with uncertainty estimated over image-level dependence structures.

Checkpoint	Presence: residual $\operatorname{AUC}(G\mid M)$		Paired tasks: accuracy change $G-M$ (percentage points)
	POPE- adversarial	TVQA non-COCO	Hard attributes (misleading)	Full attributes (mixed)	Direct contact (task-relevant)	Inverse spatial (near-flat)
LLaVA-1.5-7B	$.516$	$.552$	$+13.44$ $[10.17,16.89]$	$-0.42$ $[-1.64,0.84]$	$-5.32$ $[-12.90,3.16]$	$-2.3$
LLaVA-1.5-13B	$.514$	$.510$	$+7.68$ $[3.15,12.01]$	$-4.92$ $[-6.39,-3.36]$	$-15.96$ $[-24.47,-8.42]$	$-0.7$
Qwen2.5-VL-7B	—	$.505$	—	—	—	$-0.3$
InternVL3.5-8B	$.526$	$.539$	$+17.81$ $[6.85,28.77]$	$-0.33$ $[-2.83,2.17]$	—	$+0.4$

Table 2: Subtraction has different effects across question types for each evaluated checkpoint. Presence columns report residual

\operatorname{AUC}(G\mid M)

; all other columns report the accuracy change from replacing

M

with

G

, in percentage points. (Bracketed values are 95% image-cluster confidence intervals; — indicates that the checkpoint–task pair was not evaluated.)

Figure 4: Calibrated selective routing preserves the original score by default, using only calibration-supported operators for each question category. Question category and full-image/counterfactual probes restrict eligible operators. Calibration supports subtraction for suitable attribute questions or PEC in its validated presence setting; otherwise the router retains

M

. For presupposed questions, neutralization is followed by re-probing before a score adjustment is selected. All routing rules are fixed before evaluation, and evaluation labels are excluded from the decision.

Figure 5: Selective routing improves attributes; PEC has limited transfer. (a) Held-out AMBER attribute accuracy for the original margin

M

, global subtraction

G

, and the routed score. (b) Cached pairs illustrate rescue, break, and preservation through

\Delta G=\Delta M-\Delta M_{\mathrm{cf}}

. (c) Targeted-generation changes on LLaVA-7B/13B, with 95% image-cluster bootstrap intervals. (d) Frozen PEC effects across POPE-adversarial, TextVQA, and MME: PEC improves its calibrated POPE setting but transfers negatively or inconsistently. Complete results and paired tests are in Appendices F.1 – F.3 .

Comparison / design choice	In-domain evidence	Failure / implication
Use $G$ for every attribute (remove selective routing)	Routed score minus $M$ LLaVA-1.5-7B $+1.56$ LLaVA-1.5-13B $+1.81$ InternVL3.5-8B $+1.67$	Global $G$ minus $M$ LLaVA-1.5-7B $-0.42$ LLaVA-1.5-13B $-4.92$ InternVL3.5-8B $-0.33$
Use subtraction across question types	Presence residual AUC LLaVA-1.5-7B $.516$ LLaVA-1.5-13B $.514$ InternVL3.5-8B $.526$	Direct-contact accuracy change LLaVA-1.5-7B $-5.32$ LLaVA-1.5-13B $-15.96$
Apply frozen PEC across tasks	POPE-adversarial gain LLaVA-1.5-7B $+3.30$ LLaVA-1.5-13B $+3.90$ InternVL3.5-8B $+7.07$	Frozen transfer change Checkpoint POPE- random TextVQA MME LLaVA-1.5-7B $+0.8$ $-4.1$ $-9.9$ LLaVA-1.5-13B $+0.8$ $-2.2$ $+2.5$ Qwen2.5-VL-7B — $+0.5$ $-0.7$ InternVL3.5-8B $-0.3$ $-1.5$ $-6.0$
Retain the stability safeguard	Separate contribution not quantified.	Reject unstable corrections; no isolated accuracy gain is claimed.

Table 3: Evidence for selection and task-specific calibration across the evaluated checkpoints. Controlled comparisons and observed failure modes. Effects are percentage-point changes unless noted. (Attribute and POPE tests are in Appendix F.3 ; direct-contact intervals are in Table 2 .)

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Checkpoint	Q1 queries (EAB/ACP)	Presence claim sets	Attribute pairs	Relation pairs	Downstream evaluation
Base checkpoints used beyond Q1
LLaVA-1.5-7B	120	POPE: 3,000/3,000 TVQA: 2,000	2,378	contact: 94 spatial: 1,000	attribute router; targeted generation: 1,040; PEC: 3,000; CHAIR: 500
LLaVA-1.5-13B	120	POPE: 3,000/3,000 TVQA: 2,000	2,378	contact: 94 spatial: 1,000	attribute router; targeted generation: 1,077; PEC: 3,000; CHAIR: 500
Qwen2.5-VL-7B	120	POPE: 3,000/3,000 TVQA: 2,000	—	spatial: 1,000	PEC transfer: TextVQA/MME; POPE PEC omitted (two baseline false-Yes cases)
InternVL3.5-8B	120	POPE: 3,000/3,000 TVQA: 2,000	600	spatial: 1,000	attribute router; PEC: 3,000; CHAIR: 500
Q1-only scale replications

Appendix

Table 4: Claim-specific model and dataset coverage. Every Q1 checkpoint uses 120 queries (48 calibration/72 evaluation); Q2 and deployment claims are restricted to the populated cells to match their evaluated scope. (POPE counts are adversarial/random; — indicates that the checkpoint–task pair was not evaluated. Statistical audits report their own sample counts in Table 8 . LLaVA-NeXT-34B and Qwen2.5-VL-32B are Q1-only scale replications.)

Item	Frozen value / policy	Reproduction record
Environment and determinism
Hardware	$2\times$ NVIDIA A100-SXM4-80GB; CUDA driver 550.54.14	one checkpoint per GPU; no model sharding for reported base runs
Software	Python; PyTorch 2.8.0, Transformers 4.51.0, NumPy 1.26.4, SciPy 1.17.1	versioned dependency and environment records
Randomness	analysis seed 0; bootstrap seeds 0/1/2 where repeated	deterministic sample designs are cached
Experimental design and inference
EAB split	120 examples/checkpoint: 48 boundary calibration, 72 completely disjoint evaluation; one checkpoint-level EAB is frozen	cached split assignments and boundary-analysis records

Appendix

Table 5: Reproducibility checklist. Frozen policies and the records used to verify the reported experiments and artifacts, supporting reconstruction of the reported evaluation pipeline.

Checkpoint	LM depth	Frozen EAB	Relative EAB	Boundary resolution	Equal-RMS pre/post effect
					Ratio	95% CI
LLaVA-1.5-7B	32	12	$.375$	99.2%	$13.88\times$	$[11.52,16.49]$
LLaVA-1.5-13B	40	13	$.325$	99.2%	$13.78\times$	$[10.98,17.97]$
LLaVA-NeXT-34B	60	33	$.550$	100%	$26.10\times$	$[22.24,30.74]$
Qwen2.5-VL-7B	28	16	$.571$	100%	$17.54\times$	$[15.22,20.42]$
Qwen2.5-VL-32B	64	40	$.625$	97.5%	$18.24\times$	$[14.89,22.59]$

Appendix

Table 6: The operator-specific EAB pattern replicates across all six checkpoints. One checkpoint-level boundary is frozen from calibration queries before evaluation; relative EAB is that boundary divided by language-model depth. The resolution column summarizes the cached boundary audit, whereas the pre/post ratios and confidence intervals use the completely disjoint evaluation queries.

Comparison (depths; resolved)	$\Delta$ EAB (layers)	$\Delta$ relative EAB	Interpretation
LLaVA-1.5-13B $\rightarrow$ LLaVA-NeXT-34B 40 $\rightarrow$ 60 layers; 119/120 resolved	$+17.72$ $[16.25,19.12]$	$+.166$ $[.131,.199]$	later EAB in relative depth
Qwen2.5-VL-7B $\rightarrow$ Qwen2.5-VL-32B 28 $\rightarrow$ 64 layers; 117/120 resolved	$+22.64$ $[21.67,23.66]$	$+.019$ $[-.007,.045]$	relative EAB shift unresolved

Appendix

Table 7: Checkpoint-scale audit. (Shifts are larger minus smaller; bracketed values are 95% image-cluster confidence intervals.)

Figure 6: Spatial localization can persist after direct visual-residual leverage fades. Head-averaged last-query attention on one shared example; green marks the queried-object region.

Figure 7: Aggregate spatial-attention audit across four base checkpoints. Localization does not by itself identify whether a score intervention will improve the answer.

Figure 8: Prompt presupposition and neutralization are checkpoint-dependent operators.

Figure 9: Detailed attribute-routing and targeted-generation results. (a) The routed score improves full-set held-out attribute accuracy while global subtraction is neutral or harmful. (b) Point estimates show higher hallucination catch, small false-positive changes, and lower residual hallucination. Horizontal bars are 95% image-cluster intervals; paired tests are in Table 8 .

Figure 10: PEC has a calibrated presence operating range but does not transfer universally. Segments mark POPE gain regions; transfer whiskers are 95% paired intervals.

Figure 11: Frozen PEC transfer is heterogeneous across MME categories and checkpoints.

Outcome	Checkpoint	$n$	Effect (pp)	95% CI	$p$
Attribute subtraction ( $G-M$ )
All attributes	LLaVA-1.5-7B	2,378	$-0.42$	$[-1.64,+0.84]$	$.5491$
Hard attributes	LLaVA-1.5-7B	521	$+13.44$	$[+10.17,+16.89]$	$<10^{-4}$
All attributes	LLaVA-1.5-13B	2,378	$-4.92$	$[-6.39,-3.36]$	$<10^{-4}$
Hard attributes	LLaVA-1.5-13B	508	$+7.68$	$[+3.15,+12.01]$	$.0010$
All attributes	InternVL3.5-8B	600	$-0.33$	$[-2.83,+2.17]$	$.8946$

Appendix

Table 8: Paired significance audit. Effects are

A-B

percentage points; inference is image-cluster aware. (A dash in the

p

column indicates a ratio outcome for which only a cluster-bootstrap interval is reported.)

Figure 12: Concrete attribute decisions expose the routing condition. Counterfactual subtraction rescues, damages, or preserves a ranking according to the relative preference it removes. These examples illustrate the mechanism and are not prevalence estimates.

Figure 13: Representative grounded, borrowed-evidence, and localization-miss cases.

VLM Hallucination Mitigation

Vision-language model hallucination mitigation keeps generated claims grounded in visual evidence.

See all 228 papers

Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigation
Jun 29, 2026Xin Zou, Haolin Deng, Yibo Yan +5Multimodal Large Language Models Multimodal Hallucination
HKUST (GZ) · HKUST · OPPO AI Center
See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs
Feb 25, 2026Yongchang Zhang, Oliver Ma, Tianyi Liu +2Multimodal Reasoning VLM Reasoning
Southeast University · AIIA, Ministry of Education, China · University of Oxford
EviAnchor: Mitigating Hallucinations in Large Vision-Language Models via Regional Visual Evidence Compensation
Aug 29, 2026Sihang Jia, Shuliang Liu, Songbo Yang +1VLM Hallucination Mitigation Large Vision-Language Models
The Hong Kong University of Science and Technology (Guangzhou)

Jun 29, 2026J/K move · Enter open · S save

Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigation

Xin Zou, Haolin Deng, Yibo Yan, Shuliang Liu+4

HKUST (GZ) · HKUST · OPPO AI Center

Multimodal Large Language Models Multimodal Hallucination Visually Grounded Reasoning Visual Grounding Multimodal Foundation Models VLM Hallucination Mitigation+1

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) are prone to hallucination as their generation preferences are insufficiently calibrated to visual evidence, causing them to fall back on linguistic priors, rather than faithful grounding. In this work, we start from an empirical observation: when query-relevant visual evidence is explicitly strengthened using the model's own attention, generation becomes more accurate, suggesting that many failures do not arise solely from missing perception, but from an insufficient tendency to trust the evidence the model has already attended to. Motivated by this finding, we propose Oriented Pickup Preference Optimization (OPPO), an evidence-aware alignment objective that learns preferences over the strength of visual evidence, rather than only response quality. Concretely, OPPO contrasts the same faithful response under stronger, anchored, weaker-evidence views, turning naive visual preference into ordered visual-evidence alignment. We further combine this objective with fine-grained span-level and token-level regularization to stabilize the training. Besides, we provide a theoretical analysis showing that ordered evidence margins induce a positive lower bound on local visual sensitivity. Extensive evaluations across hallucination and general-purpose benchmarks demonstrate that OPPO consistently outperforms baseline methods.

Abstract

VLM Hallucination Mitigation

Explore similar work

Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigation

See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs

EviAnchor: Mitigating Hallucinations in Large Vision-Language Models via Regional Visual Evidence Compensation