Visual interventions provide a training-free way to reduce hallucinations in multimodal large language models. However, an intervention may have little leverage at some layers and may damage a correct answer even when it changes the output. We investigate where visual interventions retain leverage and when they improve predictions. First, we apply the same visual-residual edit at successive decoder layers, matching its RMS strength across depth. Across six checkpoints from three model families, we observe a persistent drop in intervention effect and define its operational onset for this edit as the Evidence Anchoring Boundary (EAB). With boundaries fixed on calibration queries, held-out effects on disjoint evaluation queries before EAB are 13.78×--26.10× larger than those after it. Comparison with intermediate readouts shows that the edit can lose leverage before answer preferences stabilize. Second, we compare counterfactual subtraction across presence, attribute, and relation questions. Subtraction can correct a misleading preference, leave a decision unchanged, or remove a preference supporting the correct answer. Based on these observations, we develop calibrated selective routing, which preserves the original prediction unless calibration supports an intervention for the question type. It improves full-set attribute accuracy by 1.56--1.81 percentage points across three checkpoints, while universal subtraction is neutral or harmful.
Figures & tables
Figure 1: From intervention leverage to selective visual intervention. Top: training-free intervention sites. Middle (Q1): the Evidence Anchoring Boundary (EAB) marks where the tested visual-residual edit becomes persistently weak under equal-RMS perturbations across depth, potentially before answer stabilization. Bottom (Q2): an edit can be corrective, redundant, or destructive depending on the preference it removes, motivating calibrated selective intervention.
Question
Dataset / task
Use in this paper
Q1: layer-wise effect
Shared sampled queries
EAB and answer-readout comparison
Q2: presence
POPE; TVQA-Existence
Confidence-conditioned diagnostics
Q2: attributes
AMBER claim pairs
Outcome analysis and routing
Q2: relations
AMBER claim pairs
Outcome analysis
Table 1: Main evaluation map. Q1 uses 120 shared queries per checkpoint (48 calibration, 72 disjoint evaluation); Q2 coverage varies by checkpoint as recorded in Appendix A.1 .
Figure 2: EAB identifies where the tested direct visual edit loses leverage. (a) Checkpoint-level boundaries frozen on calibration queries identify a persistent low-effect regime on disjoint evaluation queries under the same equal-RMS strength control. (b) Answer-readout reliability can continue to increase after this measured drop; ACP marks answer-readout stabilization. (c) Fully held-out pre-EAB effects are 13.78× – 26.10× post-EAB effects across six checkpoints.
Figure 3: The removed preference explains three intervention outcomes. (a) Presence: G adds little consistent discrimination between correct and incorrect Yes predictions after conditioning on M . (b) Schematic yellow-minus-red score differences illustrate ΔG=ΔM−ΔMcf : subtraction preserves, corrects, or damages the ranking according to the removed preference. The values are illustrative, not measured. (c) Observed pair-accuracy changes for hard attributes, inverse-spatial relations, and direct-contact relations; positive values favor G . Error bars show 95% image-cluster confidence intervals, with uncertainty estimated over image-level dependence structures.
Table 2: Subtraction has different effects across question types for each evaluated checkpoint. Presence columns report residual AUC(G∣M) ; all other columns report the accuracy change from replacing M with G , in percentage points. (Bracketed values are 95% image-cluster confidence intervals; — indicates that the checkpoint–task pair was not evaluated.)
Figure 4: Calibrated selective routing preserves the original score by default, using only calibration-supported operators for each question category. Question category and full-image/counterfactual probes restrict eligible operators. Calibration supports subtraction for suitable attribute questions or PEC in its validated presence setting; otherwise the router retains M . For presupposed questions, neutralization is followed by re-probing before a score adjustment is selected. All routing rules are fixed before evaluation, and evaluation labels are excluded from the decision.
Figure 5: Selective routing improves attributes; PEC has limited transfer. (a) Held-out AMBER attribute accuracy for the original margin M , global subtraction G , and the routed score. (b) Cached pairs illustrate rescue, break, and preservation through ΔG=ΔM−ΔMcf . (c) Targeted-generation changes on LLaVA-7B/13B, with 95% image-cluster bootstrap intervals. (d) Frozen PEC effects across POPE-adversarial, TextVQA, and MME: PEC improves its calibrated POPE setting but transfers negatively or inconsistently. Complete results and paired tests are in Appendices F.1 – F.3 .
Comparison / design choice
In-domain evidence
Failure / implication
Use G for every attribute (remove selective routing)
Routed score minus M LLaVA-1.5-7B +1.56 LLaVA-1.5-13B +1.81 InternVL3.5-8B +1.67
Global G minus M LLaVA-1.5-7B −0.42 LLaVA-1.5-13B −4.92 InternVL3.5-8B −0.33
POPE-adversarial gain LLaVA-1.5-7B +3.30 LLaVA-1.5-13B +3.90 InternVL3.5-8B +7.07
Frozen transfer change Checkpoint POPE- random TextVQA MME LLaVA-1.5-7B +0.8−4.1−9.9 LLaVA-1.5-13B +0.8−2.2+2.5 Qwen2.5-VL-7B — +0.5−0.7 InternVL3.5-8B −0.3−1.5−6.0
Retain the stability safeguard
Separate contribution not quantified.
Reject unstable corrections; no isolated accuracy gain is claimed.
Table 3: Evidence for selection and task-specific calibration across the evaluated checkpoints. Controlled comparisons and observed failure modes. Effects are percentage-point changes unless noted. (Attribute and POPE tests are in Appendix F.3 ; direct-contact intervals are in Table 2 .)
PEC transfer: TextVQA/MME; POPE PEC omitted (two baseline false-Yes cases)
InternVL3.5-8B
120
POPE: 3,000/3,000 TVQA: 2,000
600
spatial: 1,000
attribute router; PEC: 3,000; CHAIR: 500
Q1-only scale replications
Appendix
Table 4: Claim-specific model and dataset coverage. Every Q1 checkpoint uses 120 queries (48 calibration/72 evaluation); Q2 and deployment claims are restricted to the populated cells to match their evaluated scope. (POPE counts are adversarial/random; — indicates that the checkpoint–task pair was not evaluated. Statistical audits report their own sample counts in Table 8 . LLaVA-NeXT-34B and Qwen2.5-VL-32B are Q1-only scale replications.)
Item
Frozen value / policy
Reproduction record
Environment and determinism
Hardware
2× NVIDIA A100-SXM4-80GB; CUDA driver 550.54.14
one checkpoint per GPU; no model sharding for reported base runs
analysis seed 0; bootstrap seeds 0/1/2 where repeated
deterministic sample designs are cached
Experimental design and inference
EAB split
120 examples/checkpoint: 48 boundary calibration, 72 completely disjoint evaluation; one checkpoint-level EAB is frozen
cached split assignments and boundary-analysis records
Appendix
Table 5: Reproducibility checklist. Frozen policies and the records used to verify the reported experiments and artifacts, supporting reconstruction of the reported evaluation pipeline.
Checkpoint
LM depth
Frozen EAB
Relative EAB
Boundary resolution
Equal-RMS pre/post effect
Ratio
95% CI
LLaVA-1.5-7B
32
12
.375
99.2%
13.88×
[11.52,16.49]
LLaVA-1.5-13B
40
13
.325
99.2%
13.78×
[10.98,17.97]
LLaVA-NeXT-34B
60
33
.550
100%
26.10×
[22.24,30.74]
Qwen2.5-VL-7B
28
16
.571
100%
17.54×
[15.22,20.42]
Qwen2.5-VL-32B
64
40
.625
97.5%
18.24×
[14.89,22.59]
Appendix
Table 6: The operator-specific EAB pattern replicates across all six checkpoints. One checkpoint-level boundary is frozen from calibration queries before evaluation; relative EAB is that boundary divided by language-model depth. The resolution column summarizes the cached boundary audit, whereas the pre/post ratios and confidence intervals use the completely disjoint evaluation queries.
Table 7: Checkpoint-scale audit. (Shifts are larger minus smaller; bracketed values are 95% image-cluster confidence intervals.)
Figure 6: Spatial localization can persist after direct visual-residual leverage fades. Head-averaged last-query attention on one shared example; green marks the queried-object region.
Figure 7: Aggregate spatial-attention audit across four base checkpoints. Localization does not by itself identify whether a score intervention will improve the answer.
Figure 8: Prompt presupposition and neutralization are checkpoint-dependent operators.
Figure 9: Detailed attribute-routing and targeted-generation results. (a) The routed score improves full-set held-out attribute accuracy while global subtraction is neutral or harmful. (b) Point estimates show higher hallucination catch, small false-positive changes, and lower residual hallucination. Horizontal bars are 95% image-cluster intervals; paired tests are in Table 8 .
Figure 10: PEC has a calibrated presence operating range but does not transfer universally. Segments mark POPE gain regions; transfer whiskers are 95% paired intervals.
Figure 11: Frozen PEC transfer is heterogeneous across MME categories and checkpoints.
Outcome
Checkpoint
n
Effect (pp)
95% CI
p
Attribute subtraction ( G−M )
All attributes
LLaVA-1.5-7B
2,378
−0.42
[−1.64,+0.84]
.5491
Hard attributes
LLaVA-1.5-7B
521
+13.44
[+10.17,+16.89]
<10−4
All attributes
LLaVA-1.5-13B
2,378
−4.92
[−6.39,−3.36]
<10−4
Hard attributes
LLaVA-1.5-13B
508
+7.68
[+3.15,+12.01]
.0010
All attributes
InternVL3.5-8B
600
−0.33
[−2.83,+2.17]
.8946
Appendix
Table 8: Paired significance audit. Effects are A−B percentage points; inference is image-cluster aware. (A dash in the p column indicates a ratio outcome for which only a cluster-bootstrap interval is reported.)
Figure 12: Concrete attribute decisions expose the routing condition. Counterfactual subtraction rescues, damages, or preserves a ranking according to the relative preference it removes. These examples illustrate the mechanism and are not prevalence estimates.
Figure 13: Representative grounded, borrowed-evidence, and localization-miss cases.
Multimodal Large Language Models (MLLMs) are prone to hallucination as their generation preferences are insufficiently calibrated to visual evidence, causing them to fall back on linguistic priors, rather than faithful grounding. In this work, we start from an empirical observation: when query-relevant visual evidence is explicitly strengthened using the model's own attention, generation becomes more accurate, suggesting that many failures do not arise solely from missing perception, but from an insufficient tendency to trust the evidence the model has already attended to. Motivated by this finding, we propose Oriented Pickup Preference Optimization (OPPO), an evidence-aware alignment objective that learns preferences over the strength of visual evidence, rather than only response quality. Concretely, OPPO contrasts the same faithful response under stronger, anchored, weaker-evidence views, turning naive visual preference into ordered visual-evidence alignment. We further combine this objective with fine-grained span-level and token-level regularization to stabilize the training. Besides, we provide a theoretical analysis showing that ordered evidence margins induce a positive lower bound on local visual sensitivity. Extensive evaluations across hallucination and general-purpose benchmarks demonstrate that OPPO consistently outperforms baseline methods.
Recent large vision-language models (LVLMs) have demonstrated impressive reasoning ability by generating long chain-of-thought (CoT) responses. However, CoT reasoning in multimodal contexts is highly vulnerable to visual hallucination propagation: once an intermediate reasoning step becomes inconsistent with the visual evidence, subsequent steps-even if logically valid-can still lead to incorrect final answers. Existing solutions attempt to mitigate this issue by training models to "think with images" via reinforcement learning (RL). While effective, these methods are costly, model-specific, and difficult to generalize across architectures. Differently, we present a lightweight method that bypasses RL training and provides an iterative, training-free, plug-and-play framework for visually-grounded multimodal reasoning. Our key idea is to supervise each reasoning step at test time with visual evidence, ensuring that every decoded token is justified by corresponding visual cues. Concretely, we construct a textual visual-evidence pool that guides the model's reasoning generation. When existing evidence is insufficient, a visual decider module dynamically extracts additional relevant evidence from the image based on the ongoing reasoning context, expanding the pool until the model achieves sufficient visual certainty to terminate reasoning and produce the final answer. Extensive experiments on multiple LVLM backbones and benchmarks demonstrate the effectiveness of our approach. Our method achieves 16.5%-29.5% improvements on TreeBench and 13.7% RH-AUC gains on RH-Bench, substantially reducing hallucination rates while improving reasoning accuracy without additional training.
Yongchang Zhang, Oliver Ma, Tianyi Liu +2
Southeast University · AIIA, Ministry of Education, China · University of Oxford
Large vision-language models (LVLMs) frequently generate content unsupported by visual inputs. Preliminary experiments show that visual evidence is primarily incorporated into answer-side representations in early-to-middle decoder layers, while its direct influence progressively weakens in later layers. This attenuation suggests that visual evidence acquired earlier may be insufficiently utilized during subsequent generation. Based on this observation, we propose EviAnchor, a training-free and single-branch inference framework that preserves and reactivates visual evidence throughout generation. EviAnchor introduces Regional Evidence Anchor (REA) slots to progressively aggregate dense visual tokens into spatially structured representations. It then strengthens the current decision state's access to these visual anchors through decision-conditioned evidence routing, mitigating excessive dependence on textual context. Finally, the model resumes its native Transformer computation to integrate the retrieved visual evidence with question semantics and generation history. Experiments across POPE, CHAIR, and MMHal-Bench demonstrate consistent improvements in visual grounding.
Sihang Jia, Shuliang Liu, Songbo Yang +1
The Hong Kong University of Science and Technology (Guangzhou)