cs.AIApr 11, 2026
SaveCognitive Pivot Points and Visual Anchoring: Unveiling and Rectifying Hallucinations in Multimodal Reasoning Models
Organizations: South China Agricultural University · Gaoling School of Artificial Intelligence, Renmin University of China · South China Normal University · Monash University ·
Abstract
Multimodal Large Reasoning Models (MLRMs) have achieved remarkable strides in visual reasoning through test time compute scaling, yet long chain reasoning remains prone to hallucinations. We identify a concerning phenomenon termed the Reasoning Vision Truth Disconnect (RVTD): hallucinations are strongly correlated with cognitive bifurcation points that often exhibit high entropy states. We attribute this vulnerability to a breakdown in visual semantic anchoring, localized within the network's intermediate layers; specifically, during these high uncertainty transitions, the model fails to query visual evidence, reverting instead to language priors. Consequently, we advocate a shift from solely outcome level supervision to augmenting it with fine grained internal attention guidance. To this end, we propose V-STAR (Visual Structural Training with Attention Reinforcement), a lightweight, holistic training paradigm designed to internalize visually aware reasoning capabilities. Central to our approach is the Hierarchical Visual Attention Reward (HVAR), integrated within the GRPO framework. Upon detecting high entropy states, this mechanism dynamically incentivizes visual attention across critical intermediate layers, thereby anchoring the reasoning process back to the visual input. Furthermore, we introduce the Forced Reflection Mechanism (FRM), a trajectory editing strategy that disrupts cognitive inertia by triggering reflection around high entropy cognitive bifurcation points and encouraging verification of subsequent steps against the visual input, thereby translating external debiasing interventions into an intrinsic capability for hallucination mitigation.
Figures & tables
Fig. 1: Reasoning relies less on visual evidence. Across layers, reasoning generations show reduced visual feature signals compared to non-reasoning.
Fig. 2: A multi view analysis of hallucination triggers in multimodal reasoning. The statistics show that hallucinations tend to occur near high entropy transition words such as “However”, and the visual attention ratio is generally lower in these cases. In the example trajectory, “However” coincides with a spike in token entropy and is immediately followed by content that contradicts the image. The token level attention heatmap further indicates a strong suppression of attention to image tokens around this high entropy transition, after which the generation enters a hallucinated span highlighted by the red box, suggesting a temporary loss of visual grounding at the transition.
Fig. 3: Intermediate layer divergence between grounded and hallucinated tokens. (a) Text Image Mutual Information: The gap between grounded and hallucinated tokens peaks in the intermediate layers from 11 to 20, where grounded tokens preserve higher Text Image Mutual Information. (b) Visual attention total and concentration: In the same intermediate layer window, grounded tokens exhibit higher total visual attention and higher visual attention concentration, while hallucinated tokens show reduced together with a drop in concentration, indicating a redistribution away from visual evidence in intermediate layers.
Fig. 4: Head wise visual attention differs for grounded and hallucinated tokens. Visual attention heatmaps compare the hallucinated token “Sea” and the grounded token “Town”. In the highlighted intermediate layer region, the grounded token shows stronger and more coherent attention across heads, while the hallucinated token exhibits weaker and more sporadic activation, indicating reduced visual grounding.
Fig. 5: High Uncertainty Triggers Hallucination. (a) As the model reasons, its uncertainty (entropy) spikes significantly at logical turning points (e.g., ”However”), creating a ”spiky” pattern. (b) Our statistical analysis reveals that hallucination events (orange dots) are exclusively concentrated within these high entropy pivotal clusters, verifying a strong temporal coupling between semantic uncertainty and visual detachment.
Fig. 6: Pinpointing the breakdown of visual semantic anchoring in intermediate layers. (a) The layer wise Mahalanobis distance between the grounded answer token Town and the hallucinated token Sea shows a pronounced separation that becomes evident after Layer 11 and persists throughout the intermediate layer window from 11 to 20, consistent with reduced alignment to grounded states. (b) In an object recognition example, a Logit Lens based semantic convergence heatmap shows that image token representations undergo their strongest convergence in Layers 11 to 20. Moreover, PCA projections of representations associated with object regions, such as truck and car , indicate that intermediate layers occupy a transitional hybrid regime, making failures in this window particularly consequential.
Fig. 7: The Pseudo Reflection Paradox. Top The per token visual attention score shows that after a pivot token the model enters a visual decoupling zone and then drifts into snowballing hallucination. Even when it produces an explicit reflection cue such as “Let me check”, the visual attention does not rebound, indicating that Pseudo Reflection is not accompanied by renewed visual grounding. Bottom (a) The sample counts quantify Cognitive Inertia. In the spontaneous phase, reflection occurs in only 4 of 63 cases. (b) In the intervention phase, prompting increases reflection but successful correction occurs in only 8 of 72 cases. Together, these results support Visual Verbal Dissociation, where reflective text appears without corresponding visual evidence use.
Fig. 8: The Overall Framework of V-STAR. Our paradigm unifies microscopic attention guidance and macroscopic trajectory editing within the GRPO framework. [ 33 ] (Left) Forced Reflection Mechanism (FRM): A trajectory editing strategy that activates a reflection loop around detected high entropy cognitive bifurcation points by inserting trigger tokens. This focuses reflection on the critical transition region, encourages renewed visual grounding, disrupts cognitive inertia, and internalizes self correction behavior. (Middle) Hierarchical Visual Attention Reward (HVAR): A reward mechanism that incentivizes visual semantic anchoring in the intermediate layers (Layers 11 to 20) upon detecting high entropy cognitive bifurcation points, thereby repairing Visual Silence. (Right) GRPO based optimization: The composite reward integrates outcome correctness, visual attention reward, and reflection format reward to update the policy with GRPO.
Fig. 9: The Dynamic Dual Stream Data Synthesis Framework. Adopting a Divide and Conquer philosophy, we stratify data processing into two streams to ensure high fidelity. (Top) Logic Intensive Stream: For structured inputs, we employ a Caption then Reason cascade to ensure precise abstract deduction. (Bottom) Semantic Rich Stream: For natural scenes, we use a Generate then Refine pipeline to preserve perceptual nuance while aligning linguistic style. A final Quality Assurance module filters low quality samples.
| Task | Benchmark | Paper/Conf. | Qwen2.5VL [ 81 ] | R1-Onevision [ 82 ] | Vision-R1 [ 83 ] | VL-Rethinker [ 84 ] | VL-Cogito [ 85 ] | OpenVLThinker [ 86 ] | ThinkLite-VL [ 87 ] | V-STAR |
| Data Size | —– | ——- | —– | 155k | 210k | 39k | 80k | 59.2k | 11k | 40k |
| General Reasoning & Understanding | V-Star [ 88 ] | CVPR 2024 | 70.1 | 66.5 | 78.9 | 67.6 | 79.6 | 68.1 | 82.1 | 81.3 |
| RealWorldQA | —– | 68.8 | 60.5 | 64.3 | 69.3 | 68.1 | 62.3 | 70.1 | 72.6 | |
| MMVP [ 89 ] | CVPR 2024 | 47.3 | 43.0 | 44.0 | 42.0 | 40.0 | 46.5 | 46.7 | 47.1 | |
| MMEval-Pro [ 90 ] | NAACL 2025 | 70.6 | 69.4 | 72.2 | 73.2 | 73 | 71.5 | 72.0 | 73.6 | |
| VMCBench [ 91 ] | NeurIPS 2025 | 79.7 | 65.2 | 80.3 | 73.9 | 73.2 | 80.3 | 81.4 | 81.5 |
TABLE I: Overall Performance Comparison. We compare V-STAR with leading Multimodal Large Reasoning Models (MLRMs) across four benchmark domains: General Reasoning & Understanding, Hallucination, Mathematical Reasoning, and Scientific Reasoning. V-STAR achieves state-of-the-art performance on most datasets, outperforming both strong base models and other reasoning-specialized models.
| Benchmarks | ||||
| Methods | MMEval-Pro | MMHalu | Geo3K | RealWorld |
| Qwen2.5VL-7B | 70.6 | 3.50 | 59.2 | 68.8 |
| Reward on Answer | 71.3 | 3.81 | 62.4 | 69.7 |
| RL-HVAR-only | 72.9 | 3.92 | 67.3 | 71.7 |
| RL-FRM-only | 72.8 | 3.99 | 67.0 | 71.6 |
| V-STAR | 73.6 | 4.12 | 68.9 | 72.6 |
TABLE II: Component Ablation Study. We analyze the impact of adding the Visual Reward (HVAR) and Forced Reflection (FRM) to the base model. Results show that both components are essential, and combining them yields the highest improvement across all metrics.
| Benchmarks | ||||||
| Method | Data size | MMEval-Pro | MMHalu | Geo3K | RealWorld | |
| 0 | Base | - | 70.6 | 3.50 | 59.2 | 68.8 |
| 0.2 | Refine | 8k | 71.6 | 3.69 | 63.1 | 70.2 |
| Random | 8k | 71.0 | 3.58 | 61.4 | 69.4 | |
| 0.4 | Refine | 16k | 72.3 | 3.81 | 65.9 | 71.1 |
| Random | 16k | 71.6 | 3.74 | 63.6 | 70.3 | |
TABLE III: Ablation on Data Selection Strategy. We compare different data selection ratios ( ) and strategies. At matched data scales, the Refine strategy, which applies curated correctness-preserving selection, consistently outperforms random selection across all evaluated benchmarks.
Fig. 10: Selection ratio in data curation. Visual attention scores are reported for the anchoring layers (11–20) and the full network average, measured both in a pivot local window (10 tokens after high entropy pivots) and over the full chain. With increasing , the visual attention scores steadily rise under both measurement windows, including the pivot local window and the full reasoning chain.
Fig. 11: Linguistic quality of reflection outputs. We report automatic text quality scores (Naturalness, Fluency, Grammar; ) and perplexities (PPL1, PPL2; ) on Bingo [ 96 ] and MMHalu [ 95 ] . PPL1 and PPL2 are calculated using GPT-2, while the ratings for Grammar, Fluency, and Naturalness are provided by GPT-5. While enabling explicit reflection, V-STAR shows no degradation in language quality and achieves better results on multiple metrics.
| Random | Popular | Adversarial | |||||
| Model | Decoding | Acc | F1 | Acc | F1 | Acc | F1 |
| Qwen2.5VL-7B | Greedy | 81.4 | 80.6 | 82.5 | 81.7 | 82.3 | 81.8 |
| Sampling | 82.1 | 81.3 | 83.2 | 82.4 | 83.1 | 82.6 | |
| R1-Onevision | Greedy | 83.9 | 83.1 | 83.3 | 82.4 | 81.8 | 81.2 |
| Sampling | 84.6 | 83.8 | 84.0 | 83.1 | 82.5 | 81.9 | |
| VL-Rethinker | Greedy | 83.3 | 82.4 | 82.8 | 81.9 | 81.2 | 80.6 |
TABLE IV: Robustness on Object Hallucination (POPE). We test model performance on the POPE benchmark using different decoding strategies (Greedy vs. Sampling). V-STAR maintains high stability and accuracy across all splits (Random, Popular, Adversarial), effectively reducing object hallucinations.
Fig. 12: Attention heatmaps under identical prompting. Compared with representative baselines, V-STAR allocates more token to image attention to visual evidence during generation and concentrates this attention on semantically relevant regions, while suppressing background dominated activation. This pattern is consistent with stronger visual anchoring.
Fig. 13: Accuracy length trade-off on MathVision. We plot accuracy versus average generated token length under the same evaluation protocol. V-STAR achieves higher accuracy with shorter generations, supporting improved reasoning efficiency.
Fig. 14: Visual attention recovery during reflection. (a) Mean visual attention trajectory of V-STAR around the reflection trigger. After entering the reflection phase, the visual attention score shows a clear rebound. (b) Recovery metrics, including attention drop, recovery gain, and U score, comparing Qwen2.5-VL-7B with V-STAR. V-STAR achieves higher recovery gain and a higher U score, consistent with more grounded reflection.
More in this topic
VLM Hallucination Mitigation
Vision-language model hallucination mitigation keeps generated claims grounded in visual evidence.
See all 228 papers