Multimodal large language models (MLLMs) often suffer from perceptual impairments under extended reasoning modes, particularly in visual question answering (VQA) tasks. We identify attention dispersion as the underlying cause: during multi-step reasoning, the model's visual attention becomes scattered and drifts away from question-relevant regions, effectively "losing focus" on the visual input. To better understand this phenomenon, we analyze the attention maps of MLLMs and observe that reasoning prompts significantly reduce attention to regions critical for answering the question. We further find a strong correlation between the model's overall attention on image tokens and the spatial dispersiveness of its attention within the image. Leveraging this insight, we propose a training-free Visual Region-Guided Attention (VRGA) framework that selects visual heads based on an entropy-focus criterion and reweights their attention, effectively guiding the model to focus on question-relevant regions during reasoning. Extensive experiments on vision-language benchmarks demonstrate that our method effectively alleviates perceptual degradation, leading to improvements in visual grounding and reasoning accuracy while providing interpretable insights into how MLLMs process visual information.
Figures & tables
Figure 1: Comparison of Prompting Strategies in VQA Tasks. This figure compares three prompting strategies— Direct , Region-Guided , and Reason —in visual question answering (VQA) tasks. The attention maps show that the Direct mode focuses correctly on question-relevant regions, while the Region-Guided approach further reduces attention to irrelevant areas, enhancing visual grounding. In contrast, the Reason mode disperses attention across the scene and often guides the model to describe question-irrelevant regions, leading to incorrect or misleading summaries. In the QA results, green boxes highlight correct answers, whereas red boxes indicate errors and irrelevant reasoning. Overall, these visualizations illustrate how different prompting strategies influence attention allocation and task performance.
Figure 2: Layer-wise relevant region attention ratio across models. The vertical axis denotes the Relevant Region Attention Ratio , which measures the degree of attention allocated to question-relevant regions during VQA (as defined in Sec. 3.1 ). The horizontal axis represents the Transformer layer index, where the attention maps are averaged across heads. Green and red curves correspond to correct and wrong predictions, respectively. The solid line shows the mean value, and shaded areas denote ranges. Across all models, correct answers consistently exhibit higher attention to relevant regions, suggesting that focusing on the correct visual evidence is crucial for accurate reasoning.
Figure 3: Impact of prompting strategies on visual grounding and performance. The bar chart (left) compares TextVQA accuracy under three prompting strategies: Reason , Direct , and Region-Guided . The line plots show the layer-wise RRAR (as defined in Sec. 3.1 ) from the question-end token to relevant visual regions, which is used to measure the focus on question-related areas. Shaded regions indicate the interquartile range (Q1–Q3) of RRAR across samples. The Ocean-R1-3B [ 19 ] model is built on the Qwen2.5-VL-3B backbone, while MM-Eureka-7B [ 18 ] and ThinkLite-VL-7B [ 26 ] are based on the Qwen2.5-VL-7B backbone. For detailed explanation of the relevant region attention ratio (RRAR), please refer to Section 3. We observe that the Reason mode, which emphasizes sequential reasoning, tends to disperse visual attention and reduce focus on question-relevant regions, leading to lower accuracy. By contrast, the proposed Region-Guided prompting re-concentrates attention toward relevant regions, achieving both improved grounding and higher task accuracy.
Figure 4: Attention mechanism analysis in vision models. Left : Scatter plot showing the relationship between Rimg and Himg . Points within the red box represent heads indicating effective processing of visual information.A linear fit is performed on the points, with the correlation coefficient r representing the Pearson correlation. Right : Line graph comparing the Relevant Region Attention Ratio (RRAR) for different selection methods. The red line represents the RRAR for heads selected by our method, which considers both high Rimg and low EFR( RimgHimg ). The green line shows the RRAR for heads based solely on Rimg . The blue line indicates the RRAR across layers. Our method consistently identifies heads that precisely focus on relevant visual regions.
Model
Slope k ( μ±σ )
Intercept b ( μ±σ )
Pearson r ( μ±σ )
LLaVA-1.5-7B
0.167±0.002
−0.002±0.003
0.991±0.005
Ovis2.5-2B
0.220±0.015
−0.021±0.007
0.923±0.036
Gemma-3-4B
0.828±0.042
−0.027±0.015
0.929±0.020
Qwen2.5-VL-7B
0.215±0.015
−0.028±0.008
0.950±0.038
Qwen2.5-VL-3B
0.213±0.011
−0.024±0.008
0.951±0.033
Table 1: Linear regression statistics between the image attention ratio ( Rimg ) and attention entropy ( Himg ) across multiple MLLMs on the TextVQA dataset. For each model, we fit the linear relation Rimg=k⋅Himg+b over all visual-question pairs, and report the mean ( μ ) and standard deviation ( σ ) of the slope ( k ), intercept ( b ), and Pearson correlation coefficient ( r ). A strong positive correlation across models indicates a consistent linear dependency between visual focus and attention dispersion.
Figure 5: Overview of the Visual Region-guided Attention (VRGA) framework. Our method enhances the visual grounding capability of Multimodal Large Language Models (MLLMs) without additional training. (a) Localization of Question-relevant Regions: By integrating the attention maps from vision-focused heads Hv and background-biased heads Hb , and conducting attention sink analysis, we obtain a clean, refined attention map that facilitates the localization of tokens related to the question. (b) Attention Reweighting: During the generation phase, attention to tokens within the localized regions is amplified, steering the model to focus on relevant visual content.
Model
Baseline
Random
Low-Visual
EFR-Guided (Ours)
Qwen2.5-VL-3B
87.64
83.38
87.02
24.31
Qwen2.5-VL-7B
86.88
86.95
87.50
40.52
Qwen3-VL-30B
90.44
88.80
90.31
42.27
Table 2: Accuracy (%) comparison under different head-masking strategies on the TextVQA benchmark. EFR-Guided Masking (ours) leads to the largest accuracy drop, indicating that the selected heads are most critical for visual reasoning.
Model
HaloQuest
HallusionBench
MMStar
ACC% (↑)
S (↑)
I (↓)
ACC% (↑)
S (↑)
I (↓)
ACC% (↑)
S (↑)
I (↓)
Qwen3-VL-30B
69.20
0.454
0.762
68.55
0.507
0.611
66.1
0.521
0.542
Qwen3-VL-30B+VRGA(ours)
68.05
0.465
0.744
69.40
0.510
0.614
67.1
0.522
0.543
Qwen2.5-VL-3B
58.87
0.445
0.601
53.40
0.442
0.445
49.80
0.436
0.382
Qwen2.5-VL-3B+CCOT
41.06
/
/
53.21
/
/
46.27
/
/
Qwen2.5-VL-3B+VRGA(ours)
59.03
0.488
0.405
55.40
0.459
0.433
50.93
0.441
0.383
Table 3: Performance comparison on HALO , HallusionBench and MMStar . Each column reports the comprehensive score ( S ), accuracy ( ACC% ), and irrelevance degree ( I ). Large Language ModelGA consistently improves grounding quality and reduces irrelevant reasoning.
Reasoning in Multimodal Large Language Models (MLLMs) requires both fine-grained visual perception and rigorous logical deduction. Explicit text-based Chain-of-Thought (CoT) is computationally expensive and prone to visual hallucinations, while existing latent reasoning methods typically require costly training. Furthermore, directly adapting training-free LLM reasoning mechanisms to the multimodal setting yields unstable performance. We identify that this failure stems from their reliance on token-level entropy, which fundamentally conflates perceptual ambiguity (e.g., unclear visual details) with logical uncertainty (e.g., complex reasoning steps). To overcome this bottleneck, we present a novel training-free inference strategy for MLLMs that explicitly decouples perception and reasoning. We propose a novel metric, the vision-to-text attention ratio, to dynamically gauge the model's cognitive focus. Guided by this metric, our proposed framework, Attention-Guided Switching (AGS), adaptively triggers latent reasoning for perceptual tokens to preserve high-fidelity visual information in the continuous space, while enforcing explicit text generation for logical tokens to maintain structural anchoring. Extensive experiments demonstrate that our method achieves state-of-the-art performance, significantly improving both accuracy and inference efficiency by reducing autoregressive steps and latency. Code is released at https://github.com/swordAndSnow/MM26-AGS.
Haoqian Kang, Liupeng Li, Kuofeng Gao +5
Harbin Institute of Technology, Shenzhen Shenzhen, China · Tsinghua Shenzhen International Graduate School, Tsinghua University Shenzhen, China · Peng Cheng Laboratory Shenzhen, China +1
Multimodal large language models (MLLMs) achieve strong performance on visual reasoning tasks, yet remain prone to hallucinations and over-reliance on language priors, often generating answers without adequately using task-relevant visual evidence. Existing approaches primarily improve reasoning through reasoning-oriented supervision or inference-time strategies. In this work, we study a complementary question: can multimodal reasoning be improved by strengthening implicit visual grounding without directly supervising the reasoning process? Motivated by the functional specialization of attention heads, we investigate whether reasoning can be improved by guiding only the heads most responsive to visual evidence grounding. We propose Selective Probability Mass Concentration (sPMC), a training framework that identifies grounding-responsive heads and selectively regularizes their text-to-image attention. sPMC treats normalized attention over visual tokens as a spatial probability distribution and encourages the probability mass to be assigned to semantically relevant regions using segmentation-derived spatial priors. Adaptive Head Selection restricts this guidance to visually responsive heads while leaving the remaining heads unconstrained to preserve their complementary functions. Across 6 multimodal benchmark suites, sPMC achieves an average zero-shot improvement of 3% and gains of up to 11.3% across multiple MLLMs while regularizing only 3%-15% of their attention heads. These results demonstrate that targeted guidance of sparse and implicit visual evidence pathways can directly improve multimodal reasoning.
Jiaqi Deng, Zonghan Wu, Zhan Heng +3
University of Technology Sydney · East China Normal University · The University of New South Wales +2
Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising paradigm for advancing complex reasoning in large language models, and recent work extends RLVR to multimodal large language models (MLLMs). This transfer, however, surfaces a faithfulness challenge: faithful perception of task-relevant visual evidence and faithful use of that evidence during reasoning, leading to unsatisfactory gains on multimodal benchmarks. Specifically, existing perception supervision often operates on textual descriptions rather than natively on image regions, and faithful use is largely overlooked, exposing the perception-reasoning disconnect where correctly perceived evidence is dropped or contradicted during reasoning. To close these gaps, we propose Faithful-MR1, a training framework that anchors and reinforces visual attention to address both halves of faithful multimodal reasoning. The Anchoring stage turns perception into an explicit pre-reasoning subtask, supervising a dedicated <Focus> token's attention directly against image regions rather than through textual descriptions. The Reinforcing stage exposes faithful use through counterfactual image intervention, rewarding answer-correct trajectories that concentrate visual attention where vision causally matters. Extensive experiments demonstrate that Faithful-MR1 outperforms recent multimodal reasoning baselines on both Qwen2.5-VL-Instruct 3B and 7B backbones while using substantially less training data.
Changyuan Tian, Zhicong Lu, Huaxing Liu +7
AMAP, Alibaba Group · University of Chinese Academy of Sciences · Tsinghua University +1