Deeper Thought, Weaker Aim: Understanding and Mitigating Perceptual Impairment during Reasoning in Multimodal Large Language Models

Ruiying Peng; Xueyu Wu; Jing Lei; Lu Hou; Yuanzheng Ma; Xiaohui Li

cs.CVMar 15, 2026

Deeper Thought, Weaker Aim: Understanding and Mitigating Perceptual Impairment during Reasoning in Multimodal Large Language Models

Authors: Ruiying Peng, Xueyu Wu, Jing Lei, Lu Hou, Yuanzheng Ma, Xiaohui Li

Organizations: Tsinghua Shenzhen International Graduate School · Huawei Technologies

Abstract

Multimodal large language models (MLLMs) often suffer from perceptual impairments under extended reasoning modes, particularly in visual question answering (VQA) tasks. We identify attention dispersion as the underlying cause: during multi-step reasoning, the model's visual attention becomes scattered and drifts away from question-relevant regions, effectively "losing focus" on the visual input. To better understand this phenomenon, we analyze the attention maps of MLLMs and observe that reasoning prompts significantly reduce attention to regions critical for answering the question. We further find a strong correlation between the model's overall attention on image tokens and the spatial dispersiveness of its attention within the image. Leveraging this insight, we propose a training-free Visual Region-Guided Attention (VRGA) framework that selects visual heads based on an entropy-focus criterion and reweights their attention, effectively guiding the model to focus on question-relevant regions during reasoning. Extensive experiments on vision-language benchmarks demonstrate that our method effectively alleviates perceptual degradation, leading to improvements in visual grounding and reasoning accuracy while providing interpretable insights into how MLLMs process visual information.

Figures & tables

Figure 1: Comparison of Prompting Strategies in VQA Tasks. This figure compares three prompting strategies— Direct , Region-Guided , and Reason —in visual question answering (VQA) tasks. The attention maps show that the Direct mode focuses correctly on question-relevant regions, while the Region-Guided approach further reduces attention to irrelevant areas, enhancing visual grounding. In contrast, the Reason mode disperses attention across the scene and often guides the model to describe question-irrelevant regions, leading to incorrect or misleading summaries. In the QA results, green boxes highlight correct answers, whereas red boxes indicate errors and irrelevant reasoning. Overall, these visualizations illustrate how different prompting strategies influence attention allocation and task performance.

Figure 2: Layer-wise relevant region attention ratio across models. The vertical axis denotes the Relevant Region Attention Ratio , which measures the degree of attention allocated to question-relevant regions during VQA (as defined in Sec. 3.1 ). The horizontal axis represents the Transformer layer index, where the attention maps are averaged across heads. Green and red curves correspond to correct and wrong predictions, respectively. The solid line shows the mean value, and shaded areas denote ranges. Across all models, correct answers consistently exhibit higher attention to relevant regions, suggesting that focusing on the correct visual evidence is crucial for accurate reasoning.

Figure 3: Impact of prompting strategies on visual grounding and performance. The bar chart (left) compares TextVQA accuracy under three prompting strategies: Reason , Direct , and Region-Guided . The line plots show the layer-wise RRAR (as defined in Sec. 3.1 ) from the question-end token to relevant visual regions, which is used to measure the focus on question-related areas. Shaded regions indicate the interquartile range (Q1–Q3) of RRAR across samples. The Ocean-R1-3B [ 19 ] model is built on the Qwen2.5-VL-3B backbone, while MM-Eureka-7B [ 18 ] and ThinkLite-VL-7B [ 26 ] are based on the Qwen2.5-VL-7B backbone. For detailed explanation of the relevant region attention ratio (RRAR), please refer to Section 3. We observe that the Reason mode, which emphasizes sequential reasoning, tends to disperse visual attention and reduce focus on question-relevant regions, leading to lower accuracy. By contrast, the proposed Region-Guided prompting re-concentrates attention toward relevant regions, achieving both improved grounding and higher task accuracy.

Figure 4: Attention mechanism analysis in vision models. Left : Scatter plot showing the relationship between $R_{img}$ and $H_{img}$ . Points within the red box represent heads indicating effective processing of visual information.A linear fit is performed on the points, with the correlation coefficient $r$ representing the Pearson correlation. Right : Line graph comparing the Relevant Region Attention Ratio (RRAR) for different selection methods. The red line represents the RRAR for heads selected by our method, which considers both high $R_{img}$ and low EFR( $\frac{H_{img}}{R_{img}}$ ). The green line shows the RRAR for heads based solely on $R_{img}$ . The blue line indicates the RRAR across layers. Our method consistently identifies heads that precisely focus on relevant visual regions.

Model	Slope $k$ ( $\mu\pm\sigma$ )	Intercept $b$ ( $\mu\pm\sigma$ )	Pearson $r$ ( $\mu\pm\sigma$ )
LLaVA-1.5-7B	$0.167\pm 0.002$	$-0.002\pm 0.003$	$\textbf{0.991}\pm 0.005$
Ovis2.5-2B	$0.220\pm 0.015$	$-0.021\pm 0.007$	$\textbf{0.923}\pm 0.036$
Gemma-3-4B	$0.828\pm 0.042$	$-0.027\pm 0.015$	$\textbf{0.929}\pm 0.020$
Qwen2.5-VL-7B	$0.215\pm 0.015$	$-0.028\pm 0.008$	$\textbf{0.950}\pm 0.038$
Qwen2.5-VL-3B	$0.213\pm 0.011$	$-0.024\pm 0.008$	$\textbf{0.951}\pm 0.033$

Table 1: Linear regression statistics between the image attention ratio (

R_{\text{img}}

) and attention entropy (

H_{\text{img}}

) across multiple MLLMs on the TextVQA dataset. For each model, we fit the linear relation

R_{\text{img}}=k\cdot H_{\text{img}}+b

over all visual-question pairs, and report the mean (

\mu

) and standard deviation (

\sigma

) of the slope (

k

), intercept (

b

), and Pearson correlation coefficient (

r

). A strong positive correlation across models indicates a consistent linear dependency between visual focus and attention dispersion.

Figure 5: Overview of the Visual Region-guided Attention (VRGA) framework. Our method enhances the visual grounding capability of Multimodal Large Language Models (MLLMs) without additional training. (a) Localization of Question-relevant Regions: By integrating the attention maps from vision-focused heads $\mathcal{H}_{v}$ and background-biased heads $\mathcal{H}_{b}$ , and conducting attention sink analysis, we obtain a clean, refined attention map that facilitates the localization of tokens related to the question. (b) Attention Reweighting: During the generation phase, attention to tokens within the localized regions is amplified, steering the model to focus on relevant visual content.

Model	Baseline	Random	Low-Visual	EFR-Guided (Ours)
Qwen2.5-VL-3B	87.64	83.38	87.02	24.31
Qwen2.5-VL-7B	86.88	86.95	87.50	40.52
Qwen3-VL-30B	90.44	88.80	90.31	42.27

Table 2: Accuracy (%) comparison under different head-masking strategies on the TextVQA benchmark. EFR-Guided Masking (ours) leads to the largest accuracy drop, indicating that the selected heads are most critical for visual reasoning.

Model	HaloQuest			HallusionBench			MMStar
Model	$ACC\%$ (↑)	$S$ (↑)	$I$ (↓)	$ACC\%$ (↑)	$S$ (↑)	$I$ (↓)	$ACC\%$ (↑)	$S$ (↑)	$I$ (↓)
Qwen3-VL-30B	69.20	0.454	0.762	68.55	0.507	0.611	66.1	0.521	0.542
Qwen3-VL-30B+VRGA(ours)	68.05	0.465	0.744	69.40	0.510	0.614	67.1	0.522	0.543
Qwen2.5-VL-3B	58.87	0.445	0.601	53.40	0.442	0.445	49.80	0.436	0.382
Qwen2.5-VL-3B+CCOT	41.06	/	/	53.21	/	/	46.27	/	/
Qwen2.5-VL-3B+VRGA(ours)	59.03	0.488	0.405	55.40	0.459	0.433	50.93	0.441	0.383

Table 3: Performance comparison on HALO , HallusionBench and MMStar . Each column reports the comprehensive score (

S

), accuracy (

ACC\%

), and irrelevance degree (

I

). Large Language ModelGA consistently improves grounding quality and reduces irrelevant reasoning.

Explore similar work

Balancing Efficiency and Efficacy: Training-Free Attention-Guided Switching Between Explicit and Latent Thoughts for MLLMs
Aug 4, 2026Haoqian Kang, Liupeng Li, Kuofeng Gao +5Efficient Multimodal Inference Multimodal Large Language Models
Harbin Institute of Technology, Shenzhen Shenzhen, China · Tsinghua Shenzhen International Graduate School, Tsinghua University Shenzhen, China · Peng Cheng Laboratory Shenzhen, China +1
Mind What Matters for Reasoning: Aligning Cross-Modal Attention via Selective Probability Mass Concentration
Sep 24, 2026Jiaqi Deng, Zonghan Wu, Zhan Heng +3Multimodal Grounding Multimodal Large Language Models
University of Technology Sydney · East China Normal University · The University of New South Wales +2
Faithful-MR1: Faithful Multimodal Reasoning via Anchoring and Reinforcing Visual Attention
May 21, 2026Changyuan Tian, Zhicong Lu, Huaxing Liu +7Multimodal Large Language Models Multimodal Reasoning
AMAP, Alibaba Group · University of Chinese Academy of Sciences · Tsinghua University +1

Aug 4, 2026J/K move · Enter open · S save

Balancing Efficiency and Efficacy: Training-Free Attention-Guided Switching Between Explicit and Latent Thoughts for MLLMs

Haoqian Kang, Liupeng Li, Kuofeng Gao, Jinpeng Wang+4

Harbin Institute of Technology, Shenzhen Shenzhen, China · Tsinghua Shenzhen International Graduate School, Tsinghua University Shenzhen, China · Peng Cheng Laboratory Shenzhen, China+1

Efficient Multimodal Inference Multimodal Large Language Models Latent Visual Reasoning VLM Reasoning

Open paper AskPDF arXiv

Reasoning in Multimodal Large Language Models (MLLMs) requires both fine-grained visual perception and rigorous logical deduction. Explicit text-based Chain-of-Thought (CoT) is computationally expensive and prone to visual hallucinations, while existing latent reasoning methods typically require costly training. Furthermore, directly adapting training-free LLM reasoning mechanisms to the multimodal setting yields unstable performance. We identify that this failure stems from their reliance on token-level entropy, which fundamentally conflates perceptual ambiguity (e.g., unclear visual details) with logical uncertainty (e.g., complex reasoning steps). To overcome this bottleneck, we present a novel training-free inference strategy for MLLMs that explicitly decouples perception and reasoning. We propose a novel metric, the vision-to-text attention ratio, to dynamically gauge the model's cognitive focus. Guided by this metric, our proposed framework, Attention-Guided Switching (AGS), adaptively triggers latent reasoning for perceptual tokens to preserve high-fidelity visual information in the continuous space, while enforcing explicit text generation for logical tokens to maintain structural anchoring. Extensive experiments demonstrate that our method achieves state-of-the-art performance, significantly improving both accuracy and inference efficiency by reducing autoregressive steps and latency. Code is released at https://github.com/swordAndSnow/MM26-AGS.

Abstract

Explore similar work

Balancing Efficiency and Efficacy: Training-Free Attention-Guided Switching Between Explicit and Latent Thoughts for MLLMs

Mind What Matters for Reasoning: Aligning Cross-Modal Attention via Selective Probability Mass Concentration

Faithful-MR1: Faithful Multimodal Reasoning via Anchoring and Reinforcing Visual Attention