Mind What Matters for Reasoning: Aligning Cross-Modal Attention via Selective Probability Mass Concentration

Jiaqi Deng; Zonghan Wu; Zhan Heng; Xiaoshui Huang; Huan Huo; Guandong Xu

cs.CVSep 24, 2026

Mind What Matters for Reasoning: Aligning Cross-Modal Attention via Selective Probability Mass Concentration

Authors: Jiaqi Deng, Zonghan Wu, Zhan Heng, Xiaoshui Huang, Huan Huo, Guandong Xu

Organizations: University of Technology Sydney · East China Normal University · The University of New South Wales · Shanghai Jiaotong University · The Education University of Hong Kong

Abstract

Multimodal large language models (MLLMs) achieve strong performance on visual reasoning tasks, yet remain prone to hallucinations and over-reliance on language priors, often generating answers without adequately using task-relevant visual evidence. Existing approaches primarily improve reasoning through reasoning-oriented supervision or inference-time strategies. In this work, we study a complementary question: can multimodal reasoning be improved by strengthening implicit visual grounding without directly supervising the reasoning process? Motivated by the functional specialization of attention heads, we investigate whether reasoning can be improved by guiding only the heads most responsive to visual evidence grounding. We propose Selective Probability Mass Concentration (sPMC), a training framework that identifies grounding-responsive heads and selectively regularizes their text-to-image attention. sPMC treats normalized attention over visual tokens as a spatial probability distribution and encourages the probability mass to be assigned to semantically relevant regions using segmentation-derived spatial priors. Adaptive Head Selection restricts this guidance to visually responsive heads while leaving the remaining heads unconstrained to preserve their complementary functions. Across 6 multimodal benchmark suites, sPMC achieves an average zero-shot improvement of 3% and gains of up to 11.3% across multiple MLLMs while regularizing only 3%-15% of their attention heads. These results demonstrate that targeted guidance of sparse and implicit visual evidence pathways can directly improve multimodal reasoning.

Figures & tables

Figure 1: Overview of the proposed sPMC framework. A text-conditioned segmentation model generates a binary guidance mask; Adaptive Attention Aggregation selects grounding-responsive heads, and probability-mass concentration directs their attention toward the masked regions.

Figure 2: Exemplar visualization of attention maps from layer 6 of Qwen3-VL-2B. While several heads exhibit irrelevant attention patterns, certain heads (Head 4) demonstrate strong alignment with the task-relevant region.

Model	Base	Data	POPE	HalluBench	MMVP	VisOnly S	VisOnly R	$V^{*}$	MME
Qwen3-VL-2B	Qwen3-2B	N.A.	89.0	44.1	51.3	38.0	44.0	67.5	2025.8
+ AutoR. LoRA-FT	Qwen3-2B	0.3M	84.5	35.3	51.3	31.7	46.2	63.9	1525.0
+ sPMC LoRA-FT	Qwen3-2B	0.3M	89.9	49.1	52.7	38.3	45.8	68.1	2118.8
% improvement w.r.t. base model			1.0 $\uparrow$	11.3 $\uparrow$	2.7 $\uparrow$	0.8 $\uparrow$	0.9 $\uparrow$	1.8 $\uparrow$	4.6 $\uparrow$
Qwen3-VL-8B	Qwen3-8B	N.A.	88.0	58.5	65.3	42.2	54.6	73.8	2416.2
+ AutoR. LoRA-FT	Qwen3-8B	0.3M	86.5	56.6	64.0	40.3	50.4	71.2	2317.5

Table 1: Zero-shot performance across multimodal benchmarks. The best within each group is shown in bold, and the overall best is underlined . Fair evaluations are performed and supported by VLMEvalKit ( Duan et al. 2024 ) .

Figure 3: Head-level diagnostics for Qwen3-VL-2B-Instruct, averaged over 200 randomly sampled images. The left panels show the calibration statistics and thresholds; the right panels compare selected heads before and after sPMC fine-tuning. Fine-tuning slightly reduces the mean visual-attention mass and shifts the alignment loss downward (lower is better).

	Qwen3- VL-2B	Qwen3- VL-8B	Llama3.2- 11B-V
# Layers	28	36	8
Total Heads	448	1152	256
Selected Heads	69	35	37
Selection Rate (%)	15.40%	3.03%	14.45%

Table 2: Summary of adaptive head selection, which demonstrates the sparsity of visual grounding heads. Detailed layer-wise head selection is provided in Appendix.

Figure 4: Ablation study: zero-shot performance of head-selection variants for Qwen3-VL-2B-Instruct (28 layers and 448 heads). We partition layers into early (1–8), middle (9–16), and late (17–28) stages. Adaptive selection in late layers performs best on all five metrics.

Method	Align	POPE	Hall.	MMVP	Vstar	Avg.
Qwen3-VL-2B
No Guidance	$\times$	89.0	44.1	51.3	67.5	62.98
AutoR.	$\times$	84.5	35.3	51.3	63.9	58.75
CrossEntropy	$\checkmark$	81.5	32.2	42.0	58.9	53.65
sPMC	$\checkmark$	89.9	49.1	52.7	68.1	64.95
Qwen3-VL-8B

Table 3: Ablation of the training objective. AutoR. denotes LoRA fine-tuning with only the next-token objective. Cross-Entropy directly matches attention to the normalized mask. The average is the unweighted mean of the four benchmark scores;

\times

denotes no attention-alignment loss.

Figure 5: Qualitative comparison between Qwen3-VL-2B and its sPMC-tuned counterpart. The examples cover diagram interpretation, OCR, commonsense reasoning, and fine-grained visual discrimination. Yellow marks incorrect answers and green marks correct answers. The bottom-left example is a failure case introduced by sPMC.

Figure 6: Qualitative attention maps highlighting representative heads (e.g., Layer 27, head 6 and 9) that consistently capture relevant visual evidence across samples. Other head’s attention (e.g., head 6, and 14) are scattered in the image or not focusing on the relevant region.

Figure 7: Qualitative attention maps demonstrating visual evidence aggregation behavior exists across different model backbones: Qwen3-VL-2B, Qwen3-VL-8B, Llama-3.2-11B-V. For various backbones, the selected heads appeared to be more concentrated on relevant regions compared to other heads.

Label	Human %	Human #	AI %	AI #
Accurate	77.0%	154	61.5%	123
Usable	22.0%	44	36.0%	72
Unusable	1.0%	2	1.5%	3
Unjudgeable	0%	0	1.0%	2

Table 4: Segmentation Mask Quality Audit Summary Statistics

Figure 8: Examples from our curated mask dataset and quality audit result. Red masks are the generated segmentation masks.

Figure 9: Examples from our curated mask dataset and quality audit result (Continued).

Stage / Base	Computation Time
Mask Generation
Noun Phrase Extraction	0.04 s/iteration
Grounding Mask Generation	0.18 s/iteration
Training
Qwen3-VL-2B	2 h 25 min
Qwen3-VL-8B	4 h 43 min

Table 5: Computation time for semantically enriched grounding masks generation and model’s instruction tuning.

Model	#L	Sel. Rate (%)	Selected Heads (by layer)
Qwen3-VL-2B ∗	28	15.40% (69/448)	L17: [6, 7, 8, 9, 10, 14, 15, 16]; L18: [2, 3, 9, 11, 12, 13]; L19: [3, 4, 6, 12]; L20: [5, 6, 10, 14]; L21: [4, 5, 7, 9, 12, 13, 14, 15, 16]; L22: [2, 9, 10, 13, 14, 16]; L23: [8, 10, 13]; L24: [1, 2, 5, 7, 15, 16]; L27: [3, 5, 6, 9, 15]
Qwen3-VL-8B ∗	36	3.03% (35/1152)	L22: [11, 19, 23, 24, 27]; L23: [23, 24]; L25: [9, 30, 31]; L26: [31]; L29: [1, 24]; L30: [1, 4, 8, 13, 15]; L31: [9, 10, 12, 25, 26, 28]; L32: [5, 6, 7, 8, 14, 15, 16]; L33: [1, 11, 12]; L34: [24]
Llama3.2-11B-V	8	14.45% (37/256)	L3: [9, 11, 25, 26, 27]; L4: [6, 8, 9, 10, 12, 21, 24, 25]; L5: [1, 2, 3, 4, 5, 6, 7, 8, 13, 16, 17, 18, 19, 20, 25, 27, 32]; L6: [6, 9, 10, 12, 21, 23]; L7: [24]

Table 6: Full head selection across base models. Selection rate (Sel. Rate) is computed as the ratio of selected heads to total heads of the model. ∗ indicates that cross-modal attention is achieved via self attention only. #L means the total number of attention layers for each model. Selected head indexes are summarized in lists.

Explore similar work

May 21, 2026cs.CL

Faithful-MR1: Faithful Multimodal Reasoning via Anchoring and Reinforcing Visual Attention

Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising paradigm for advancing complex reasoning in large language models, and recent work extends RLVR to multimodal large language models (MLLMs). This transfer, however, surfaces a faithfulness challenge: faithful perception of task-relevant visual evidence and faithful use of that evidence during reasoning, leading to unsatisfactory gains on multimodal benchmarks. Specifically, existing perception supervision often operates on textual descriptions rather than natively on image regions, and faithful use is largely overlooked, exposing the perception-reasoning disconnect where correctly perceived evidence is dropped or contradicted during reasoning. To close these gaps, we propose Faithful-MR1, a training framework that anchors and reinforces visual attention to address both halves of faithful multimodal reasoning. The Anchoring stage turns perception into an explicit pre-reasoning subtask, supervising a dedicated <Focus> token's attention directly against image regions rather than through textual descriptions. The Reinforcing stage exposes faithful use through counterfactual image intervention, rewarding answer-correct trajectories that concentrate visual attention where vision causally matters. Extensive experiments demonstrate that Faithful-MR1 outperforms recent multimodal reasoning baselines on both Qwen2.5-VL-Instruct 3B and 7B backbones while using substantially less training data.

Changyuan Tian, Zhicong Lu, Huaxing Liu +7

AMAP, Alibaba Group · University of Chinese Academy of Sciences · Tsinghua University +1

Multimodal Reasoning Recent Vision-Language Models Visual Evidence Reinforcement Learning With Verifiable Reward

Ask PDF

Aug 4, 2026cs.MM

Balancing Efficiency and Efficacy: Training-Free Attention-Guided Switching Between Explicit and Latent Thoughts for MLLMs

Reasoning in Multimodal Large Language Models (MLLMs) requires both fine-grained visual perception and rigorous logical deduction. Explicit text-based Chain-of-Thought (CoT) is computationally expensive and prone to visual hallucinations, while existing latent reasoning methods typically require costly training. Furthermore, directly adapting training-free LLM reasoning mechanisms to the multimodal setting yields unstable performance. We identify that this failure stems from their reliance on token-level entropy, which fundamentally conflates perceptual ambiguity (e.g., unclear visual details) with logical uncertainty (e.g., complex reasoning steps). To overcome this bottleneck, we present a novel training-free inference strategy for MLLMs that explicitly decouples perception and reasoning. We propose a novel metric, the vision-to-text attention ratio, to dynamically gauge the model's cognitive focus. Guided by this metric, our proposed framework, Attention-Guided Switching (AGS), adaptively triggers latent reasoning for perceptual tokens to preserve high-fidelity visual information in the continuous space, while enforcing explicit text generation for logical tokens to maintain structural anchoring. Extensive experiments demonstrate that our method achieves state-of-the-art performance, significantly improving both accuracy and inference efficiency by reducing autoregressive steps and latency. Code is released at https://github.com/swordAndSnow/MM26-AGS.

Haoqian Kang, Liupeng Li, Kuofeng Gao +5

Harbin Institute of Technology, Shenzhen Shenzhen, China · Tsinghua Shenzhen International Graduate School, Tsinghua University Shenzhen, China · Peng Cheng Laboratory Shenzhen, China +1

Multimodal Large Language Models LLM Reasoning Strategies Recent Vision-Language Models Latent Thoughts Chain-of-Thought Reasoning Training-Free+2

Ask PDF

May 27, 2026cs.AI

Look on Demand: A Cognitive Scheduling Framework for Visual Evidence Acquisition in Multimodal Reasoning

Existing multimodal reasoning approaches predominantly follow two paradigms: converting visual inputs into text prior to reasoning, or performing end-to-end reasoning within a unified vision-language representation space. Despite their empirical progress, both paradigms suffer from fundamental structural limitations. The former relies on static visual-to-text conversion, which tends to compress and lose fine-grained visual details. The latter is prone to linguistic dominance induced by joint optimization and attention mechanisms, leading to systematically weakened faithfulness to visual evidence during reasoning. In this work, we argue that a central challenge is how and when visual evidence is introduced into the reasoning process. Motivated by this insight, we propose CSMR, a multimodal reasoning framework in which a language model controls the reasoning process by deciding when to invoke an independent visual perception module to acquire task-relevant visual evidence. Experiments across multiple multimodal reasoning benchmarks show that CSMR consistently outperforms representative baseline methods in accuracy under a zero-shot setting. Further experimental analysis confirms that these advantages primarily arise from the proposed cognitive scheduling mechanism.

Yang Zhang, Xiaoshuai Sun, Rui Zhao +5

Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. · Sino-Russian ResearchCenter for Digital Economy. · School of Informatics, Xiamen University, China. +2

Multimodal Reasoning Multimodal Reasoning Benchmarks Visual Evidence Schedulers

Ask PDF