Mind What Matters for Reasoning: Aligning Cross-Modal Attention via Selective Probability Mass Concentration

Jiaqi Deng; Zonghan Wu; Zhan Heng; Xiaoshui Huang; Huan Huo; Guandong Xu

cs.CVSep 24, 2026

Mind What Matters for Reasoning: Aligning Cross-Modal Attention via Selective Probability Mass Concentration

Authors: Jiaqi Deng, Zonghan Wu, Zhan Heng, Xiaoshui Huang, Huan Huo, Guandong Xu

Organizations: University of Technology Sydney · East China Normal University · The University of New South Wales · Shanghai Jiaotong University · The Education University of Hong Kong

Abstract

Multimodal large language models (MLLMs) achieve strong performance on visual reasoning tasks, yet remain prone to hallucinations and over-reliance on language priors, often generating answers without adequately using task-relevant visual evidence. Existing approaches primarily improve reasoning through reasoning-oriented supervision or inference-time strategies. In this work, we study a complementary question: can multimodal reasoning be improved by strengthening implicit visual grounding without directly supervising the reasoning process? Motivated by the functional specialization of attention heads, we investigate whether reasoning can be improved by guiding only the heads most responsive to visual evidence grounding. We propose Selective Probability Mass Concentration (sPMC), a training framework that identifies grounding-responsive heads and selectively regularizes their text-to-image attention. sPMC treats normalized attention over visual tokens as a spatial probability distribution and encourages the probability mass to be assigned to semantically relevant regions using segmentation-derived spatial priors. Adaptive Head Selection restricts this guidance to visually responsive heads while leaving the remaining heads unconstrained to preserve their complementary functions. Across 6 multimodal benchmark suites, sPMC achieves an average zero-shot improvement of 3% and gains of up to 11.3% across multiple MLLMs while regularizing only 3%-15% of their attention heads. These results demonstrate that targeted guidance of sparse and implicit visual evidence pathways can directly improve multimodal reasoning.

Figures & tables

Figure 1: Overview of the proposed sPMC framework. A text-conditioned segmentation model generates a binary guidance mask; Adaptive Attention Aggregation selects grounding-responsive heads, and probability-mass concentration directs their attention toward the masked regions.

Figure 2: Exemplar visualization of attention maps from layer 6 of Qwen3-VL-2B. While several heads exhibit irrelevant attention patterns, certain heads (Head 4) demonstrate strong alignment with the task-relevant region.

Model	Base	Data	POPE	HalluBench	MMVP	VisOnly S	VisOnly R	$V^{*}$	MME
Qwen3-VL-2B	Qwen3-2B	N.A.	89.0	44.1	51.3	38.0	44.0	67.5	2025.8
+ AutoR. LoRA-FT	Qwen3-2B	0.3M	84.5	35.3	51.3	31.7	46.2	63.9	1525.0
+ sPMC LoRA-FT	Qwen3-2B	0.3M	89.9	49.1	52.7	38.3	45.8	68.1	2118.8
% improvement w.r.t. base model			1.0 $\uparrow$	11.3 $\uparrow$	2.7 $\uparrow$	0.8 $\uparrow$	0.9 $\uparrow$	1.8 $\uparrow$	4.6 $\uparrow$
Qwen3-VL-8B	Qwen3-8B	N.A.	88.0	58.5	65.3	42.2	54.6	73.8	2416.2
+ AutoR. LoRA-FT	Qwen3-8B	0.3M	86.5	56.6	64.0	40.3	50.4	71.2	2317.5

Table 1: Zero-shot performance across multimodal benchmarks. The best within each group is shown in bold, and the overall best is underlined . Fair evaluations are performed and supported by VLMEvalKit ( Duan et al. 2024 ) .

Figure 3: Head-level diagnostics for Qwen3-VL-2B-Instruct, averaged over 200 randomly sampled images. The left panels show the calibration statistics and thresholds; the right panels compare selected heads before and after sPMC fine-tuning. Fine-tuning slightly reduces the mean visual-attention mass and shifts the alignment loss downward (lower is better).

	Qwen3- VL-2B	Qwen3- VL-8B	Llama3.2- 11B-V
# Layers	28	36	8
Total Heads	448	1152	256
Selected Heads	69	35	37
Selection Rate (%)	15.40%	3.03%	14.45%

Table 2: Summary of adaptive head selection, which demonstrates the sparsity of visual grounding heads. Detailed layer-wise head selection is provided in Appendix.

Figure 4: Ablation study: zero-shot performance of head-selection variants for Qwen3-VL-2B-Instruct (28 layers and 448 heads). We partition layers into early (1–8), middle (9–16), and late (17–28) stages. Adaptive selection in late layers performs best on all five metrics.

Method	Align	POPE	Hall.	MMVP	Vstar	Avg.
Qwen3-VL-2B
No Guidance	$\times$	89.0	44.1	51.3	67.5	62.98
AutoR.	$\times$	84.5	35.3	51.3	63.9	58.75
CrossEntropy	$\checkmark$	81.5	32.2	42.0	58.9	53.65
sPMC	$\checkmark$	89.9	49.1	52.7	68.1	64.95
Qwen3-VL-8B

Table 3: Ablation of the training objective. AutoR. denotes LoRA fine-tuning with only the next-token objective. Cross-Entropy directly matches attention to the normalized mask. The average is the unweighted mean of the four benchmark scores;

\times

denotes no attention-alignment loss.

Figure 5: Qualitative comparison between Qwen3-VL-2B and its sPMC-tuned counterpart. The examples cover diagram interpretation, OCR, commonsense reasoning, and fine-grained visual discrimination. Yellow marks incorrect answers and green marks correct answers. The bottom-left example is a failure case introduced by sPMC.

Figure 6: Qualitative attention maps highlighting representative heads (e.g., Layer 27, head 6 and 9) that consistently capture relevant visual evidence across samples. Other head’s attention (e.g., head 6, and 14) are scattered in the image or not focusing on the relevant region.

Figure 7: Qualitative attention maps demonstrating visual evidence aggregation behavior exists across different model backbones: Qwen3-VL-2B, Qwen3-VL-8B, Llama-3.2-11B-V. For various backbones, the selected heads appeared to be more concentrated on relevant regions compared to other heads.

Label	Human %	Human #	AI %	AI #
Accurate	77.0%	154	61.5%	123
Usable	22.0%	44	36.0%	72
Unusable	1.0%	2	1.5%	3
Unjudgeable	0%	0	1.0%	2

Table 4: Segmentation Mask Quality Audit Summary Statistics

Figure 8: Examples from our curated mask dataset and quality audit result. Red masks are the generated segmentation masks.

Figure 9: Examples from our curated mask dataset and quality audit result (Continued).

Stage / Base	Computation Time
Mask Generation
Noun Phrase Extraction	0.04 s/iteration
Grounding Mask Generation	0.18 s/iteration
Training
Qwen3-VL-2B	2 h 25 min
Qwen3-VL-8B	4 h 43 min

Table 5: Computation time for semantically enriched grounding masks generation and model’s instruction tuning.

Model	#L	Sel. Rate (%)	Selected Heads (by layer)
Qwen3-VL-2B ∗	28	15.40% (69/448)	L17: [6, 7, 8, 9, 10, 14, 15, 16]; L18: [2, 3, 9, 11, 12, 13]; L19: [3, 4, 6, 12]; L20: [5, 6, 10, 14]; L21: [4, 5, 7, 9, 12, 13, 14, 15, 16]; L22: [2, 9, 10, 13, 14, 16]; L23: [8, 10, 13]; L24: [1, 2, 5, 7, 15, 16]; L27: [3, 5, 6, 9, 15]
Qwen3-VL-8B ∗	36	3.03% (35/1152)	L22: [11, 19, 23, 24, 27]; L23: [23, 24]; L25: [9, 30, 31]; L26: [31]; L29: [1, 24]; L30: [1, 4, 8, 13, 15]; L31: [9, 10, 12, 25, 26, 28]; L32: [5, 6, 7, 8, 14, 15, 16]; L33: [1, 11, 12]; L34: [24]
Llama3.2-11B-V	8	14.45% (37/256)	L3: [9, 11, 25, 26, 27]; L4: [6, 8, 9, 10, 12, 21, 24, 25]; L5: [1, 2, 3, 4, 5, 6, 7, 8, 13, 16, 17, 18, 19, 20, 25, 27, 32]; L6: [6, 9, 10, 12, 21, 23]; L7: [24]

Table 6: Full head selection across base models. Selection rate (Sel. Rate) is computed as the ratio of selected heads to total heads of the model. ∗ indicates that cross-modal attention is achieved via self attention only. #L means the total number of attention layers for each model. Selected head indexes are summarized in lists.

Explore similar work

Faithful-MR1: Faithful Multimodal Reasoning via Anchoring and Reinforcing Visual Attention
May 21, 2026Changyuan Tian, Zhicong Lu, Huaxing Liu +7Multimodal Reasoning Recent Vision-Language Models
AMAP, Alibaba Group · University of Chinese Academy of Sciences · Tsinghua University +1
Balancing Efficiency and Efficacy: Training-Free Attention-Guided Switching Between Explicit and Latent Thoughts for MLLMs
Aug 4, 2026Haoqian Kang, Liupeng Li, Kuofeng Gao +5Multimodal Large Language Models LLM Reasoning Strategies
Harbin Institute of Technology, Shenzhen Shenzhen, China · Tsinghua Shenzhen International Graduate School, Tsinghua University Shenzhen, China · Peng Cheng Laboratory Shenzhen, China +1
Look on Demand: A Cognitive Scheduling Framework for Visual Evidence Acquisition in Multimodal Reasoning
May 27, 2026Yang Zhang, Xiaoshuai Sun, Rui Zhao +5Multimodal Reasoning Multimodal Reasoning Benchmarks
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. · Sino-Russian ResearchCenter for Digital Economy. · School of Informatics, Xiamen University, China. +2

May 21, 2026, cs.CLJ/K move · Enter open · S save

Faithful-MR1: Faithful Multimodal Reasoning via Anchoring and Reinforcing Visual Attention

Changyuan Tian, Zhicong Lu, Huaxing Liu, Xiang Wang+6

AMAP, Alibaba Group · University of Chinese Academy of Sciences · Tsinghua University+1

Multimodal Reasoning Recent Vision-Language Models Visual Evidence Reinforcement Learning With Verifiable Reward

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising paradigm for advancing complex reasoning in large language models, and recent work extends RLVR to multimodal large language models (MLLMs). This transfer, however, surfaces a faithfulness challenge: faithful perception of task-relevant visual evidence and faithful use of that evidence during reasoning, leading to unsatisfactory gains on multimodal benchmarks. Specifically, existing perception supervision often operates on textual descriptions rather than natively on image regions, and faithful use is largely overlooked, exposing the perception-reasoning disconnect where correctly perceived evidence is dropped or contradicted during reasoning. To close these gaps, we propose Faithful-MR1, a training framework that anchors and reinforces visual attention to address both halves of faithful multimodal reasoning. The Anchoring stage turns perception into an explicit pre-reasoning subtask, supervising a dedicated <Focus> token's attention directly against image regions rather than through textual descriptions. The Reinforcing stage exposes faithful use through counterfactual image intervention, rewarding answer-correct trajectories that concentrate visual attention where vision causally matters. Extensive experiments demonstrate that Faithful-MR1 outperforms recent multimodal reasoning baselines on both Qwen2.5-VL-Instruct 3B and 7B backbones while using substantially less training data.

Abstract

Explore similar work

Faithful-MR1: Faithful Multimodal Reasoning via Anchoring and Reinforcing Visual Attention

Balancing Efficiency and Efficacy: Training-Free Attention-Guided Switching Between Explicit and Latent Thoughts for MLLMs

Look on Demand: A Cognitive Scheduling Framework for Visual Evidence Acquisition in Multimodal Reasoning