VAttnPO: Visual Attention-Guided Policy Optimization

Lihao Zheng; Zhenwei Shao; Zhou Yu; Yan Yang; Haoyi Sun; Qian Wang; Ning Mao; Pan Zhou

cs.CVOct 12, 2026

VAttnPO: Visual Attention-Guided Policy Optimization

Authors: Lihao Zheng, Zhenwei Shao, Zhou Yu, Yan Yang, Haoyi Sun, Qian Wang, Ning Mao, Pan Zhou

Organizations: School of Computer Science and Technology, Hangzhou Dianzi University · Base Model, Li Auto

Abstract

Multimodal reinforcement learning with verifiable rewards commonly assigns one sequence advantage to an entire response. This outcome-level signal distinguishes overall response quality but leaves changes in visual-evidence use across generated tokens implicit. We introduce VAttnPO (Visual Attention-Guided Policy Optimization), which uses answer-conditioned visual attention to refine token-level credit while preserving the verifier-derived trajectory-mean advantage. A frozen old policy constructs a visual reference by teacher forcing the known correct answer and produces a visual distribution for each rollout token. VAttnPO measures their agreement in visual-attention space using Jensen--Shannon divergence, rather than reading guidance from teacher output-token distributions or introducing an attention-matching objective. Trajectory centering, relative-position weighting, and credit balancing turn this agreement into a zero-mean, position-aware correction to the GRPO sequence advantage. Token intervention, visual-index permutation, and trajectory analysis provide diagnostic evidence linking this signal to model behavior, visual correspondence, and generation position. On Qwen3-VL-8B-Instruct, VAttnPO raises six-task image AVE from 70.97 with GRPO to 73.50 and scores higher on every reported structured, video, and MME metric. Gains over GRPO also hold on Qwen2.5-VL-7B-Instruct and InternVL3-8B-Instruct.

Figures & tables

Panel of Figure 1 — Figure 1: Diagnostic evidence from visual attention. (a) Paired Cohen’s $d$ for High tokens against Low, Random, and Position-matched controls. (b) Wrong-minus-correct JSD gaps under the correct-answer and patch-shuffled references. (c) Mean JSD for correct and incorrect responses and their wrong-minus-correct gap over completion progress. The final point denotes the last valid marker. Error bars in (a) and (b), and shaded bands around the correct and incorrect curves in (c), show confidence intervals obtained by bootstrap resampling at the example level.

Figure 2: VAttnPO overview. Within each update, the frozen old policy extracts visual distributions by teacher forcing the correct answer and by processing sampled responses. Their similarity is centered within each trajectory, weighted by relative position, and balanced into a token correction of the sequence advantage.

Model Type	Method	Fine-grained Perception			Reasoning			AVE
Model Type	Method	VStar	HRBench 4K	MMVP	MMMU-Pro	MuirBench	BLINK	AVE
Base	Base model	81.15	77.38	80.00	52.95	60.35	67.18	69.84
RLVR	GRPO	83.25	77.50	81.00	54.28	61.54	68.23	70.97
RLVR	SAPO	84.29	77.75	81.33	53.82	60.27	68.33	70.97
On-policy Distillation	GKD	79.06	79.00	76.33	53.87	62.62	63.97	69.14
	OPD	85.34	78.75	79.33	54.57	63.65	65.97	71.27

Table 1: Image reasoning and fine-grained perception. AVE is the mean of the six tasks shown. Best results are shown in bold .

Model Type	Method	Structured & Document				Hallucination		CountBenchQA
Model Type	Method	ChartQA	OCRBench	DocVQA	InfoVQA	POPE	HallusionBench	CountBenchQA
Base	Base model	82.72	88.10	95.27	82.38	87.77	70.87	88.71
RLVR	GRPO	82.92	89.50	95.40	82.97	88.21	71.61	89.53
RLVR	SAPO	82.92	89.50	95.41	82.44	88.19	70.56	89.94
On-policy Distillation	GKD	81.36	88.70	94.86	82.11	88.44	71.50	85.42
	OPD	81.44	88.70	95.04	82.18	87.58	73.29	88.71

Table 2: Structured, document, and reliability results. HallusionBench reports aAcc only. Best results are shown in bold .

Model Type	Method	Video Understanding				MME
Model Type	Method	MVBench	LongVideoBench	TempCompass	Video-MME	MME-per	MME-rea
Base	Base model	70.72	62.2	69.10	67.7	1724.7	671.4
RLVR	GRPO	71.25	62.8	70.04	68.4	1736.1	673.9
RLVR	SAPO	71.58	62.9	69.97	68.4	1733.2	652.1
On-policy Distillation	GKD	68.53	57.3	70.04	64.4	1722.7	656.8
	OPD	70.97	62.1	71.38	66.6	1727.5	681.8

Table 3: Video understanding and MME. MME-per and MME-rea denote MME-Perception and MME-Reasoning. Best results are shown in bold .

Table 4: Four controlled ablation groups. AVE denotes the mean over the six image tasks in Table 1 .

Panel of Figure 3 — Figure 3: (a) AVE under image-only and image–video training. (b) AVE at steps 200 and 400.

Backbone	Base	GRPO	VAttnPO	$\Delta$
Qwen2.5-VL-7B-Instruct	59.44	61.64	62.55	+0.91
InternVL3-8B-Instruct	59.54	60.46	61.03	+0.56

Table 5: Cross-backbone AVE.

\Delta

denotes the VAttnPO gain over GRPO.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Data type	Number of examples
Image	64,473
Video	103,987
Total	168,460

Appendix

Table A.1: Training-set composition.

Item	Setting
Backbone	Qwen3-VL-8B-Instruct
SFT warm-up	None
Training steps	400
Loss type	GRPO
Learning rate	$2\times 10^{-6}$
Optimizer / schedule	AdamW Torch Fused / cosine

Appendix

Table B.1: Shared training configuration for GRPO and VAttnPO.

Variant	Sole change
Position-only	Center and unit-RMS normalize each trajectory’s position curve, then multiply by the fixed $r$ -scale proxy 0.14
No position	Set $g^{\mathrm{pos}}_{i,t}=1$ , so $u_{i,t}=r_{i,t}$
Position-shuffled	Deterministically permute $g^{\mathrm{pos}}_{i,t}$ over valid tokens per response with seed 42
No balance	Set $d_{i,t}=u_{i,t}$
Mean recenter	Set $d_{i,t}=u_{i,t}-L_{i}^{-1}\sum_{v}u_{i,v}$

Appendix

Table F.1: Controlled variants and their sole change from VAttnPO.

Variant	VStar	MMMU-Pro	MuirBench	BLINK	HRBench 4K	MMVP	AVE
GRPO	83.25	54.28	61.54	68.23	77.50	81.00	70.97
Position-only	83.34	54.18	62.23	67.85	78.12	79.33	70.84
No balance	83.77	54.41	62.04	67.75	77.38	79.67	70.84
Mean recenter	84.82	54.47	62.04	68.23	77.88	80.67	71.35
No position	85.34	55.55	63.35	68.70	79.50	81.67	72.35
Position-shuffled	84.95	55.48	63.25	68.39	79.38	81.67	72.19

Appendix

Table F.2: Per-benchmark results for position and credit-balance variants.

Backbone	Method	VStar	MMMU-Pro	MuirBench	BLINK	HRBench 4K	MMVP	AVE
Qwen2.5-VL-7B-Instruct	Base	73.82	38.44	53.42	51.76	68.88	70.33	59.44
	GRPO	75.92	39.08	53.69	54.55	72.25	74.33	61.64
	VAttnPO	76.39	40.02	55.58	54.71	73.62	75.00	62.55
InternVL3-8B-Instruct	Base	68.59	26.36	55.65	56.60	70.37	79.67	59.54
	GRPO	71.20	27.59	56.42	56.81	70.75	80.00	60.46
	VAttnPO	71.68	28.36	56.98	57.08	71.38	80.67	61.03

Appendix

Table H.1: Cross-backbone results on the six tasks used to compute AVE.

(a) Qwen3-VL-8B-Instruct Layer Depth

\Delta^{\mathrm{late}}_{\ell}

95% CI L8 23%

4.4\times 10^{-3}

[1.8\times 10^{-3},\,7.1\times 10^{-3}]

L12 34%

3.6\times 10^{-3}

[7.0\times 10^{-4},\,6.6\times 10^{-3}]

L16 46%

1.0\times 10^{-2}

[6.3\times 10^{-3},\,1.4\times 10^{-2}]

L20 57%

\mathbf{1.1\times 10^{-2}}

\mathbf{[6.5\times 10^{-3},\,1.5\times 10^{-2}]}

L24 69%

6.7\times 10^{-3}

[3.2\times 10^{-3},\,1.0\times 10^{-2}]

L28 80%

6.9\times 10^{-3}

[3.2\times 10^{-3},\,1.1\times 10^{-2}]

L32 91%

5.3\times 10^{-3}

[2.4\times 10^{-3},\,8.2\times 10^{-3}]

(b) Qwen2.5-VL-7B-Instruct Layer Depth

\Delta^{\mathrm{late}}_{\ell}

95% CI L0 0%

8.2\times 10^{-4}

[9.5\times 10^{-5},\,1.6\times 10^{-3}]

L4 15%

3.5\times 10^{-4}

[-4.1\times 10^{-3},\,4.5\times 10^{-3}]

L9 33%

1.1\times 10^{-3}

[-4.1\times 10^{-3},\,5.6\times 10^{-3}]

L14 52%

4.8\times 10^{-3}

[-1.5\times 10^{-3},\,1.1\times 10^{-2}]

L18 67%

3.5\times 10^{-3}

[-8.2\times 10^{-4},\,7.3\times 10^{-3}]

L22 81%

2.8\times 10^{-3}

[-3.3\times 10^{-3},\,8.9\times 10^{-3}]

L27 100%

2.8\times 10^{-3}

[-2.3\times 10^{-3},\,8.1\times 10^{-3}]

\Delta^{\mathrm{late}}_{\ell}

95% CI L0 0%

6.9\times 10^{-4}

[1.1\times 10^{-4},\,1.3\times 10^{-3}]

L4 15%

1.3\times 10^{-3}

[-1.1\times 10^{-3},\,3.3\times 10^{-3}]

L9 33%

2.8\times 10^{-3}

[-3.2\times 10^{-4},\,5.5\times 10^{-3}]

L14 52%

4.3\times 10^{-3}

[-2.0\times 10^{-3},\,1.1\times 10^{-2}]

L18 67%

4.3\times 10^{-3}

[1.6\times 10^{-4},\,8.7\times 10^{-3}]

L22 81%

3.9\times 10^{-3}

[-7.7\times 10^{-4},\,8.6\times 10^{-3}]

L27 100%

2.3\times 10^{-3}

[-1.6\times 10^{-4},\,5.0\times 10^{-3}]

Appendix

Table I.1: Layer-wise wrong-minus-correct JSD gaps, averaged equally over the final five completion-progress bins. Positive gaps indicate lower reference alignment for incorrect responses; intervals are 95% prompt-cluster bootstrap. Bold marks Qwen3-VL layer 20, used in the main experiments, and does not denote statistical superiority.

RL for VLMs

Reinforcement learning for vision-language models aligns multimodal reasoning, tool use, and action with task-specific feedback.

See all 476 papers

Apr 10, 2026

Visually-Guided Policy Optimization for Multimodal Reasoning

Reinforcement learning with verifiable rewards (RLVR) has significantly advanced the reasoning ability of vision-language models (VLMs). However, the inherent text-dominated nature of VLMs often leads to insufficient visual faithfulness, characterized by sparse attention activation to visual tokens. More importantly, our empirical analysis reveals that temporal visual forgetting along reasoning steps exacerbates this deficiency. To bridge this gap, we propose Visually-Guided Policy Optimization (VGPO), a novel framework to reinforce visual focus during policy optimization. Specifically, VGPO initially introduces a Visual Attention Compensation mechanism that leverages visual similarity to localize and amplify visual cues, while progressively elevating visual expectations in later steps to counteract visual forgetting. Building on this mechanism, we implement a dual-grained advantage re-weighting strategy: the intra-trajectory level highlights tokens exhibiting relatively high visual activation, while the inter-trajectory level prioritizes trajectories demonstrating superior visual accumulation. Extensive experiments demonstrate that VGPO achieves better visual activation and superior performance in mathematical multimodal reasoning and visual-dependent tasks. The code has been released at https://github.com/wzb-bupt/VGPO.

Zengbin Wang, Feng Xiong, Liang Lin +5

AMAP, Alibaba Group · SYSU · BUPT

VLM Reasoning RL for VLMs Vision-Language Models Faithfulness in VLMs Reinforcement Learning with Verifiable Rewards

Ask PDF

Apr 2, 2026

Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models

While Reinforcement Learning from Verifiable Rewards (RLVR) has advanced reasoning in Large Vision-Language Models (LVLMs), prevailing frameworks suffer from a foundational methodological flaw: by distributing identical advantages across all generated tokens, these methods inherently dilute the learning signals essential for optimizing the critical, visually-grounded steps of multimodal reasoning. To bridge this gap, we formulate Token Visual Dependency, quantifying the causal information gain of visual inputs via the Kullback-Leibler (KL) divergence between visual-conditioned and text-only predictive distributions. Revealing that this dependency is highly sparse and semantically pivotal, we introduce Perception-Grounded Policy Optimization (PGPO), which is a novel fine-grained credit assignment framework that dynamically reshapes advantages at the token level. Through a threshold-gated, mass-conserving mechanism, PGPO actively amplifies learning signals for visually-dependent tokens while suppressing gradient noise from linguistic priors. Extensive experiments based on the Qwen2.5-VL series across seven challenging multimodal reasoning benchmarks demonstrate that PGPO boosts models by 18.7% on average. Both theoretical and empirical analyses confirm that PGPO effectively reduces gradient variance, prevents training collapse, and acts as a potent regularizer for robust, perception-grounded multimodal reasoning. Code will be released on https://github.com/Yzk1114/PGPO.

Zekai Ye, Qiming Li, Xiaocheng Feng +6

Harbin Institute of Technology · Peng Cheng Laboratory · Huawei Technologies Co., Ltd

RL for VLMs Large Vision-Language Models VLM Reasoning Token-Level Credit Assignment

Ask PDF

May 8, 2026

Structured Role-Aware Policy Optimization for Multimodal Reasoning

Reinforcement learning from verifiable rewards (RLVR), especially with Group Relative Policy Optimization (GRPO), has shown strong potential for improving the reasoning capabilities of large vision-language models (LVLMs). However, in multimodal reasoning, final-answer rewards are typically assigned at the sequence level and do not distinguish the functional roles of different tokens, making it difficult to determine whether a correct answer is supported by task-relevant visual evidence. In this paper, we revisit multimodal RLVR from the perspective of role-aware token-level credit assignment, where structured responses are decomposed into perception tokens for extracting visual evidence and reasoning tokens for deriving answers from that evidence. Based on this perspective, we propose Structured Role-aware Policy Optimization (SRPO), which refines the sequence-level GRPO advantage into role-aware token-level advantages without changing the reward function. Specifically, SRPO assigns role-specific credit by using self-distilled on-policy contrasts: perception tokens are emphasized according to their visual dependency under original versus corrupted visual inputs, while reasoning tokens are emphasized according to their consistency with the generated perception. These role-specific signals are further unified through a shared trajectory-level baseline, yielding positive token weights that adjust relative update magnitudes while preserving the original GRPO reward and optimization direction, without requiring external reward models or separate teachers. Experiments across diverse multimodal reasoning benchmarks show that SRPO improves evidence-grounded reasoning, highlighting the importance of moving beyond uniform sequence-level credit toward role-aware optimization for reliable multimodal reasoning.

Bingqing Jiang, Difan Zou

School of Computing & Data Science, The University of Hong Kong · School of Computing & Data Science and Institute of Data Science, The University of Hong Kong

Token-Level Credit Assignment Multimodal Reasoning Reinforcement Learning with Verifiable Rewards RL for VLMs

Ask PDF

Abstract

RL for VLMs

Explore similar work

Visually-Guided Policy Optimization for Multimodal Reasoning

Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models

Structured Role-Aware Policy Optimization for Multimodal Reasoning