VAttnPO: Visual Attention-Guided Policy Optimization

Lihao Zheng; Zhenwei Shao; Zhou Yu; Yan Yang; Haoyi Sun; Qian Wang; Ning Mao; Pan Zhou

cs.CVOct 12, 2026

VAttnPO: Visual Attention-Guided Policy Optimization

Authors: Lihao Zheng, Zhenwei Shao, Zhou Yu, Yan Yang, Haoyi Sun, Qian Wang, Ning Mao, Pan Zhou

Organizations: School of Computer Science and Technology, Hangzhou Dianzi University · Base Model, Li Auto

Abstract

Multimodal reinforcement learning with verifiable rewards commonly assigns one sequence advantage to an entire response. This outcome-level signal distinguishes overall response quality but leaves changes in visual-evidence use across generated tokens implicit. We introduce VAttnPO (Visual Attention-Guided Policy Optimization), which uses answer-conditioned visual attention to refine token-level credit while preserving the verifier-derived trajectory-mean advantage. A frozen old policy constructs a visual reference by teacher forcing the known correct answer and produces a visual distribution for each rollout token. VAttnPO measures their agreement in visual-attention space using Jensen--Shannon divergence, rather than reading guidance from teacher output-token distributions or introducing an attention-matching objective. Trajectory centering, relative-position weighting, and credit balancing turn this agreement into a zero-mean, position-aware correction to the GRPO sequence advantage. Token intervention, visual-index permutation, and trajectory analysis provide diagnostic evidence linking this signal to model behavior, visual correspondence, and generation position. On Qwen3-VL-8B-Instruct, VAttnPO raises six-task image AVE from 70.97 with GRPO to 73.50 and scores higher on every reported structured, video, and MME metric. Gains over GRPO also hold on Qwen2.5-VL-7B-Instruct and InternVL3-8B-Instruct.

Figures & tables

Panel of Figure 1 — Figure 1: Diagnostic evidence from visual attention. (a) Paired Cohen’s $d$ for High tokens against Low, Random, and Position-matched controls. (b) Wrong-minus-correct JSD gaps under the correct-answer and patch-shuffled references. (c) Mean JSD for correct and incorrect responses and their wrong-minus-correct gap over completion progress. The final point denotes the last valid marker. Error bars in (a) and (b), and shaded bands around the correct and incorrect curves in (c), show confidence intervals obtained by bootstrap resampling at the example level.

Figure 2: VAttnPO overview. Within each update, the frozen old policy extracts visual distributions by teacher forcing the correct answer and by processing sampled responses. Their similarity is centered within each trajectory, weighted by relative position, and balanced into a token correction of the sequence advantage.

Model Type	Method	Fine-grained Perception			Reasoning			AVE
Model Type	Method	VStar	HRBench 4K	MMVP	MMMU-Pro	MuirBench	BLINK	AVE
Base	Base model	81.15	77.38	80.00	52.95	60.35	67.18	69.84
RLVR	GRPO	83.25	77.50	81.00	54.28	61.54	68.23	70.97
RLVR	SAPO	84.29	77.75	81.33	53.82	60.27	68.33	70.97
On-policy Distillation	GKD	79.06	79.00	76.33	53.87	62.62	63.97	69.14
	OPD	85.34	78.75	79.33	54.57	63.65	65.97	71.27

Table 1: Image reasoning and fine-grained perception. AVE is the mean of the six tasks shown. Best results are shown in bold .

Model Type	Method	Structured & Document				Hallucination		CountBenchQA
Model Type	Method	ChartQA	OCRBench	DocVQA	InfoVQA	POPE	HallusionBench	CountBenchQA
Base	Base model	82.72	88.10	95.27	82.38	87.77	70.87	88.71
RLVR	GRPO	82.92	89.50	95.40	82.97	88.21	71.61	89.53
RLVR	SAPO	82.92	89.50	95.41	82.44	88.19	70.56	89.94
On-policy Distillation	GKD	81.36	88.70	94.86	82.11	88.44	71.50	85.42
	OPD	81.44	88.70	95.04	82.18	87.58	73.29	88.71

Table 2: Structured, document, and reliability results. HallusionBench reports aAcc only. Best results are shown in bold .

Model Type	Method	Video Understanding				MME
Model Type	Method	MVBench	LongVideoBench	TempCompass	Video-MME	MME-per	MME-rea
Base	Base model	70.72	62.2	69.10	67.7	1724.7	671.4
RLVR	GRPO	71.25	62.8	70.04	68.4	1736.1	673.9
RLVR	SAPO	71.58	62.9	69.97	68.4	1733.2	652.1
On-policy Distillation	GKD	68.53	57.3	70.04	64.4	1722.7	656.8
	OPD	70.97	62.1	71.38	66.6	1727.5	681.8

Table 3: Video understanding and MME. MME-per and MME-rea denote MME-Perception and MME-Reasoning. Best results are shown in bold .

Table 4: Four controlled ablation groups. AVE denotes the mean over the six image tasks in Table 1 .

Panel of Figure 3 — Figure 3: (a) AVE under image-only and image–video training. (b) AVE at steps 200 and 400.

Backbone	Base	GRPO	VAttnPO	$\Delta$
Qwen2.5-VL-7B-Instruct	59.44	61.64	62.55	+0.91
InternVL3-8B-Instruct	59.54	60.46	61.03	+0.56

Table 5: Cross-backbone AVE.

\Delta

denotes the VAttnPO gain over GRPO.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Data type	Number of examples
Image	64,473
Video	103,987
Total	168,460

Appendix

Table A.1: Training-set composition.

Item	Setting
Backbone	Qwen3-VL-8B-Instruct
SFT warm-up	None
Training steps	400
Loss type	GRPO
Learning rate	$2\times 10^{-6}$
Optimizer / schedule	AdamW Torch Fused / cosine

Appendix

Table B.1: Shared training configuration for GRPO and VAttnPO.

Variant	Sole change
Position-only	Center and unit-RMS normalize each trajectory’s position curve, then multiply by the fixed $r$ -scale proxy 0.14
No position	Set $g^{\mathrm{pos}}_{i,t}=1$ , so $u_{i,t}=r_{i,t}$
Position-shuffled	Deterministically permute $g^{\mathrm{pos}}_{i,t}$ over valid tokens per response with seed 42
No balance	Set $d_{i,t}=u_{i,t}$
Mean recenter	Set $d_{i,t}=u_{i,t}-L_{i}^{-1}\sum_{v}u_{i,v}$

Appendix

Table F.1: Controlled variants and their sole change from VAttnPO.

Variant	VStar	MMMU-Pro	MuirBench	BLINK	HRBench 4K	MMVP	AVE
GRPO	83.25	54.28	61.54	68.23	77.50	81.00	70.97
Position-only	83.34	54.18	62.23	67.85	78.12	79.33	70.84
No balance	83.77	54.41	62.04	67.75	77.38	79.67	70.84
Mean recenter	84.82	54.47	62.04	68.23	77.88	80.67	71.35
No position	85.34	55.55	63.35	68.70	79.50	81.67	72.35
Position-shuffled	84.95	55.48	63.25	68.39	79.38	81.67	72.19

Appendix

Table F.2: Per-benchmark results for position and credit-balance variants.

Backbone	Method	VStar	MMMU-Pro	MuirBench	BLINK	HRBench 4K	MMVP	AVE
Qwen2.5-VL-7B-Instruct	Base	73.82	38.44	53.42	51.76	68.88	70.33	59.44
	GRPO	75.92	39.08	53.69	54.55	72.25	74.33	61.64
	VAttnPO	76.39	40.02	55.58	54.71	73.62	75.00	62.55
InternVL3-8B-Instruct	Base	68.59	26.36	55.65	56.60	70.37	79.67	59.54
	GRPO	71.20	27.59	56.42	56.81	70.75	80.00	60.46
	VAttnPO	71.68	28.36	56.98	57.08	71.38	80.67	61.03

Appendix

Table H.1: Cross-backbone results on the six tasks used to compute AVE.

(a) Qwen3-VL-8B-Instruct Layer Depth

\Delta^{\mathrm{late}}_{\ell}

95% CI L8 23%

4.4\times 10^{-3}

[1.8\times 10^{-3},\,7.1\times 10^{-3}]

L12 34%

3.6\times 10^{-3}

[7.0\times 10^{-4},\,6.6\times 10^{-3}]

L16 46%

1.0\times 10^{-2}

[6.3\times 10^{-3},\,1.4\times 10^{-2}]

L20 57%

\mathbf{1.1\times 10^{-2}}

\mathbf{[6.5\times 10^{-3},\,1.5\times 10^{-2}]}

L24 69%

6.7\times 10^{-3}

[3.2\times 10^{-3},\,1.0\times 10^{-2}]

L28 80%

6.9\times 10^{-3}

[3.2\times 10^{-3},\,1.1\times 10^{-2}]

L32 91%

5.3\times 10^{-3}

[2.4\times 10^{-3},\,8.2\times 10^{-3}]

(b) Qwen2.5-VL-7B-Instruct Layer Depth

\Delta^{\mathrm{late}}_{\ell}

95% CI L0 0%

8.2\times 10^{-4}

[9.5\times 10^{-5},\,1.6\times 10^{-3}]

L4 15%

3.5\times 10^{-4}

[-4.1\times 10^{-3},\,4.5\times 10^{-3}]

L9 33%

1.1\times 10^{-3}

[-4.1\times 10^{-3},\,5.6\times 10^{-3}]

L14 52%

4.8\times 10^{-3}

[-1.5\times 10^{-3},\,1.1\times 10^{-2}]

L18 67%

3.5\times 10^{-3}

[-8.2\times 10^{-4},\,7.3\times 10^{-3}]

L22 81%

2.8\times 10^{-3}

[-3.3\times 10^{-3},\,8.9\times 10^{-3}]

L27 100%

2.8\times 10^{-3}

[-2.3\times 10^{-3},\,8.1\times 10^{-3}]

\Delta^{\mathrm{late}}_{\ell}

95% CI L0 0%

6.9\times 10^{-4}

[1.1\times 10^{-4},\,1.3\times 10^{-3}]

L4 15%

1.3\times 10^{-3}

[-1.1\times 10^{-3},\,3.3\times 10^{-3}]

L9 33%

2.8\times 10^{-3}

[-3.2\times 10^{-4},\,5.5\times 10^{-3}]

L14 52%

4.3\times 10^{-3}

[-2.0\times 10^{-3},\,1.1\times 10^{-2}]

L18 67%

4.3\times 10^{-3}

[1.6\times 10^{-4},\,8.7\times 10^{-3}]

L22 81%

3.9\times 10^{-3}

[-7.7\times 10^{-4},\,8.6\times 10^{-3}]

L27 100%

2.3\times 10^{-3}

[-1.6\times 10^{-4},\,5.0\times 10^{-3}]

Appendix

Table I.1: Layer-wise wrong-minus-correct JSD gaps, averaged equally over the final five completion-progress bins. Positive gaps indicate lower reference alignment for incorrect responses; intervals are 95% prompt-cluster bootstrap. Bold marks Qwen3-VL layer 20, used in the main experiments, and does not denote statistical superiority.

RL for VLMs

Reinforcement learning for vision-language models aligns multimodal reasoning, tool use, and action with task-specific feedback.

See all 476 papers

Visually-Guided Policy Optimization for Multimodal Reasoning
Apr 10, 2026Zengbin Wang, Feng Xiong, Liang Lin +5VLM Reasoning RL for VLMs
AMAP, Alibaba Group · SYSU · BUPT
Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models
Apr 2, 2026Zekai Ye, Qiming Li, Xiaocheng Feng +6RL for VLMs Large Vision-Language Models
Harbin Institute of Technology · Peng Cheng Laboratory · Huawei Technologies Co., Ltd
Structured Role-Aware Policy Optimization for Multimodal Reasoning
May 8, 2026Bingqing Jiang, Difan ZouToken-Level Credit Assignment Multimodal Reasoning
School of Computing & Data Science, The University of Hong Kong · School of Computing & Data Science and Institute of Data Science, The University of Hong Kong

Apr 10, 2026J/K move · Enter open · S save

Visually-Guided Policy Optimization for Multimodal Reasoning

Zengbin Wang, Feng Xiong, Liang Lin, Xuecai Hu+4

AMAP, Alibaba Group · SYSU · BUPT

VLM Reasoning RL for VLMs Vision-Language Models Faithfulness in VLMs Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) has significantly advanced the reasoning ability of vision-language models (VLMs). However, the inherent text-dominated nature of VLMs often leads to insufficient visual faithfulness, characterized by sparse attention activation to visual tokens. More importantly, our empirical analysis reveals that temporal visual forgetting along reasoning steps exacerbates this deficiency. To bridge this gap, we propose Visually-Guided Policy Optimization (VGPO), a novel framework to reinforce visual focus during policy optimization. Specifically, VGPO initially introduces a Visual Attention Compensation mechanism that leverages visual similarity to localize and amplify visual cues, while progressively elevating visual expectations in later steps to counteract visual forgetting. Building on this mechanism, we implement a dual-grained advantage re-weighting strategy: the intra-trajectory level highlights tokens exhibiting relatively high visual activation, while the inter-trajectory level prioritizes trajectories demonstrating superior visual accumulation. Extensive experiments demonstrate that VGPO achieves better visual activation and superior performance in mathematical multimodal reasoning and visual-dependent tasks. The code has been released at https://github.com/wzb-bupt/VGPO.

Abstract

RL for VLMs

Explore similar work

Visually-Guided Policy Optimization for Multimodal Reasoning

Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models

Structured Role-Aware Policy Optimization for Multimodal Reasoning