Visually-Guided Policy Optimization for Multimodal Reasoning

Zengbin Wang; Feng Xiong; Liang Lin; Xuecai Hu; Yong Wang; Yanlin Wang; Man Zhang; Xiangxiang Chu

cs.CVApr 10, 2026

Visually-Guided Policy Optimization for Multimodal Reasoning

Authors: Zengbin Wang, Feng Xiong, Liang Lin, Xuecai Hu, Yong Wang, Yanlin Wang, Man Zhang, Xiangxiang Chu

Organizations: AMAP, Alibaba Group · SYSU · BUPT

Abstract

Reinforcement learning with verifiable rewards (RLVR) has significantly advanced the reasoning ability of vision-language models (VLMs). However, the inherent text-dominated nature of VLMs often leads to insufficient visual faithfulness, characterized by sparse attention activation to visual tokens. More importantly, our empirical analysis reveals that temporal visual forgetting along reasoning steps exacerbates this deficiency. To bridge this gap, we propose Visually-Guided Policy Optimization (VGPO), a novel framework to reinforce visual focus during policy optimization. Specifically, VGPO initially introduces a Visual Attention Compensation mechanism that leverages visual similarity to localize and amplify visual cues, while progressively elevating visual expectations in later steps to counteract visual forgetting. Building on this mechanism, we implement a dual-grained advantage re-weighting strategy: the intra-trajectory level highlights tokens exhibiting relatively high visual activation, while the inter-trajectory level prioritizes trajectories demonstrating superior visual accumulation. Extensive experiments demonstrate that VGPO achieves better visual activation and superior performance in mathematical multimodal reasoning and visual-dependent tasks. The code has been released at https://github.com/wzb-bupt/VGPO.

Figures & tables

Figure 1: A multimodal reasoning example with visual input. Our VGPO not only attends to the query and visual elements in the early reasoning steps, like the base model Bai et al. (2025b) , but also exhibits better visual activation and sustains visual focus in the later steps to ensure consistent visual faithfulness.

Figure 2: Analysis of the inference nature of multimodal reasoning trajectory (based on Qwen2.5-VL-7B Bai et al. (2025b) ). (a) An example of attention allocation across image, query, and generated text tokens (normalized to 1 at each step). (b) Average attention statistics on four visual-dependent benchmarks Xiao et al. (2024) ; Li et al. (2023) ; Yue et al. (2025) ; Zhang et al. (2024b) . (c) Distribution of late/early visual accumulation ratios for incorrect (left) vs. correct (right) samples of these four benchmarks. Incorrect samples often exhibit higher visual forgetting.

Figure 3: Overview of V isually- G uided P olicy O ptimization framework. Given query and image, (a) VGPO firstly utilizes the intrinsic hidden state similarity between generated tokens and visual prototype to derive a Visual Focus Score for visual token localization. (b) Then, Visual Attention Compensation (VAC) mechanism leverages this score to re-focus visual tokens, while progressively elevating visual expectations along decoding steps to counteract temporal visual forgetting. (c) Finally, Dual-grained Advantage Re-weighting strategy integrates VAC mechanism into intra- and inter-trajectory levels to explicitly incentivize sustained visual faithfulness during policy updates.

Open-Source General MLLMs (32B-72B)
Models	General Mathematical & Geometric Reasoning						Avg- Math	Vision-dependent Multimodal Reasoning				Avg- Vision
Models	MathVista	MathVerse	WeMath	MMK12	GeoMath	Geo3k	Avg- Math	LogicVista	Counting	MMMU-Pro	MathVerse V	Avg- Vision
Qwen2.5-VL-72B	74.8	65.2 †	69.4 †	65.9 †	53.5 †	54.2 †	63.8	57.9 †	80.5 †	51.1	57.6	61.8
Qwen3-VL-32B	83.8	71.5 †	75.2	63.1 †	45.7 †	65.4 †	67.5	62.2	95.5 †	65.3	76.8	75.0
Open-Source Multimodal Reasoning Models ( $\sim$ 7B)
ThinkLite-VL-7B †	73.6	33.7	43.7	50.5	44.6	45.3	48.6	44.3	88.5	32.3	30.9	49.0

Table 1: Performance comparisons across mathematical and vision-dependent multimodal reasoning benchmarks. “ ∗ ” indicates NoisyRollout is trained on Geo3K. “ † ” indicates our reproduction with official checkpoints. Other values are sourced from the official technical report. Bold and underlined indicate best and second-best results.

Exploring More Scalable Models (7B $\xrightarrow{}$ 3B, 32B)
Models	General Mathematical & Geometric Reasoning						Avg- Math	Vision-dependent Multimodal Reasoning				Avg- Vision
Models	MathVista	MathVerse	WeMath	MMK12	GeoMath	Geo3k	Avg- Math	LogicVista	Counting	MMMU-Pro	MathVerse V	Avg- Vision
Qwen2.5-VL-3B	54.0	44.1	50.2	44.1	44.3	27.6	44.1	39.2	58.5	28.6	37.5	41.0
+ DAPO	63.9	57.1	62.9	64.1	47.6	36.3	55.3 △25.4%	43.0	65.5	31.5	53.1	48.3 △17.8%
+ VGPO	65.0	61.4	62.1	71.6	50.2	36.1	57.7 △30.8%	45.9	78.0	32.5	58.1	53.6 △30.7%
Qwen2.5-VL-32B	76.8	62.4	68.8	60.0	53.7	50.8	62.1	55.7	80.5	46.0	58.4	60.2

Table 2: Performance comparisons across different model scales and training datasets.

Method	Avg- Math	Avg- Vision	Overall
DAPO Baseline	63.8	59.6	62.2
+ Intra-trajectory	66.1	62.5	64.6
+ Inter-trajectory	65.3	62.0	64.0
+ Intra- & Inter-trajectory	66.6	63.3	65.3

Table 3: Ablation study on the impact of Intra- and Inter-trajectory re-weighting strategies.

Method	Avg- Math	Avg- Vision	Overall
Qwen2.5-VL-7B	50.0	48.7	49.5
+ DAPO	63.8	59.6	62.2
+ DAPO w/ Entropy	65.6	61.9	64.1
+ DAPO w/ KL perception	65.7	61.3	63.9
+ VGPO (Ours)	66.6	63.3	65.3

Table 4: Comparisons with other advantage shaping strategies, including the Entropy-based Cheng et al. (2026) and KL-based Huang et al. (2025a) methods.

Panel of Figure 4 — Figure 4: Training dynamics based on Qwen2.5-VL-7B: (a) training rewards and (b) validation accuracy on MMK12 Meng et al. (2025) across GRPO Shao et al. (2024) , DAPO Yu et al. (2026) , and our VGPO.

Panel of Figure 5 — Figure 5: Ablation study of training dynamics of the late/early ratio and performance on hyperparameters $\beta$ , $\kappa$ , and $\gamma$ . The Late/Early Ratio is calculated by dividing the visual attention score of the late reasoning stage by that of the early stage.

Method	Avg- Math	Avg- Vision	Overall
Baseline (DAPO)	63.8	59.6	62.2
+ Step-Function	64.7	60.7	63.1
+ Exponential	65.1	61.0	63.5
+ Linear (Ours)	66.6	63.3	65.3

Table 5: Ablation study on the impact of different compensation schedules.

Method	Avg- Math	Avg- Vision	Overall
Baseline (DAPO)	63.8	59.6	62.2
+ Full-trajectory	53.0	54.2	53.5
+ Late-trajectory (Ours)	66.6	63.3	65.3

Table 6: Ablation study on the impact of Full-trajectory versus Late-trajectory compensation.

Figure 6: Comparison of the vision attention ratio distribution before and after our VGPO across four visual-dependent multimodal reasoning benchmarks.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

General Settings
Parameter	Configuration
Rollout Number	8
Learning Rate	1e-6
Global Batch Size	128
Rollout Batch Size	512
Val Batch Size	1024

Appendix

Table 7: Experimental hyperparameter configurations of our VGPO across 3B, 7B, and 32B-based backbones and different scalable training datasets.

Models	General Mathematical & Geometric Reasoning						Avg- Math	Vision-dependent Multimodal Reasoning				Avg- Vision
Models	MathVista	MathVerse	WeMath	MMK12	GeoMath	Geo3k	Avg- Math	LogicVista	Counting	MMMU-Pro	MathVerse V	Avg- Vision
DAPO Baseline	68.7	69.6	70.8	77.0	51.3	45.6	63.8	47.4	85.5	39.0	66.6	59.6
+ Intra-trajectory	74.1	71.1	69.8	81.7	54.3	45.6	66.1	48.1	95.0	40.2	66.5	62.5
+ Inter-trajectory	72.1	68.8	69.9	80.7	54.9	45.3	65.3	47.9	95.5	40.2	64.6	62.0
+ Intra- & Inter-trajectory	74.1	71.6	72.5	81.5	54.3	45.8	66.6	49.4	95.5	40.5	67.6	63.3

Appendix

Table 8: The detailed ablation study on the impact of Intra- and Inter-trajectory re-weighting strategies.

Models	General Mathematical & Geometric Reasoning						Avg- Math	Vision-dependent Multimodal Reasoning				Avg- Vision
Models	MathVista	MathVerse	WeMath	MMK12	GeoMath	Geo3k	Avg- Math	LogicVista	Counting	MMMU-Pro	MathVerse V	Avg- Vision
Qwen2.5-VL-7B	68.5	40.2	47.8	49.4	51.2	42.9	50.0	45.2	76.5	36.4	36.6	48.7
+ DAPO	68.7	69.6	70.8	77.0	51.3	45.6	63.8	47.4	85.5	39.0	66.6	59.6
+ DAPO w/ Entropy	70.2	72.1	70.8	80.9	53.9	45.6	65.6	48.3	90.5	40.1	68.7	61.9
+ DAPO w/ KL perception	70.5	71.1	70.6	81.8	53.2	46.9	65.7	48.8	88.5	40.2	67.6	61.3
+ VGPO (Ours)	74.1	71.6	72.5	81.5	54.3	45.8	66.6	49.4	95.5	40.5	67.6	63.3

Appendix

Table 9: The detailed comparisons with other advantage shaping strategies, including the Entropy-based Cheng et al. (2026) and KL-based Huang et al. (2025a) methods.

Panel of Figure 7 — Figure 7: Training dynamics of Qwen2.5-VL-3B: (a) training rewards and (b) validation accuracy on MMK12 Meng et al. (2025) across DAPO Yu et al. (2026) , and our VGPO.

Panel of Figure 8 — Figure 8: Training dynamics of Qwen2.5-VL-32B: (a) training rewards and (b) validation accuracy on MMK12 Meng et al. (2025) across DAPO Yu et al. (2026) , and our VGPO.

Compensation Schedules	General Mathematical & Geometric Reasoning						Avg- Math	Vision-dependent Multimodal Reasoning				Avg- Vision
Compensation Schedules	MathVista	MathVerse	WeMath	MMK12	GeoMath	Geo3k	Avg- Math	LogicVista	Counting	MMMU-Pro	MathVerse V	Avg- Vision
Baseline (DAPO)	68.7	69.6	70.8	77.0	51.3	45.6	63.8	47.4	85.5	39.0	66.6	59.6
w/ Step-Function	69.1	69.2	69.4	80.3	52.3	47.6	64.7	47.5	91.5	39.2	64.7	60.7
w/ Exponential	70.6	69.9	69.3	81.4	54.2	44.9	65.1	49.0	90.5	38.8	65.5	61.0
w/ Linear (Ours)	74.1	71.6	72.5	81.5	54.3	45.8	66.6	49.4	95.5	40.5	67.6	63.3

Appendix

Table 10: Results of General Mathematical & Geometric Reasoning and Vision-dependent Multimodal Reasoning tasks with different compensation schedules. Note : For the Exponential schedule, we test powers of 1.0 and 2.0 (reporting the best result from 2.0). We test 1.0 for the Step-wise schedule.

Models	General Mathematical & Geometric Reasoning						Avg- Math	Vision-dependent Multimodal Reasoning				Avg- Vision
Models	MathVista	MathVerse	WeMath	MMK12	GeoMath	Geo3k	Avg- Math	LogicVista	Counting	MMMU-Pro	MathVerse V	Avg- Vision
Baseline (DAPO, Qwen2.5-VL-7B)	68.7	69.6	70.8	77.0	51.3	45.6	63.8	47.4	85.5	39.0	66.6	59.6
+VGPO (Ours)	74.1	71.6	72.5	81.5	54.3	45.8	66.6 △4.4%	49.4	95.5	40.5	67.6	63.3 △6.2%
Baseline (DAPO, Qwen2.5-VL-3B)	63.9	57.1	62.9	64.1	47.6	36.3	55.3	43.0	65.5	31.5	53.1	48.3
+VGPO (Ours)	65.0	61.4	62.1	71.6	50.2	36.1	57.7 △4.3%	45.9	78.0	32.5	58.1	53.6 △11.0%
Baseline (DAPO, Qwen2-VL-2B)	50.8	33.2	36.3	42.3	34.8	8.7	34.4	30.0	66.5	19.8	31.6	37.0

Appendix

Table 11: Performance comparison of stronger to weaker visual encoders on General Mathematical & Geometric Reasoning and Vision-dependent Multimodal Reasoning tasks.

Compensation Schedules	General Mathematical & Geometric Reasoning						Avg- Math	Vision-dependent Multimodal Reasoning				Avg- Vision
Compensation Schedules	MathVista	MathVerse	WeMath	MMK12	GeoMath	Geo3k	Avg- Math	LogicVista	Counting	MMMU-Pro	MathVerse V	Avg- Vision
Baseline (DAPO)	68.7	69.6	70.8	77.0	51.3	45.6	63.8	47.4	85.5	39.0	66.6	59.6
Full-trajectory	71.4	50.3	53.8	60.1	44.9	37.3	53.0	40.5	92.0	36.4	47.8	54.2
Late-trajectory (Ours)	74.1	71.6	72.5	81.5	54.3	45.8	66.6	49.4	95.5	40.5	67.6	63.3

Appendix

Table 12: Performance comparison of Full-trajectory compensation versus our Late-trajectory compensation strategy.

Explore similar work

Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models
Apr 2, 2026Zekai Ye, Qiming Li, Xiaocheng Feng +6RL for VLMs Large Vision-Language Models
Harbin Institute of Technology · Peng Cheng Laboratory · Huawei Technologies Co., Ltd
Beyond Where to Look: Trajectory-Guided Reinforcement Learning for Multimodal RLVR
Mar 27, 2026Jinda Lu, Junkang Wu, Jinghan Li +6RL for VLMs Multimodal Reasoning
University of Science and Technology of China · Pecking University · Institute of Software, Chinese Academy of Sciences
Anchoring What Matters: A Dual-Level Learning Framework for Visually-Grounded Multimodal Reasoning
Sep 16, 2026Xinxin Song, Siyuan Li, Tingxiong Xiao +1Credit Assignment in RL Experience Replay
Department of Automation, Tsinghua University · Institute for Brain and Cognitive Science, Tsinghua University

Apr 2, 2026J/K move · Enter open · S save

Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models

Zekai Ye, Qiming Li, Xiaocheng Feng, Ruihan Chen+5

Harbin Institute of Technology · Peng Cheng Laboratory · Huawei Technologies Co., Ltd

RL for VLMs Large Vision-Language Models VLM Reasoning Token-Level Credit Assignment

Open paper AskPDF arXiv

While Reinforcement Learning from Verifiable Rewards (RLVR) has advanced reasoning in Large Vision-Language Models (LVLMs), prevailing frameworks suffer from a foundational methodological flaw: by distributing identical advantages across all generated tokens, these methods inherently dilute the learning signals essential for optimizing the critical, visually-grounded steps of multimodal reasoning. To bridge this gap, we formulate Token Visual Dependency, quantifying the causal information gain of visual inputs via the Kullback-Leibler (KL) divergence between visual-conditioned and text-only predictive distributions. Revealing that this dependency is highly sparse and semantically pivotal, we introduce Perception-Grounded Policy Optimization (PGPO), which is a novel fine-grained credit assignment framework that dynamically reshapes advantages at the token level. Through a threshold-gated, mass-conserving mechanism, PGPO actively amplifies learning signals for visually-dependent tokens while suppressing gradient noise from linguistic priors. Extensive experiments based on the Qwen2.5-VL series across seven challenging multimodal reasoning benchmarks demonstrate that PGPO boosts models by 18.7% on average. Both theoretical and empirical analyses confirm that PGPO effectively reduces gradient variance, prevents training collapse, and acts as a potent regularizer for robust, perception-grounded multimodal reasoning. Code will be released on https://github.com/Yzk1114/PGPO.

Abstract

Explore similar work

Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models

Beyond Where to Look: Trajectory-Guided Reinforcement Learning for Multimodal RLVR

Anchoring What Matters: A Dual-Level Learning Framework for Visually-Grounded Multimodal Reasoning