Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models

Zekai Ye; Qiming Li; Xiaocheng Feng; Ruihan Chen; Ziming Li; Haoyu Ren; Kun Chen; Dandan Tu

cs.AIApr 2, 2026

Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models

Authors: Zekai Ye, Qiming Li, Xiaocheng Feng, Ruihan Chen, Ziming Li, Haoyu Ren, Kun Chen, Dandan Tu, +1 more

Organizations: Harbin Institute of Technology · Peng Cheng Laboratory · Huawei Technologies Co., Ltd

Abstract

While Reinforcement Learning from Verifiable Rewards (RLVR) has advanced reasoning in Large Vision-Language Models (LVLMs), prevailing frameworks suffer from a foundational methodological flaw: by distributing identical advantages across all generated tokens, these methods inherently dilute the learning signals essential for optimizing the critical, visually-grounded steps of multimodal reasoning. To bridge this gap, we formulate Token Visual Dependency, quantifying the causal information gain of visual inputs via the Kullback-Leibler (KL) divergence between visual-conditioned and text-only predictive distributions. Revealing that this dependency is highly sparse and semantically pivotal, we introduce Perception-Grounded Policy Optimization (PGPO), which is a novel fine-grained credit assignment framework that dynamically reshapes advantages at the token level. Through a threshold-gated, mass-conserving mechanism, PGPO actively amplifies learning signals for visually-dependent tokens while suppressing gradient noise from linguistic priors. Extensive experiments based on the Qwen2.5-VL series across seven challenging multimodal reasoning benchmarks demonstrate that PGPO boosts models by 18.7% on average. Both theoretical and empirical analyses confirm that PGPO effectively reduces gradient variance, prevents training collapse, and acts as a potent regularizer for robust, perception-grounded multimodal reasoning. Code will be released on https://github.com/Yzk1114/PGPO.

Figures & tables

Figure 1: Unlike standard uniform credit assignment, our proposed method dynamically allocates higher reinforcement learning advantage to pivotal tokens that heavily rely on visual perception.

Figure 2: Empirical analysis results of visual dependency. (a) The skewed distribution of token-level visual dependency. (b,c,d) Average visual dependency comparison between specific category tokens and others.

Figure 3: Overview of our proposed PGPO framework. The PGPO pipeline begins by quantifying token visual dependency $\mathcal{S}_{t}$ via KL divergence to isolate the causal information gain from visual inputs. These raw dependency signals are then transformed into bounded token visual dependency score $I_{t}$ through logarithmic compression and min-max normalization. Finally, a threshold-gated mechanism dynamically reshapes the sequence-level GRPO advantage—amplifying learning signals for visually-dependent tokens and suppressing modality-independent noise—while applying a sum-preserving normalization to guarantee stable policy optimization.

Model	Geo3k	MMK12	MathVerse	DynaMath	MathVision	LogicVista	MMMU-Pro	$\text{MathVerse}_{V}$	Avg.
MM-Eureka-7B	40.30	67.68	67.15	55.01	27.13	46.30	30.30	62.41	49.54
VL-Rethinker-7B	40.70	68.30	68.80	54.97	27.91	46.47	37.13	64.96	51.16
NoisyRollout-7B	51.98 ∗	50.00	67.80	54.89	22.10	47.49	34.50	63.78	49.07
R1-ShareVL-7B	41.20	70.94	68.15	54.80	26.13	45.76	35.10	64.34	50.80
Qwen2.5-VL-7B	37.29	43.04	38.36	48.15	17.58	43.01	26.67	33.63	35.97
+ GRPO	30.91	76.45	68.88	54.84	27.66	46.81	36.84	64.78	50.90

Table 1: Main Results (avg@8 acc %).

\text{MathVerse}_{V}

refers to the vision-centric subset of MathVerse. ∗ NoisyRollout is trained using the training set of Geo3k.

Panel of Figure 4 — Figure 4: Training dynamics on Qwen2.5-VL-7B.

Variant	MathVerse	MMMU-Pro
DAPO	58.76	27.20
Suppression-Only ( $\tau=1$ )	60.34	28.71
Boosting-Only ( $\tau=0$ )	59.04	28.15
PGPO w/o Norm.	60.18	28.10
PGPO	62.06	29.33

Table 2: Ablation of advantage modulation strategies.

Threshold ( $\tau$ )	Boosting ( $\beta$ )	MathVerse	MMMU-Pro
0.2	1.0	58.42	27.15
0.2	2.0	59.18	27.89
0.4	1.0	60.36	28.98
0.4	2.0	62.06	29.33
0.6	1.0	59.12	28.40
0.6	2.0	56.11	28.48

Table 3: Hyperparameter sensitivity analysis results.

Method	MMBench	MMStar	MME
Qwen2.5-VL-3B	74.00	43.05	69.27
GRPO	83.88	57.34	75.36
DAPO	84.05	57.76	75.47
PGPO (Ours)	84.96	58.43	77.43

Table 4: General benchmarks evaluation on Qwen2.5-VL-3B. Performance is reported as avg@8 accuracy.

Method	Total Training Time (hours)	Overhead (%)	Avg. Performance
DAPO	25.4	-	43.84
$\text{DAPO}_{E}$	28.1	-	43.93
PGPO	28.1	+10.6%	46.00

Table 5: Analysis results of computational overhead on Qwen2.5-VL-3B with 2 NVIDIA H100 80G GPUs.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
General Training
Optimizer	AdamW
Learning Rate	1e-6
LR Schedule	Constant (no warmup or decay)
Epochs	2
Freeze Vision Tower	False

Appendix

Table 6: Key hyperparameters for training and evaluation.

Model	Geo3k	MMK12	MathVerse	DynaMath	MathVision	LogicVista	MMMU-Pro	$\text{MathVerse}_{V}$	Avg.
Qwen2.5-VL-3B	19.65	37.37	33.16	32.88	13.88	30.17	20.95	30.31	27.30
+ GRPO	32.19	62.95	58.39	47.82	25.59	40.85	28.02	54.36	43.77
+ DAPO	33.80	63.36	58.76	45.86	26.19	40.63	27.20	54.89	43.84
+ PGPO w/ GRPO	33.81	63.96	59.62	47.91	26.00	41.53	28.29	56.75	44.73

Appendix

Table 7: Results on PGPO by applying it to GRPO (avg@8 acc %).

Figure 5: Top 200 $\mathcal{S}$ tokens word cloud on all generated tokens of vision-dominant MathVerse.

Explore similar work

PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping
Jun 7, 2026Qiming Li, Tianlun Li, Xiaolong Cheng +5Token-Level Credit Assignment VLM Reasoning
Amap CV Lab, Alibaba Group · Peking University
Bridging Perception and Reasoning: Token Reweighting for RLVR in Multimodal LLMs
Mar 26, 2026Jinda Lu, Junkang Wu, Jinghan Li +6RL for VLMs VLM Reasoning
University of Science and Technology of China · Peking University · Independent Researcher
Visually-Guided Policy Optimization for Multimodal Reasoning
Apr 10, 2026Zengbin Wang, Feng Xiong, Liang Lin +5VLM Reasoning RL for VLMs
AMAP, Alibaba Group · SYSU · BUPT

Jun 7, 2026J/K move · Enter open · S save

PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping

Qiming Li, Tianlun Li, Xiaolong Cheng, Hangyu Li+4

Amap CV Lab, Alibaba Group · Peking University

Token-Level Credit Assignment VLM Reasoning Visual Grounding RL for VLMs

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) has become an effective paradigm for improving the reasoning capability of Large Vision-Language Models (LVLMs). However, existing RLVR methods primarily rely on trajectory-level outcome rewards, which assign identical learning signals across all generated tokens. This coarse-grained credit assignment is fundamentally mismatched to multimodal reasoning, where only a sparse subset of tokens is causally grounded in visual evidence. Consequently, these pivotal perceptual tokens receive weak supervision and are often overwhelmed by language priors or reasoning-template tokens. To address this limitation, we propose Perception-Reinforced Policy Optimization (PRPO), a token-level reinforcement learning framework that explicitly identifies and reinforces pivotal perceptual tokens within long-horizon multimodal reasoning trajectories. PRPO introduces Robust Visual Dependency (RVD), a principled metric that identifies tokens whose predictions are both visually grounded and perturbation-stable, filtering out brittle or noisy visual tokens. Based on RVD, we further propose Perceptual Advantage Reshaping (PAR), a token-level credit assignment technique that amplifies perceptually informative tokens while preserving stable gradients for non-perceptual tokens. Extensive experiments on seven multimodal reasoning benchmarks demonstrate that PRPO consistently outperforms strong LVLM baselines across both 3B and 7B model scales, achieving average gains of 23.3% and 21.1%, respectively. PRPO achieves state-of-the-art performance with improved training efficiency and stronger cross-task generalization. Our findings highlight the importance of fine-grained credit assignment for scalable multimodal reinforcement learning.

Abstract

Explore similar work

PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping

Bridging Perception and Reasoning: Token Reweighting for RLVR in Multimodal LLMs

Visually-Guided Policy Optimization for Multimodal Reasoning