Bridging Perception and Reasoning: Token Reweighting for RLVR in Multimodal LLMs

Jinda Lu; Junkang Wu; Jinghan Li; Kexin Huang; Shuo Yang; Guoyin Wang; Jiancan Wu; Xiang Wang

cs.CVMar 26, 2026

Bridging Perception and Reasoning: Token Reweighting for RLVR in Multimodal LLMs

Authors: Jinda Lu, Junkang Wu, Jinghan Li, Kexin Huang, Shuo Yang, Guoyin Wang, Jiancan Wu, Xiang Wang, +1 more

Organizations: University of Science and Technology of China · Peking University · Independent Researcher

Abstract

Extending Reinforcement Learning with Verifiable Rewards (RLVR) to multimodal large language models (MLLMs) faces a fundamental challenge: their responses inherently interleave perception-related tokens, which ground visual content, with reasoning-related tokens, which construct reasoning chains. These token types instantiate distinct yet interdependent capacities -- visual grounding and symbolic reasoning -- making isolated optimization insufficient. Through token-level empirical analysis, we demonstrate that optimizing either perception- or reasoning-only tokens consistently underperforms full optimization, underscoring their inherent coupling. To address this, we propose a plug-and-play Token-Reweighting (ToR) strategy that explicitly models this interdependence by identifying critical tokens of both types and dynamically reweighting them during RLVR training. Applied on top of existing methods (e.g., GRPO and DAPO), ToR delivers consistent performance gains across multiple multi-modal reasoning benchmarks, achieving state-of-the-art performance with both accurate visual grounding and coherent reasoning.

Figures & tables

Figure 1: MLLM responses typically involve two types of critical tokens: (1) reasoning-related tokens to construct reasoning chains, and (2) perception-related tokens to ground visual content.

Figure 2: Performance comparison over the wemath benchmark ( Qiao et al., 2024 ) when optimizing different token types with GRPO ( Shao et al., 2024 ) . Results across selection ratios 20%, 30%, or 50% show that optimizing either reasoning-only or perception-only tokens underperforms all tokens. Qualitative examples are selected from the best-performing checkpoints.

Figure 3: Comparison of perception strength and reasoning uncertainty under different token ratios during GRPO optimization. (a) Reasoning-token-only optimization suffers from limited perception capacity. (b) Perception-token-only optimization is sensitive to high reasoning uncertainty. (c) Token reweighting yields a balanced regime by adaptively trading off perception strength and reasoning uncertainty, where the shaded region indicates effective optimization outcomes. Dashed lines denote the balance where increased perception strength compensates for reasoning uncertainty.

$\alpha_{r}$	HalluBench	MathVerse	WeMath	MathVision	MathVista
20%	62.8	49.3	63.9	26.5	69.2
30%	65.1	50.7	65.6	27.0	70.0
40%	66.1	50.1	65.9	26.3	69.4
50%	67.1	50.4	66.0	27.0	68.7
60%	67.1	50.0	67.1	27.3	70.3
70%	67.5	50.8	67.3	26.7	70.0

Table 1: Performance comparison with different ratios of reasoning-related tokens (

\alpha_{r}

). We vary the proportion of reasoning-related tokens from 20% to 80%.

$\alpha_{p}$	HalluBench	MathVerse	WeMath	MathVision	MathVista
20%	65.9	38.6	61.4	22.7	65.5
30%	69.3	44.9	64.8	24.5	67.8
40%	67.8	44.7	65.9	25.1	67.4
50%	68.3	39.5	62.9	27.6	68.7
60%	68.1	45.1	65.8	25.0	67.4
70%	65.1	45.8	66.8	25.2	68.9

Table 2: Performance comparison with different ratios of perception-related tokens (

\alpha_{p}

). We vary the proportion of perception-related tokens from 20% to 80%.

$\gamma_{p}$	HalluBench	MathVerse	WeMath	MathVision	MathVista
0.1	66.5	50.8	65.5	27.3	69.7
0.3	69.1	52.1	67.4	28.6	70.8
0.5	71.3	53.0	68.7	28.3	71.9
0.7	70.5	51.1	68.1	27.5	70.8
0.9	72.2	50.9	67.0	27.2	71.4
1.1	71.3	49.9	67.3	26.9	69.3

Table 3: Performance under different combinations of reasoning and perception tokens. We vary the perception-related weight from 0.1 to 1.5 while keeping the reasoning-related weight fixed at 1.0.

Open-source Models
Model	Data Size	MathVerse	MathVision	MathVista	WeMath	HalluBench
InternVL-2.5-8B ( Chen et al., 2024 )	-	39.5	19.7	64.4	-	67.3
InternVL-3-8B ( Zhu et al., 2025 )	-	39.5	29.3	71.6	37.1	-
LLaVA-OneVision-7B ( Li et al., 2025a )	-	26.2	-	63.2	-	48.4
Qwen2.5-VL-7B-Instruct ( Bai et al., 2025 )	-	46.2	25.0	67.5	63.1	64.6
RLVR on Qwen-2.5-VL-7B

Table 4: Performance comparison of multi-modal LLMs under different model and data scales. Following existing works ( Liu et al., 2025a ; Xiao et al., 2025 ) , we highlight the data size with blue and red for SFT and RL , respectively. The best value in each column is shown in bold , and the second-best is underlined .

Figure 4: Comparison of optimization behaviors under different token selection strategies. Vanilla GRPO optimizes uniformly across all tokens. Reasoning-only and perception-only optimization concentrate on a single token type, leading to imbalanced training. Token reweighting jointly emphasizes reasoning- and perception-related tokens, achieving a more balanced optimization.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Perception Proxy	HallusionBench	WeMath	MathVerse	Avg. Gain
Baseline (GRPO)	69.8	67.4	50.8	–
Prob Diff (best visual grounding)	71.8 (+2.0)	68.5 (+1.1)	52.6 (+1.8)	+1.6%
Entropy Diff (best info-theoretic)	70.9 (+1.1)	67.9 (+0.5)	51.5 (+0.7)	+0.8%
Logp Diff (best balance)	72.4 (+2.6)	68.9 (+1.5)	53.0 (+2.2)	+2.1%

Appendix

Table 5: Comparison of different perception proxies within the ToR framework. Performance is reported on HallusionBench, WeMath, and MathVerse. Gains are computed relative to the GRPO baseline.

Figure 11: Correlation heatmaps between different image token selection strategies over the validation set.

Figure 12: Correlation heatmaps between different image token selection strategies over the hallubench benchmark.

Figure 13: Correlation heatmaps between different image token selection strategies over the wemath benchmark.

Figure 14: Correlation comparison across layers between different image token selection strategies over the validation set.

Figure 15: Correlation comparison across layers between different image token selection strategies over the hallubench benchmark.

Figure 16: Correlation comparison across layers between different image token selection strategies over the wemath benchmark.

Figure 17: violin plot between different image token selection strategies over the validation set.

Figure 18: violin plot between different image token selection strategies over the Hallubench benchmark.

Figure 19: violin plot between different image token selection strategies over the wemath benchmark.

Figure 20: Distribution of selected tokens over the sampled rollouts, where we employ the Qwen-VL-2.5 7B model with a batch of 512 samples, each sample generates 16 rollout responses.

Explore similar work

PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping
Jun 7, 2026Qiming Li, Tianlun Li, Xiaolong Cheng +5Token-Level Credit Assignment VLM Reasoning
Amap CV Lab, Alibaba Group · Peking University
Reinforcing Multimodal Reasoning via Token-Level Perception-Grounded Advantage Estimation
Sep 30, 2026Zhihan Zhang, Lizi LiaoToken-Level Credit Assignment Multimodal Large Language Models
Singapore Management University Singapore, Singapore
Beyond Where to Look: Trajectory-Guided Reinforcement Learning for Multimodal RLVR
Mar 27, 2026Jinda Lu, Junkang Wu, Jinghan Li +6RL for VLMs Multimodal Reasoning
University of Science and Technology of China · Pecking University · Institute of Software, Chinese Academy of Sciences

Jun 7, 2026J/K move · Enter open · S save

PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping

Qiming Li, Tianlun Li, Xiaolong Cheng, Hangyu Li+4

Amap CV Lab, Alibaba Group · Peking University

Token-Level Credit Assignment VLM Reasoning Visual Grounding RL for VLMs

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) has become an effective paradigm for improving the reasoning capability of Large Vision-Language Models (LVLMs). However, existing RLVR methods primarily rely on trajectory-level outcome rewards, which assign identical learning signals across all generated tokens. This coarse-grained credit assignment is fundamentally mismatched to multimodal reasoning, where only a sparse subset of tokens is causally grounded in visual evidence. Consequently, these pivotal perceptual tokens receive weak supervision and are often overwhelmed by language priors or reasoning-template tokens. To address this limitation, we propose Perception-Reinforced Policy Optimization (PRPO), a token-level reinforcement learning framework that explicitly identifies and reinforces pivotal perceptual tokens within long-horizon multimodal reasoning trajectories. PRPO introduces Robust Visual Dependency (RVD), a principled metric that identifies tokens whose predictions are both visually grounded and perturbation-stable, filtering out brittle or noisy visual tokens. Based on RVD, we further propose Perceptual Advantage Reshaping (PAR), a token-level credit assignment technique that amplifies perceptually informative tokens while preserving stable gradients for non-perceptual tokens. Extensive experiments on seven multimodal reasoning benchmarks demonstrate that PRPO consistently outperforms strong LVLM baselines across both 3B and 7B model scales, achieving average gains of 23.3% and 21.1%, respectively. PRPO achieves state-of-the-art performance with improved training efficiency and stronger cross-task generalization. Our findings highlight the importance of fine-grained credit assignment for scalable multimodal reinforcement learning.

Abstract

Explore similar work

PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping

Reinforcing Multimodal Reasoning via Token-Level Perception-Grounded Advantage Estimation

Beyond Where to Look: Trajectory-Guided Reinforcement Learning for Multimodal RLVR