Group Relative Policy Optimization (GRPO) has emerged as the de facto Reinforcement Learning (RL) objective driving recent advancements in Multimodal Large Language Models. However, extending this success to open-source multimodal generalist models remains heavily constrained by two primary challenges: the extreme variance in reward topologies across diverse visual tasks, and the inherent difficulty of balancing fine-grained perception with multi-step reasoning capabilities. To address these issues, we introduce Gaussian GRPO (G2RPO), a novel RL training objective that replaces standard linear scaling with non-linear distributional matching. By mathematically forcing the advantage distribution of any given task to strictly converge to a standard normal distribution, N(0,1), G2RPO theoretically ensures inter-task gradient equity, mitigates vulnerabilities to heavy-tail outliers, and offers symmetric update for positive and negative rewards. Leveraging the enhanced training stability provided by G2RPO, we introduce two task-level shaping mechanisms to seamlessly balance perception and reasoning. First, response length shaping dynamically elicits extended reasoning chains for complex queries while enforce direct outputs to bolster visual grounding. Second, entropy shaping tightly bounds the model's exploration zone, effectively preventing both entropy collapse and entropy explosion. Integrating these methodologies, we present OpenVLThinkerV2, a highly robust, general-purpose multimodal model. Extensive evaluations across 18 diverse benchmarks demonstrate its superior performance over strong open-source and leading proprietary frontier models.
Figures & tables
Figure 1: Performance improvement (relative) of OpenVLThinkerV2 over its baseline Qwen3-VL-Instruct-8B across diverse visual tasks.
Figure 3: Comparison of response length dynamics during training. G 2 RPO effectively encourage early convergence. a) It scales up reasoning length for complex question. b) It reduces overthinking for visual-centric tasks, enhancing perceptual grounding and mitigating hallucinations. c) For both reasoning and perception heavy tasks, the generation length stabilizes within an optimal range, effectively balancing both capabilities.
pi=Nrank(Ri)−0.5
(6)
Algorithm 1 G 2 RPO : Distributional Matching via 1D Optimal Transport
Figure 4: Effect of task-level entropy shaping. G 2 RPO effectively prevents entropy explosion for reasoning-centric tasks and OOD task (spatial reasoning) while concurrently mitigating entropy collapse in vision-centric tasks.
General VQA
Math VQA
Chart VQA
Model
MMMU
MMBench
MMStar
MathVista
MathVerse
MathVision
AI2D
ChartQA
CharXiv(RQ)
GPT-4o ( Hurst et al., 2024 )
70.7
84.3
65.1
63.8
41.2
30.4
84.9
86.7
47.1
Gemini 2.5 Pro ( Comanici et al., 2025 )
81.7
90.1
77.5
82.7
82.9
73.3
88.4
83.3
67.9
MM-Eureka-7B ( Meng et al., 2025a )
57.3
87.7
64.4
73.0
50.3
26.9
84.1
77.3
39.5
OpenVLThinker-7B ( Deng et al., 2025 )
55.8
87.8
59.1
70.2
47.9
25.3
81.8
85.7
39.3
VL-Rethinker-7B ( Wang et al., 2025b )
56.7
87.9
62.7
74.9
54.2
32.3
80.8
84.7
39.8
Table 1: Performance comparison of various models across visual reasoning benchmarks. We report our reproduced results for Qwen3-VL-Instruct-8B under the same setting. Our model consistently outperforms strong open-source baselines.
Table 2: Results on (a) document understanding and (b) spatial reasoning benchmarks. We compare against both general multimodal models and task-specific expert baselines.
Model
RefCOCO
RefCOCO+
RefCOCOg
Gemini 1.5 Pro ( Gemini Team, 2024 )
73.2
62.5
75.2
Grounding DINO ( Liu et al., 2023 )
90.6
88.2
86.1
VLM-R1 ( Shen et al., 2025 )
90.5
84.3
87.1
DeepEyes ( Zheng et al., 2025b )
89.8
83.6
86.7
OneThinker-8B ( Feng et al., 2025b )
92.0
87.0
89.2
Qwen3-VL-Instruct
89.9
84.5
86.8
Table 3: Evaluation results for Grounding task. We report scores on val splits. Our model consistently outperforms previous baselines.
Model
General VQA
Math VQA
Chart VQA
Grounding
Document Understanding
Spatial Reasoning
Qwen3-VL-Instruct-8B
71.3
59.2
69.9
87.1
86.8
60.9
Qwen3-VL + G 2 RPO
76.9
64.8
74.5
90.2
90.6
62.3
+ task-level entropy loss
77.0
65.1
75.3
90.4
90.8
62.8
+ task-level length reward
77.4
65.7
75.4
90.5
91.1
63.2
OpenVLThinkerV2 (Ours)
77.9
66.2
76.0
90.7
91.4
63.6
Table 4: Ablation study evaluating the impact of different training components across the six main domains. Scores represent the average performance within each task category.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Average Accuracy Reward Comparison across all tasks on the Validation set during Training. G 2 RPO demonstrates stable and superior performance overall.
Figure 6: Average Length Reward Comparison across all tasks on the Validation set during Training. G 2 RPO demonstrates stable and superior performance overall.
Figure 7: Average Format Reward Comparison across all tasks on the Validation set during Training. G 2 RPO demonstrates stable and superior performance overall.
Figure 8: Average Structure Reward Comparison across all tasks on the Validation set during Training. G 2 RPO demonstrates stable and superior performance overall.
Reinforcement learning from verifiable rewards (RLVR), especially with Group Relative Policy Optimization (GRPO), has shown strong potential for improving the reasoning capabilities of large vision-language models (LVLMs). However, in multimodal reasoning, final-answer rewards are typically assigned at the sequence level and do not distinguish the functional roles of different tokens, making it difficult to determine whether a correct answer is supported by task-relevant visual evidence. In this paper, we revisit multimodal RLVR from the perspective of role-aware token-level credit assignment, where structured responses are decomposed into perception tokens for extracting visual evidence and reasoning tokens for deriving answers from that evidence. Based on this perspective, we propose Structured Role-aware Policy Optimization (SRPO), which refines the sequence-level GRPO advantage into role-aware token-level advantages without changing the reward function. Specifically, SRPO assigns role-specific credit by using self-distilled on-policy contrasts: perception tokens are emphasized according to their visual dependency under original versus corrupted visual inputs, while reasoning tokens are emphasized according to their consistency with the generated perception. These role-specific signals are further unified through a shared trajectory-level baseline, yielding positive token weights that adjust relative update magnitudes while preserving the original GRPO reward and optimization direction, without requiring external reward models or separate teachers. Experiments across diverse multimodal reasoning benchmarks show that SRPO improves evidence-grounded reasoning, highlighting the importance of moving beyond uniform sequence-level credit toward role-aware optimization for reliable multimodal reasoning.
Bingqing Jiang, Difan Zou
School of Computing & Data Science, The University of Hong Kong · School of Computing & Data Science and Institute of Data Science, The University of Hong Kong
Reinforcement Learning with Verifiable Rewards (RLVR) has become an effective paradigm for improving the reasoning capability of Large Vision-Language Models (LVLMs). However, existing RLVR methods primarily rely on trajectory-level outcome rewards, which assign identical learning signals across all generated tokens. This coarse-grained credit assignment is fundamentally mismatched to multimodal reasoning, where only a sparse subset of tokens is causally grounded in visual evidence. Consequently, these pivotal perceptual tokens receive weak supervision and are often overwhelmed by language priors or reasoning-template tokens. To address this limitation, we propose Perception-Reinforced Policy Optimization (PRPO), a token-level reinforcement learning framework that explicitly identifies and reinforces pivotal perceptual tokens within long-horizon multimodal reasoning trajectories. PRPO introduces Robust Visual Dependency (RVD), a principled metric that identifies tokens whose predictions are both visually grounded and perturbation-stable, filtering out brittle or noisy visual tokens. Based on RVD, we further propose Perceptual Advantage Reshaping (PAR), a token-level credit assignment technique that amplifies perceptually informative tokens while preserving stable gradients for non-perceptual tokens. Extensive experiments on seven multimodal reasoning benchmarks demonstrate that PRPO consistently outperforms strong LVLM baselines across both 3B and 7B model scales, achieving average gains of 23.3% and 21.1%, respectively. PRPO achieves state-of-the-art performance with improved training efficiency and stronger cross-task generalization. Our findings highlight the importance of fine-grained credit assignment for scalable multimodal reinforcement learning.
While Reinforcement Learning from Verifiable Rewards (RLVR) has advanced reasoning in Large Vision-Language Models (LVLMs), prevailing frameworks suffer from a foundational methodological flaw: by distributing identical advantages across all generated tokens, these methods inherently dilute the learning signals essential for optimizing the critical, visually-grounded steps of multimodal reasoning. To bridge this gap, we formulate Token Visual Dependency, quantifying the causal information gain of visual inputs via the Kullback-Leibler (KL) divergence between visual-conditioned and text-only predictive distributions. Revealing that this dependency is highly sparse and semantically pivotal, we introduce Perception-Grounded Policy Optimization (PGPO), which is a novel fine-grained credit assignment framework that dynamically reshapes advantages at the token level. Through a threshold-gated, mass-conserving mechanism, PGPO actively amplifies learning signals for visually-dependent tokens while suppressing gradient noise from linguistic priors. Extensive experiments based on the Qwen2.5-VL series across seven challenging multimodal reasoning benchmarks demonstrate that PGPO boosts models by 18.7% on average. Both theoretical and empirical analyses confirm that PGPO effectively reduces gradient variance, prevents training collapse, and acts as a potent regularizer for robust, perception-grounded multimodal reasoning. Code will be released on https://github.com/Yzk1114/PGPO.
Zekai Ye, Qiming Li, Xiaocheng Feng +6
Harbin Institute of Technology · Peng Cheng Laboratory · Huawei Technologies Co., Ltd