Reinforcement learning with verifiable rewards (RLVR) improves the reasoning ability of vision-language models (VLMs), and diversifying the rollouts within each optimization group amplifies its gains. Existing approaches diversify through decoding temperature or pixel-space image distortion; we ask whether the perturbation belongs in the model's latent space instead. We introduce Noise-Contrastive GRPO (NC-GRPO), which injects scale-calibrated Gaussian noise into the last hidden layer of the prompt-encoding pass for half of each rollout group, branching those rollouts from a displaced departure state. Branches that reach the answer despite the displacement are reinforced over those derailed by it, converting sensitivity at the branch point into policy-gradient signal; the objective, reward, and inference protocol are untouched. On Qwen2.5-VL-7B trained on Geometry3K, NC-GRPO significantly improves out-of-domain mathematical reasoning over vanilla GRPO across five held-out benchmarks (pooled McNemar p≤0.001) while also improving in-domain accuracy and hallucination robustness -- the latter an axis on which image-space noise regresses even while posting a larger OOD average on perception-heavy benchmarks. Mechanism ablations indicate that independent stochastic diversity, not noise budget or direction, is the active ingredient, and a noise-scale study exposes a dial between reasoning specialization and general capability. NC-GRPO is designed to be modality-agnostic and integrates into a standard RLVR pipeline as a ~50-line change to the inference engine.
Figures & tables
Figure 1: Training dynamics (Qwen2.5-VL-7B, Geometry3K, 60 steps; checkpoints at 0/20/40/60). Training reward (top left) matches vanilla step-for-step (mean difference −0.003±0.005 ; ± is std across steps, not seed spread) while held-out accuracy diverges: by step 60 NC-GRPO leads on four of five OOD benchmarks, in-domain accuracy, and POPE. Gains appear OOD, not in training reward — convergence to a different, more robust solution ( Section 3.3 ).
Method
Geo3K (ID)
Hallusion
MathVista
MathVerse
MathVision
WeMath
OOD avg
Vanilla GRPO
52.9
70.2
70.7
43.0
27.1
49.1
52.0
NC-GRPO σ0=0.2
53.9
71.3
70.1
44.9 ∗
28.4
50.5
53.0 ∗
NC-GRPO σ0=0.5
53.7
71.0
71.2
45.3 ∗
27.6
52.1 ∗
53.5 ∗
Table 1: Main result: reasoning benchmarks. Accuracy (%) vs. vanilla GRPO (Qwen2.5-VL-7B, Geometry3K, greedy eval, rule-based scoring). ∗ : exact McNemar p<0.05 vs. vanilla; pooled OOD p=0.001 ( σ0=0.2 ), p<10−4 ( σ0=0.5 ). NC-GRPO is the only intervention in our study improving in-domain, hallucination, and OOD simultaneously; the OOD and in-domain components reproduce on a second seed, hallucination is single-seed ( Section 5 ).
POPE
MMStar
AI2D
RealWorldQA
ChartQA
Pooled vs. vanilla
Vanilla GRPO
86.8
62.9
81.1
65.9
80.3
—
NoisyRollout (image)
86.5
63.9
81.0
65.6
80.6
p=0.67
NC-GRPO σ0=0.2
87.0 ‡
62.6
80.8
66.3
79.8
p=0.89
NC-GRPO σ0=0.5
85.9
63.3
79.7
66.3
80.4
p=0.0007 ( − )
Table 2: General-capability battery (accuracy %; identical rule-based scoring). ‡ NC-GRPO σ0=0.2 beats image noise on POPE ( p=0.007 ). At σ0=0.2 NC-GRPO is indistinguishable from vanilla pooled over the battery ( p=0.89 ) — reasoning gains at no measurable general-capability cost — while σ0=0.5 trades a significant pooled regression ( p=0.0007 ) for larger reasoning gains: noise scale is a specialization dial.
Geo3K (ID)
Hallusion
MathVista
MathVerse
MathVision
WeMath
OOD avg
NoisyRollout (image)
52.1
70.7
71.5
46.5 ∗
27.8
64.3 ∗
56.2 ∗
Δ vs. vanilla
−0.8
+0.5
+0.8
+3.5
+0.7
+15.2
+4.2
Table 3: Image-space noise on the reasoning suite (accuracy %; same protocol and scorer as Table 1 ; vanilla row there). ∗ : exact McNemar p<0.05 vs. vanilla (MathVerse p<10−7 , WeMath p<10−32 ; pooled OOD p<10−4 ). NoisyRollout is pixel-space augmentation rather than a latent intervention, so we compare it here: its OOD margin is carried almost entirely by WeMath and MathVerse, and it does not improve in-domain.
Variant
Geo3K (ID)
WeMath
OOD avg
Vanilla GRPO
52.9
49.1
52.0
NC-GRPO (indep. noise)
53.9
50.5
53.0
w/ antithetic pairs
52.9
40.5
50.1 †
w/ SPSA steering
50.1
37.9
50.1
Table 4: Variance reduction hurts. Antithetic pairing (exact (+ϵ,−ϵ) pairs, identical budget, σ0=0.2 ) is significantly worse than both independent noise ( p<10−4 ) and vanilla ( † p=0.006 , pooled OOD McNemar). SPSA steering ( Eq. 8 ) on top does not recover the loss ( p=0.38 vs. antithetic) despite converging to a stable direction ( ∥μ∥ grows 30× over training). The antithetic and SPSA OOD averages coincide at this precision.
Model
Method
Geo3K (ID)
HalluBench
MathVista
MathVerse
MathVision
WeMath
OOD avg
Qwen2.5-VL-3B
Vanilla GRPO
42.8
59.6
61.6
36.3
23.0
44.5
45.0
NC-GRPO (ours)
42.9
59.7
60.0
34.2
24.3
47.0
45.0
Qwen2.5-VL-7B
Vanilla GRPO
52.9
70.2
70.7
43.0
27.1
49.1
52.0
NC-GRPO (ours)
53.9
71.3
70.1
44.9
28.4
50.5
53.0 ∗
Qwen3-VL-8B-Thinking
Vanilla GRPO
63.1
73.9
74.4
50.0
30.4
72.0
60.1
NC-GRPO (ours)
63.2
74.3
73.9
51.3
33.1
74.4
61.4 ∗
Table 5: NC-GRPO across model scales and generations within the Qwen line (Geometry3K training, identical protocol per model; greedy decoding). ∗ pooled per-question McNemar over n=10,038 OOD questions significant in favor of NC-GRPO ( p=0.001 for Qwen2.5-VL-7B; p<10−4 for Qwen3-VL-8B-Thinking). Qwen2.5-VL-3B shows no OOD benefit and is analyzed in Table 6 .
Benchmark
Vanilla
NC-GRPO
p
Geo3K (ID)
42.8
42.9
1.00
HalluBench
59.6
59.7
0.60
MathVista
61.6
60.0
0.32
MathVerse
36.3
34.2
0.014 ∗
MathVision
23.0
24.3
0.10
WeMath
44.5
47.0
0.023 ∗
Table 6: Scale ablation: Qwen2.5-VL-3B (NC-GRPO at σ0=0.2 transferred from 7B per Proposition 1 , which guarantees scale calibration across widths, not optimality at 3B). The 7B OOD benefit is absent: pooled McNemar over n=10,038 paired questions is indistinguishable ( p=0.81 ), in-domain unchanged, and two benchmarks move significantly in opposite directions and cancel (WeMath +2.5 , MathVerse −2.1 ) — harmless but not beneficial below a capacity threshold. ∗ uncorrected.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: The NC-GRPO workflow. Each prompt is prefilled twice: once clean (solid, gray) and once with scale-calibrated Gaussian noise added to the hidden states returned by the last layer (dashed, blue). The perturbation shifts the logits at the final prompt position, so each noisy branch departs from a displaced state and then decodes from a clean cache with no decode-time noise ( Section 3.1 ); rewards are normalized over the joint 2n -rollout group, so systematic disagreement between the clean and noisy halves — the contrast term of Proposition 2 — becomes learning signal. The noise scale σk anneals to zero over training ( Eq. 4 ), and the policy-gradient forward pass sees only unperturbed states.
Figure 3: NC-GRPO vs. vanilla GRPO at both noise scales ( σ0∈{0.2,0.5} ): accuracy gains in percentage points, with vanilla’s absolute accuracy beneath each benchmark; visualization of the main-results table in the main paper. Shaded column: OOD average over the five held-out benchmarks. ∗ : exact McNemar p<0.05 on paired per-question outcomes.
Figure 4: In-domain dynamics of all three methods (Geometry3K): training reward (top) and validation accuracy (bottom) for vanilla GRPO, NoisyRollout (image noise), and NC-GRPO (latent noise). The three training-reward curves are nearly indistinguishable and in-domain validation separates only marginally — the methods differ OOD ( Tables 1 and 3 ), not in-domain. Referenced from the image-comparison subsection of the main paper.
Recent studies have demonstrated that Reinforcement Learning (RL), notably Group Relative Policy Optimization (GRPO), can intrinsically elicit and enhance the reasoning capabilities of Vision-Language Models (VLMs). However, despite the promise, the underlying mechanisms that drive the effectiveness of RL models as well as their limitations remain underexplored. In this paper, we highlight a fundamental behavioral distinction between RL and base models, where the former engages in deeper yet narrow reasoning, while base models, despite less refined along individual path, exhibit broader and more diverse thinking patterns. Through further analysis of training dynamics, we show that GRPO is prone to diversity collapse, causing models to prematurely converge to a limited subset of reasoning strategies while discarding the majority of potential alternatives, leading to local optima and poor scalability. To address this, we propose Multi-Group Policy Optimization (MUPO), a simple yet effective approach designed to incentivize divergent thinking across multiple solutions, and demonstrate its effectiveness on established benchmarks. Project page: https://xytian1008.github.io/MUPO/
Xinyu Tian, Shu Zou, Zhaoyuan Yang +3
Australian National University · Shanghai AI Lab · GE Research
Vision-Language (VL) reasoning requires a model to both extract relevant and accurate information from an image (visual reasoning, VR), and to infer the answer from it (language reasoning, LR). Reinforcement learning with verifiable rewards typically trains both through a single chain-of-thought with a final-answer reward. This gives every CoT token the same sequence-level advantage, failing to distinguish capability specific errors. We propose SPLIT-RL, a staged post-training approach that trains VR and LR in disjoint phases. Because a group's rollouts differ along one capability at a time, the group-relative advantage isolates it, and each phase is optimized using phase-specific reward. We further introduce Claim-Level Advantage (CLA-GRPO), which decomposes VR-phase rollouts into atomic visual claims and provides a fine-grained advantage at claim level based on visual-type group formation. Although trained in two phases, trained policy is evaluated like GRPO model, with a single CoT call at inference time. Under this protocol, SPLIT-RL improves average accuracy over GRPO by 1.4-6.1 points across Qwen3-VL models from 2B to 30B-A3B and InternVL3.5-8B. Evaluating each capability using an oracle based diagnostic shows that answer-only GRPO leaves perception unchanged, whereas SPLIT-RL improves both VR and LR.
Reinforcement learning with verifiable rewards (RLVR) has emerged as a highly effective framework for improving LLM reasoning, with methods such as GRPO among its most successful instantiations. However, GRPO relies on repeated generation of long chain-of-thought rollouts. Training time scales with the number of rollouts, a large fraction of which are uninformative. Thus, GRPO is computationally expensive and unstable. To mitigate this, existing approaches either generate a larger pool of rollouts and filter the most informative prompts, or leverage historical signals for filtering at later stages of training. These strategies offer modest performance gains, but slow down the overall process. To address this, we propose VarIance Guided Online Rollout allocation (VIGOR) which instead of allocating a fixed rollout budget per example, begins with a small number of rollouts for all examples in a batch and iteratively allocates additional rollouts to those with the highest group reward variance until a fixed total rollout budget is reached. Theoretically, we show that under RLVR, reward variance controls the gradient magnitude, and derive VIGOR's closed-form speedup ratio over GRPO, which grows with refinement rounds under Pareto-distributed reward variance. Experiments on mathematical reasoning and coding tasks show that VIGOR reaches target accuracy with up to 2.3× fewer rollouts on math, reaches GRPO's final coding full pass rate with 1.49× fewer rollouts, and improves the coding average test pass rate by 3.4 points.