Perturb the Thought, Not the Pixels: Latent-Space Rollout Diversification for Reinforcement Learning of Vision-Language Models

Michael Jerge; Joseph Pelczar; Justin Downes

cs.CVAug 21, 2026

Perturb the Thought, Not the Pixels: Latent-Space Rollout Diversification for Reinforcement Learning of Vision-Language Models

Authors: Michael Jerge, Joseph Pelczar, Justin Downes

Organizations: Amazon Web Services

Abstract

Reinforcement learning with verifiable rewards (RLVR) improves the reasoning ability of vision-language models (VLMs), and diversifying the rollouts within each optimization group amplifies its gains. Existing approaches diversify through decoding temperature or pixel-space image distortion; we ask whether the perturbation belongs in the model's latent space instead. We introduce Noise-Contrastive GRPO (NC-GRPO), which injects scale-calibrated Gaussian noise into the last hidden layer of the prompt-encoding pass for half of each rollout group, branching those rollouts from a displaced departure state. Branches that reach the answer despite the displacement are reinforced over those derailed by it, converting sensitivity at the branch point into policy-gradient signal; the objective, reward, and inference protocol are untouched. On Qwen2.5-VL-7B trained on Geometry3K, NC-GRPO significantly improves out-of-domain mathematical reasoning over vanilla GRPO across five held-out benchmarks (pooled McNemar $p \le 0.001$ ) while also improving in-domain accuracy and hallucination robustness -- the latter an axis on which image-space noise regresses even while posting a larger OOD average on perception-heavy benchmarks. Mechanism ablations indicate that independent stochastic diversity, not noise budget or direction, is the active ingredient, and a noise-scale study exposes a dial between reasoning specialization and general capability. NC-GRPO is designed to be modality-agnostic and integrates into a standard RLVR pipeline as a ~50-line change to the inference engine.

Figures & tables

Figure 1: Training dynamics (Qwen2.5-VL-7B, Geometry3K, 60 steps; checkpoints at 0/20/40/60). Training reward (top left) matches vanilla step-for-step (mean difference $-0.003\pm 0.005$ ; $\pm$ is std across steps, not seed spread) while held-out accuracy diverges: by step 60 NC-GRPO leads on four of five OOD benchmarks, in-domain accuracy, and POPE. Gains appear OOD, not in training reward — convergence to a different, more robust solution ( Section 3.3 ).

Method	Geo3K (ID)	Hallusion	MathVista	MathVerse	MathVision	WeMath	OOD avg
Vanilla GRPO	52.9	70.2	70.7	43.0	27.1	49.1	52.0
NC-GRPO $\sigma_{0}{=}0.2$	53.9	71.3	70.1	44.9 ∗	28.4	50.5	53.0 ∗
NC-GRPO $\sigma_{0}{=}0.5$	53.7	71.0	71.2	45.3 ∗	27.6	52.1 ∗	53.5 ∗

Table 1: Main result: reasoning benchmarks. Accuracy (%) vs. vanilla GRPO (Qwen2.5-VL-7B, Geometry3K, greedy eval, rule-based scoring). ∗ : exact McNemar

p{<}0.05

vs. vanilla; pooled OOD

p{=}0.001

(

\sigma_{0}{=}0.2

p{<}10^{-4}

(

\sigma_{0}{=}0.5

). NC-GRPO is the only intervention in our study improving in-domain, hallucination, and OOD simultaneously; the OOD and in-domain components reproduce on a second seed, hallucination is single-seed ( Section 5 ).

	POPE	MMStar	AI2D	RealWorldQA	ChartQA	Pooled vs. vanilla
Vanilla GRPO	86.8	62.9	81.1	65.9	80.3	—
NoisyRollout (image)	86.5	63.9	81.0	65.6	80.6	$p{=}0.67$
NC-GRPO $\sigma_{0}{=}0.2$	87.0 ‡	62.6	80.8	66.3	79.8	$p{=}0.89$
NC-GRPO $\sigma_{0}{=}0.5$	85.9	63.3	79.7	66.3	80.4	$p{=}0.0007$ ( $-$ )

Table 2: General-capability battery (accuracy %; identical rule-based scoring). ‡ NC-GRPO

\sigma_{0}{=}0.2

beats image noise on POPE (

p{=}0.007

). At

\sigma_{0}{=}0.2

NC-GRPO is indistinguishable from vanilla pooled over the battery (

p{=}0.89

) — reasoning gains at no measurable general-capability cost — while

\sigma_{0}{=}0.5

trades a significant pooled regression (

p{=}0.0007

) for larger reasoning gains: noise scale is a specialization dial.

	Geo3K (ID)	Hallusion	MathVista	MathVerse	MathVision	WeMath	OOD avg
NoisyRollout (image)	52.1	70.7	71.5	46.5 ∗	27.8	64.3 ∗	56.2 ∗
$\Delta$ vs. vanilla	$-0.8$	$+0.5$	$+0.8$	$+3.5$	$+0.7$	$+15.2$	$+4.2$

Table 3: Image-space noise on the reasoning suite (accuracy %; same protocol and scorer as Table 1 ; vanilla row there). ∗ : exact McNemar

p{<}0.05

vs. vanilla (MathVerse

p{<}10^{-7}

, WeMath

p{<}10^{-32}

; pooled OOD

p{<}10^{-4}

). NoisyRollout is pixel-space augmentation rather than a latent intervention, so we compare it here: its OOD margin is carried almost entirely by WeMath and MathVerse, and it does not improve in-domain.

Variant	Geo3K (ID)	WeMath	OOD avg
Vanilla GRPO	52.9	49.1	52.0
NC-GRPO (indep. noise)	53.9	50.5	53.0
w/ antithetic pairs	52.9	40.5	50.1 †
w/ SPSA steering	50.1	37.9	50.1

Table 4: Variance reduction hurts. Antithetic pairing (exact

(+\epsilon,-\epsilon)

pairs, identical budget,

\sigma_{0}{=}0.2

) is significantly worse than both independent noise (

p{<}10^{-4}

) and vanilla ( †

p{=}0.006

, pooled OOD McNemar). SPSA steering ( Eq. 8 ) on top does not recover the loss (

p{=}0.38

vs. antithetic) despite converging to a stable direction (

\lVert\mu\rVert

grows

30\times

over training). The antithetic and SPSA OOD averages coincide at this precision.

Model	Method	Geo3K (ID)	HalluBench	MathVista	MathVerse	MathVision	WeMath	OOD avg
Qwen2.5-VL-3B	Vanilla GRPO	42.8	59.6	61.6	36.3	23.0	44.5	45.0
Qwen2.5-VL-3B	NC-GRPO (ours)	42.9	59.7	60.0	34.2	24.3	47.0	45.0
Qwen2.5-VL-7B	Vanilla GRPO	52.9	70.2	70.7	43.0	27.1	49.1	52.0
Qwen2.5-VL-7B	NC-GRPO (ours)	53.9	71.3	70.1	44.9	28.4	50.5	53.0 ∗
Qwen3-VL-8B-Thinking	Vanilla GRPO	63.1	73.9	74.4	50.0	30.4	72.0	60.1
Qwen3-VL-8B-Thinking	NC-GRPO (ours)	63.2	74.3	73.9	51.3	33.1	74.4	61.4 ∗

Table 5: NC-GRPO across model scales and generations within the Qwen line (Geometry3K training, identical protocol per model; greedy decoding). ∗ pooled per-question McNemar over

n{=}10{,}038

OOD questions significant in favor of NC-GRPO (

p{=}0.001

for Qwen2.5-VL-7B;

p{<}10^{-4}

for Qwen3-VL-8B-Thinking). Qwen2.5-VL-3B shows no OOD benefit and is analyzed in Table 6 .

Benchmark	Vanilla	NC-GRPO	$p$
Geo3K (ID)	42.8	42.9	1.00
HalluBench	59.6	59.7	0.60
MathVista	61.6	60.0	0.32
MathVerse	36.3	34.2	0.014 ∗
MathVision	23.0	24.3	0.10
WeMath	44.5	47.0	0.023 ∗

Table 6: Scale ablation: Qwen2.5-VL-3B (NC-GRPO at

\sigma_{0}{=}0.2

transferred from 7B per Proposition 1 , which guarantees scale calibration across widths, not optimality at 3B). The 7B OOD benefit is absent: pooled McNemar over

n{=}10{,}038

paired questions is indistinguishable (

p{=}0.81

), in-domain unchanged, and two benchmarks move significantly in opposite directions and cancel (WeMath

+2.5

, MathVerse

-2.1

) — harmless but not beneficial below a capacity threshold. ∗ uncorrected.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: The NC-GRPO workflow. Each prompt is prefilled twice: once clean (solid, gray) and once with scale-calibrated Gaussian noise added to the hidden states returned by the last layer (dashed, blue). The perturbation shifts the logits at the final prompt position, so each noisy branch departs from a displaced state and then decodes from a clean cache with no decode-time noise ( Section 3.1 ); rewards are normalized over the joint $2n$ -rollout group, so systematic disagreement between the clean and noisy halves — the contrast term of Proposition 2 — becomes learning signal. The noise scale $\sigma_{k}$ anneals to zero over training ( Eq. 4 ), and the policy-gradient forward pass sees only unperturbed states.

Figure 3: NC-GRPO vs. vanilla GRPO at both noise scales ( $\sigma_{0}\in\{0.2,0.5\}$ ): accuracy gains in percentage points, with vanilla’s absolute accuracy beneath each benchmark; visualization of the main-results table in the main paper. Shaded column: OOD average over the five held-out benchmarks. ∗ : exact McNemar $p{<}0.05$ on paired per-question outcomes.

Figure 4: In-domain dynamics of all three methods (Geometry3K): training reward (top) and validation accuracy (bottom) for vanilla GRPO, NoisyRollout (image noise), and NC-GRPO (latent noise). The three training-reward curves are nearly indistinguishable and in-domain validation separates only marginally — the methods differ OOD ( Tables 1 and 3 ), not in-domain. Referenced from the image-comparison subsection of the main paper.

RL for VLMs

Reinforcement learning for vision-language models aligns multimodal reasoning, tool use, and action with task-specific feedback.

See all 472 papers

All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Models
Apr 1, 2026Xinyu Tian, Shu Zou, Zhaoyuan Yang +3RL for VLMs VLM Reasoning
Australian National University · Shanghai AI Lab · GE Research
SPLIT-RL: Staged Perception-Language Reasoning Training with Claim-Level Advantages
Oct 7, 2026Raja Kumar, Rajat Koner, Ritwick Chaudhry +3VLM Reasoning RL for VLMs
University of Southern California · Amazon AGI
Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning
Jul 24, 2026Heyang Jiang, Henry Liu, Baharan MirzasoleimanRL for Code Generation RL for Language Model Reasoning
University of California, Los Angeles

Apr 1, 2026J/K move · Enter open · S save

All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Models

Xinyu Tian, Shu Zou, Zhaoyuan Yang, Mengqi He+2

Australian National University · Shanghai AI Lab · GE Research

RL for VLMs VLM Reasoning

Open paper AskPDF arXiv

Recent studies have demonstrated that Reinforcement Learning (RL), notably Group Relative Policy Optimization (GRPO), can intrinsically elicit and enhance the reasoning capabilities of Vision-Language Models (VLMs). However, despite the promise, the underlying mechanisms that drive the effectiveness of RL models as well as their limitations remain underexplored. In this paper, we highlight a fundamental behavioral distinction between RL and base models, where the former engages in deeper yet narrow reasoning, while base models, despite less refined along individual path, exhibit broader and more diverse thinking patterns. Through further analysis of training dynamics, we show that GRPO is prone to diversity collapse, causing models to prematurely converge to a limited subset of reasoning strategies while discarding the majority of potential alternatives, leading to local optima and poor scalability. To address this, we propose Multi-Group Policy Optimization (MUPO), a simple yet effective approach designed to incentivize divergent thinking across multiple solutions, and demonstrate its effectiveness on established benchmarks. Project page: https://xytian1008.github.io/MUPO/

Abstract

RL for VLMs

Explore similar work

All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Models

SPLIT-RL: Staged Perception-Language Reasoning Training with Claim-Level Advantages

Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning