Prism-GRPO: Faster VLA Policy Optimization via Splitting Same-outcome Groups

Zeyun Deng; Yuzhe Lu; Yawei Wang; Linbo Liu; Qing Ping; Han Ding; Guande Wu; Panpan Xu

cs.ROAug 18, 2026

Prism-GRPO: Faster VLA Policy Optimization via Splitting Same-outcome Groups

Authors: Zeyun Deng, Yuzhe Lu, Yawei Wang, Linbo Liu, Qing Ping, Han Ding, Guande Wu, Panpan Xu,

Organizations: Purdue University · AWS AI

Abstract

GRPO is increasingly used for reinforcement learning of vision-language-action (VLA) policies because, unlike PPO, it does not require training a critic. This simplification comes with a sampling cost: group-relative advantages require multiple rollouts from each scene. Under binary success rewards, groups whose rollouts all succeed or all fail have zero advantage and are discarded by dynamic sampling. These groups are especially common early in training, when most rollouts fail, wasting much of the expensive robotic rollout budget. We introduce Prism-GRPO, which augments binary outcome reward with a weighted trajectory-level execution-quality score. By splitting same-outcome groups into a quality spectrum, Prism-GRPO recovers training signal while ensuring that every success still outranks every failure. Quality scores can be derived from simulator contacts, executed actions, or visual observations, avoiding task-specific progress rewards. We prove that Prism-GRPO never increases the probability that a sampled group is discarded for having zero advantages, and derive a gradient-alignment condition under which its combined update remains a local ascent direction for task success. Across four RoboTwin tasks spanning different horizons and coordination patterns, Prism-GRPO improves success and quality at matched rollout budgets and reaches target success rates with up to 56% fewer rollouts. It also suppresses a reward-hacking shortcut, with the cleaner behavior transferring under direct deployment to a real robot. Through ablations, we show consistent gains across contact-, smoothness-, and VLM-derived quality signals.

Figures & tables

Figure 1: Illustrative overview. Binary GRPO discards all-success and all-failure groups because identical rewards yield zero advantage. Prism-GRPO splits them by execution quality to recover informative, success-aligned updates.

Figure 2: Prism-GRPO splits same-outcome groups into a reward spectrum. Colors indicate rewards. $\lambda$ =0.2.

Figure 3: Main results.

\lambda=0.2

. Stars mark each curve’s peak mean success, and arrows show the rollouts saved by Prism-GRPO when matching Binary GRPO’s peak mean success.

Figure 4: Qualitative examples. Left: Move Can Pot : RL-ZVP shove-cheat vs. clean executions by the other methods. Right: Handover Block shows worse failures cause stronger unintended contacts and larger scene disturbance.

Figure 5: Shove-cheat behavior on Move Can Pot . 5 seeds each; ticks = per-seed rate, band = min–max, dot = mean.

		Random					Prism-GRPO
Task	Metric	0.2	0.5	0.9	1	2	0.2	0.5	0.9	1	2
Lift Pot	CS	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	56	49	45	25	53
Lift Pot	Q	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	11	14	15	14	26
Move Can Pot	CS	24	33	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	41	48	41	3	$\boldsymbol{\times}$
Move Can Pot	Q	1	0	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	7	10	13	16	$\boldsymbol{\times}$

Table 1: Quality-weight

\lambda

ablation. CS is the rollout saving (%) to match Binary GRPO’s calibrated success; Q is the quality gain at that point.

\boldsymbol{\times}

denotes an unreached success.

Figure 6: Quality-source ablation. Evaluated on Lift Pot .

Figure 7: Real-world behaviors. Top: clean placement from Prism-GRPO (Ours). Bottom: shove-cheat from RL-ZVP.

Metric	Binary GRPO	RL-ZVP	Prism-GRPO
Clean Success	4/25 (16%)	2/25 (8%)	6/25 (24%)
Shove-Cheat	1/25 (4%)	5/25 (20%)	0/25 (0%)

Table 2: Real-world results. Clean Success counts trials that move the can beside the pot without using the shove-cheat.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Peak non-target contact and object displacement on Lift Pot . Stronger unintended contact is associated with larger displacement of non-target objects.

Figure 9: Collision-threshold calibration. Dashed lines mark the thresholds selected at the boundary between the low-impulse cluster and the high-impulse tail.

Figure 10: Smoothness-threshold calibration on Lift Pot .

Figure 11: Sensitivity to the quality threshold on Lift Pot . Prism-Peak shows similar performance for

T_{\mathrm{peak}}\in\{3,10\}\,\mathrm{N\,s}

, with

10\,\mathrm{N\,s}

used as the default.

Figure 12: Task overview. Four tasks under randomized object placements, clutter, lighting, and backgrounds.

Figure 13: Dynamic-filtering statistics across four tasks. Each task panel reports the percentage of zero-variance groups discarded by the trainer (left) and the total number of trajectories generated per optimizer step (right). Prism-GRPO consistently discards fewer groups and consequently remains closer to the minimum cost of

512

rollouts per step.

Figure 14: Rollout-generation cost. Per-step generation time scales nearly linearly with the number of requested rollouts across ten RL runs (

1{,}295

measurements). Each color denotes one run.

Figure 15: Full collision-quality results.

Figure 16: Full smoothness-quality results.

Figure 17: Quality-term placement ablation. Full results across success and collision-quality metrics.

Figure 18: Effect of group size. Each optimizer update uses

512

generated rollouts, while the group size varies over

G\in\{8,16,32\}

. Prism-GRPO provides the largest gain at

G=8

, while the gain decreases as the group size increases.

Figure 19: Quality-weight ablation on Lift Pot .

Figure 20: Quality-weight ablation on Move Can Pot .

Figure 21: Quality-weight decay ablation. Solid lines use a fixed quality weight, dashed lines use decay, and colors indicate different values of

\lambda

Figure 22: Effect of the advantage estimator. RLOO yields more stable learning and more consistent rollout savings than standard group-normalized GRPO.

\lambda=0.2

Figure 23: Collision-based success–quality alignment. Each task is evaluated using Max-Force, Contact-Count, and Sum-Impulse Quality. Within-scene plots compare the mean quality of successful and failed trajectories generated from the same mixed-outcome scene; positive gaps indicate that successful trajectories are cleaner. Cross-scene plots report the Spearman correlation between scene success rate and mean quality.

Figure 24: Smoothness-based success–quality alignment. Each task is evaluated using maximum joint reversals, mean joint reversals, and action jerk. The within-scene and cross-scene diagnostics follow the same definitions as Figure 23 .

Robotic RL

Robotic reinforcement learning uses trial-and-error learning to help robots control, navigate, and manipulate objects.

See all 297 papers

Temporal GRPO: Beyond Trajectory-Level Credit in Vision-Language-Action Reinforcement Learning
Aug 13, 2026Yao Zhou, Hang Gao, Fengge Wu +2Group Relative Policy Optimization Step-Level Credit Assignment in RL
Institute of Software Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Learn Where Outcomes Diverge: Efficient VLA RL via Probabilistic Chunk Masking
May 15, 2026Vaidehi Bagaria, Nikshep Grampurohit, Pulkit VermaReinforcement Learning Neural Network Optimization
Indian Institute of Technology Madras
When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning
Oct 23, 2026Zongyao Zhao, Cheng-Jie Zhao, Ruihang Jiang +3Policy Optimization Group Relative Policy Optimization
The University of Hong Kong · Tsinghua University

Aug 13, 2026J/K move · Enter open · S save

Temporal GRPO: Beyond Trajectory-Level Credit in Vision-Language-Action Reinforcement Learning

Yao Zhou, Hang Gao, Fengge Wu, Changwen Zheng+1

Institute of Software Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

Group Relative Policy Optimization Step-Level Credit Assignment in RL Temporal Credit Assignment RL Post-Training Policy Learning RL for VLMs

Open paper AskPDF arXiv

Outcome-driven reinforcement learning offers a scalable way to post-train vision-language-action (VLA) policies from sparse task-success feedback. In common GRPO-based VLA post-training, one rollout-level advantage is applied to every action in the trajectory. A rollout that completes several valid stages but fails later can therefore penalize the actions that produced its earlier progress. We call this trajectory-level credit aliasing. Temporal GRPO addresses this problem by constructing detectable task stages, aligning each rollout with stage-specific action intervals, and comparing only rollouts that have entered the same stage. The resulting stage advantages are applied to their corresponding intervals in a single policy update. On RoboTwin 2.0, Temporal GRPO improves task success and sample efficiency, with consistent gains across task horizons. Controlled updates on LIBERO-Long preserve shared prerequisite stages and concentrate improvement at the first stage where rollout outcomes diverge.

Abstract

Robotic RL

Explore similar work

Temporal GRPO: Beyond Trajectory-Level Credit in Vision-Language-Action Reinforcement Learning

Learn Where Outcomes Diverge: Efficient VLA RL via Probabilistic Chunk Masking

When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning