GRPO is increasingly used for reinforcement learning of vision-language-action (VLA) policies because, unlike PPO, it does not require training a critic. This simplification comes with a sampling cost: group-relative advantages require multiple rollouts from each scene. Under binary success rewards, groups whose rollouts all succeed or all fail have zero advantage and are discarded by dynamic sampling. These groups are especially common early in training, when most rollouts fail, wasting much of the expensive robotic rollout budget. We introduce Prism-GRPO, which augments binary outcome reward with a weighted trajectory-level execution-quality score. By splitting same-outcome groups into a quality spectrum, Prism-GRPO recovers training signal while ensuring that every success still outranks every failure. Quality scores can be derived from simulator contacts, executed actions, or visual observations, avoiding task-specific progress rewards. We prove that Prism-GRPO never increases the probability that a sampled group is discarded for having zero advantages, and derive a gradient-alignment condition under which its combined update remains a local ascent direction for task success. Across four RoboTwin tasks spanning different horizons and coordination patterns, Prism-GRPO improves success and quality at matched rollout budgets and reaches target success rates with up to 56% fewer rollouts. It also suppresses a reward-hacking shortcut, with the cleaner behavior transferring under direct deployment to a real robot. Through ablations, we show consistent gains across contact-, smoothness-, and VLM-derived quality signals.
Figures & tables
Figure 1: Illustrative overview. Binary GRPO discards all-success and all-failure groups because identical rewards yield zero advantage. Prism-GRPO splits them by execution quality to recover informative, success-aligned updates.
Figure 2: Prism-GRPO splits same-outcome groups into a reward spectrum. Colors indicate rewards. λ =0.2.
Figure 3: Main results. λ=0.2 . Stars mark each curve’s peak mean success, and arrows show the rollouts saved by Prism-GRPO when matching Binary GRPO’s peak mean success.
Figure 4: Qualitative examples. Left: Move Can Pot : RL-ZVP shove-cheat vs. clean executions by the other methods. Right: Handover Block shows worse failures cause stronger unintended contacts and larger scene disturbance.
Figure 5: Shove-cheat behavior on Move Can Pot . 5 seeds each; ticks = per-seed rate, band = min–max, dot = mean.
Random
Prism-GRPO
Task
Metric
0.2
0.5
0.9
1
2
0.2
0.5
0.9
1
2
Lift Pot
CS
×
×
×
×
×
56
49
45
25
53
Q
×
×
×
×
×
11
14
15
14
26
Move Can Pot
CS
24
33
×
×
×
41
48
41
3
×
Q
1
0
×
×
×
7
10
13
16
×
Table 1: Quality-weight λ ablation. CS is the rollout saving (%) to match Binary GRPO’s calibrated success; Q is the quality gain at that point. × denotes an unreached success.
Figure 6: Quality-source ablation. Evaluated on Lift Pot .
Figure 7: Real-world behaviors. Top: clean placement from Prism-GRPO (Ours). Bottom: shove-cheat from RL-ZVP.
Metric
Binary GRPO
RL-ZVP
Prism-GRPO
Clean Success
4/25 (16%)
2/25 (8%)
6/25 (24%)
Shove-Cheat
1/25 (4%)
5/25 (20%)
0/25 (0%)
Table 2: Real-world results. Clean Success counts trials that move the can beside the pot without using the shove-cheat.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Peak non-target contact and object displacement on Lift Pot . Stronger unintended contact is associated with larger displacement of non-target objects.
Figure 9: Collision-threshold calibration. Dashed lines mark the thresholds selected at the boundary between the low-impulse cluster and the high-impulse tail.
Figure 10: Smoothness-threshold calibration on Lift Pot .
Figure 11: Sensitivity to the quality threshold on Lift Pot . Prism-Peak shows similar performance for Tpeak∈{3,10}Ns , with 10Ns used as the default.
Figure 12: Task overview. Four tasks under randomized object placements, clutter, lighting, and backgrounds.
Figure 13: Dynamic-filtering statistics across four tasks. Each task panel reports the percentage of zero-variance groups discarded by the trainer (left) and the total number of trajectories generated per optimizer step (right). Prism-GRPO consistently discards fewer groups and consequently remains closer to the minimum cost of 512 rollouts per step.
Figure 14: Rollout-generation cost. Per-step generation time scales nearly linearly with the number of requested rollouts across ten RL runs ( 1,295 measurements). Each color denotes one run.
Figure 15: Full collision-quality results.
Figure 16: Full smoothness-quality results.
Figure 17: Quality-term placement ablation. Full results across success and collision-quality metrics.
Figure 18: Effect of group size. Each optimizer update uses 512 generated rollouts, while the group size varies over G∈{8,16,32} . Prism-GRPO provides the largest gain at G=8 , while the gain decreases as the group size increases.
Figure 19: Quality-weight ablation on Lift Pot .
Figure 20: Quality-weight ablation on Move Can Pot .
Figure 21: Quality-weight decay ablation. Solid lines use a fixed quality weight, dashed lines use decay, and colors indicate different values of λ .
Figure 22: Effect of the advantage estimator. RLOO yields more stable learning and more consistent rollout savings than standard group-normalized GRPO. λ=0.2
Figure 23: Collision-based success–quality alignment. Each task is evaluated using Max-Force, Contact-Count, and Sum-Impulse Quality. Within-scene plots compare the mean quality of successful and failed trajectories generated from the same mixed-outcome scene; positive gaps indicate that successful trajectories are cleaner. Cross-scene plots report the Spearman correlation between scene success rate and mean quality.
Figure 24: Smoothness-based success–quality alignment. Each task is evaluated using maximum joint reversals, mean joint reversals, and action jerk. The within-scene and cross-scene diagnostics follow the same definitions as Figure 23 .
Outcome-driven reinforcement learning offers a scalable way to post-train vision-language-action (VLA) policies from sparse task-success feedback. In common GRPO-based VLA post-training, one rollout-level advantage is applied to every action in the trajectory. A rollout that completes several valid stages but fails later can therefore penalize the actions that produced its earlier progress. We call this trajectory-level credit aliasing. Temporal GRPO addresses this problem by constructing detectable task stages, aligning each rollout with stage-specific action intervals, and comparing only rollouts that have entered the same stage. The resulting stage advantages are applied to their corresponding intervals in a single policy update. On RoboTwin 2.0, Temporal GRPO improves task success and sample efficiency, with consistent gains across task horizons. Controlled updates on LIBERO-Long preserve shared prerequisite stages and concentrate improvement at the first stage where rollout outcomes diverge.
Yao Zhou, Hang Gao, Fengge Wu +2
Institute of Software Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Reinforcement learning (RL) allows vision-language-action (VLA) policies to generalize beyond their training distribution by optimizing directly for task success, but post-training is computationally expensive. A natural response has been to speed rollout collection through faster simulators and world models. In GRPO-based VLA RL, we find that the dominant cost lies elsewhere: gradient computation accounts for approximately 78% of wall-clock time per step in our runs, while rollout collection accounts for only 21%. Gradient cost dominates because much of this computation is spent on phases that contribute little to learning. GRPO's learning signal is driven by advantage variance: only phases where successful and failed rollouts diverge produce learning signal. However, GRPO assigns the same advantage to every chunk in a rollout. As a result, actor-update compute is spent uniformly across the trajectory, including phases the policy already handles after pre-training and supervised fine-tuning. This paper presents Probabilistic Chunk Masking (PCM), a drop-in modification to GRPO that allocates gradient computation to a small, probabilistically selected subset of chunks per trajectory. PCM scores semantic phases using success-failure action variance, a rollout-derived proxy for per-phase gradient variance, and samples a fixed chunk budget with online-updated phase-level keep probabilities. We formalize per-phase gradient variance as the quantity determines where gradient computation is useful and show that success-failure action variance provides a measurable proxy for it. PCM requires no reward model or learned critic. On three LIBERO benchmarks, PCM matches the final success rate of standard GRPO while achieving 2.38 times wall-clock speedup, 4.8 times faster gradient updates, and 60% lower peak activation memory, while backpropagating through fewer than 20% of trajectory chunks.
A failed vision-language-action (VLA) rollout does not imply that every action contributed to failure. Yet in group-relative policy optimization (GRPO), a single trajectory-level advantage is assigned to all actions in a rollout. Potentially useful recovery attempts can thus be penalized alongside the errors they aim to correct. Ideal stopping conditions for failure feedback are characterized using success-probability and policy-score martingales. At these stops, the removed gradient contribution has zero conditional expectation given the retained information. Truncation therefore preserves the expectation of a reference GRPO group-gradient estimator and can reduce its covariance. One such stop occurs when the conditional probability of success is zero, motivating error and recovery cues for selecting candidate feedback boundaries. Martingale GRPO (MART-GRPO) is introduced with a vision-language model (VLM) for event-guided truncation. The VLM is fine-tuned on our annotated training set using low-rank adaptation (LoRA). The last visible error without observed recovery is localized by the frozen VLM in complete rollout videos. Failure feedback is retained through the selected error interval and removed afterward, while successful-rollout loss terms and original loss normalization are preserved. For retrospective boundaries, changes in expectation and variance are analyzed to assess departures from ideal stopping. Experiments with OpenVLA-OFT on RoboTwin 2.0 show that MART-GRPO improves success rates over full-trajectory GRPO across multiple tasks.