A failed vision-language-action (VLA) rollout does not imply that every action contributed to failure. Yet in group-relative policy optimization (GRPO), a single trajectory-level advantage is assigned to all actions in a rollout. Potentially useful recovery attempts can thus be penalized alongside the errors they aim to correct. Ideal stopping conditions for failure feedback are characterized using success-probability and policy-score martingales. At these stops, the removed gradient contribution has zero conditional expectation given the retained information. Truncation therefore preserves the expectation of a reference GRPO group-gradient estimator and can reduce its covariance. One such stop occurs when the conditional probability of success is zero, motivating error and recovery cues for selecting candidate feedback boundaries. Martingale GRPO (MART-GRPO) is introduced with a vision-language model (VLM) for event-guided truncation. The VLM is fine-tuned on our annotated training set using low-rank adaptation (LoRA). The last visible error without observed recovery is localized by the frozen VLM in complete rollout videos. Failure feedback is retained through the selected error interval and removed afterward, while successful-rollout loss terms and original loss normalization are preserved. For retrospective boundaries, changes in expectation and variance are analyzed to assess departures from ideal stopping. Experiments with OpenVLA-OFT on RoboTwin 2.0 show that MART-GRPO improves success rates over full-trajectory GRPO across multiple tasks.
Figures & tables
Figure 1: Errors and recovery attempts within a failed rollout. Selected RoboTwin hammer frames from two scenes, a success (top) and a failure (bottom). The red outline marks the hammer after it is lost at chunk 2, and the later frames show recovery attempts. In mixed-outcome groups, full GRPO assigns the error and subsequent recovery attempts the same negative advantage.
Figure 2: Feedback truncation and gradient estimation. Schematic views of (a) ideal conditional projection, (b) a finite-batch estimate before and after suffix masking, and (c) the trade-off between variance reduction and expectation shift.
Figure 3: MART-GRPO: from error localization to feedback truncation. A VLM is fine-tuned offline on event annotations, then frozen during RL. It localizes the last visible error without observed recovery in complete failed rollouts. The interval’s right endpoint sets B : retain negative feedback through B and mask the suffix. Success terms and original token denominators are preserved.
Figure 4: VLM event localization. Per-task interval IoU (a) and cumulative endpoint accuracy (b) on test failures. Epoch 8 is validation-selected.
SFT
Full GRPO ( Li et al., 2026a )
MART-GRPO
Task
All
Seen
Held-out
All
Best
Seen
Held-out
All
Best
Δ40
ΔBest
Hammer
30.47
78.12
70.31
74.22
74.61
82.81
79.69
81.25
81.25
+7.03
+6.64
Handover
21.09
38.28
31.25
34.77
35.55
40.62
46.88
43.75
44.53
+8.98
+8.98
Lift pot
5.47
60.16
56.25
58.20
58.20
57.03
60.16
58.59
58.59
+0.39
+0.39
Dual bottles
20.70
50.00
39.84
44.92
44.92
53.91
43.75
48.83
51.17
+3.91
+6.25
Container on Plate
87.11
92.97
94.53
93.75
94.92
92.97
90.62
91.80
94.92
-1.95
+0.00
Table 1: Seven-task policy performance. Success (%) on 128 training-seed (Seen) and 128 held-out-seed (Held-out) episodes. All pools both splits. RL scores are at update 40 except Best, the maximum All over updates 5,10,…,40 . SFT is the initial score. Δ40 and ΔBest give MART-GRPO gains over Full GRPO (percentage points). Means and differences are computed before rounding.
Method
Failure feedback
Hammer
Handover
10
20
30
40
10
20
30
40
Full GRPO
Full trajectory
48.05
53.91
65.23
74.22
32.03
31.25
35.55
34.77
Uniform scaling
Uniformly scaled
51.56
63.28
65.23
77.73
29.69
33.98
37.89
40.62
Random prefix
Random prefix
51.17
67.19
75.00
71.88
25.78
32.03
38.67
41.41
MART-GRPO
Event-guided prefix
49.61
69.14
78.12
81.25
25.39
35.16
42.58
43.75
Table 2: Feedback ablations. Success (%) pooled across both seed splits at updates 10, 20, 30, and 40. Column bests are bold. Reported differences are computed before rounding.
Figure 5: Final performance on Hammer and Handover. Success after 40 updates, with SFT references and task-specific horizontal scales.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Setting
Policy and rollout collection
Server capacity
8 NVIDIA H100 GPUs
Initialization
Task-specific SFT checkpoint
Actor observations
Head-camera view and proprioception
Environment
Randomized RoboTwin 2.0 scenes, seen instructions
Training initial conditions
1,000 per task
Appendix
Table 3: Policy training and evaluation settings. Configuration for the reported policy comparisons and feedback ablations.
Task
Episode budget
Full GRPO success (%) at update
Max. action steps
Max. chunks
0
5
10
15
20
25
30
35
40
Hammer
200
8
30.5
35.5
48.0
46.9
53.9
63.7
65.2
74.6
74.2
Handover
400
16
21.1
24.6
32.0
30.5
31.2
34.4
35.5
33.6
34.8
Lift pot
200
8
5.5
20.7
29.3
35.9
44.5
46.1
52.0
54.3
58.2
Dual bottles
100
4
20.7
21.5
32.0
30.1
35.9
44.1
43.0
41.4
44.9
Container on Plate
150
6
87.1
91.4
91.4
93.8
92.6
93.0
93.8
94.9
93.8
Appendix
Table 4: Task budgets and Full GRPO learning progress. Success (%) pooled over 128 seen-seed and 128 held-out-seed episodes at each update. Update 0 is the SFT evaluation. Episode budgets are maxima, with 25 action steps per chunk.
Three synchronized video sequences. Native 320×240 RGB frames are extended to 320×288 with camera, frame, and chunk labels. Sampling uses the initial frame and up to six frames per chunk.
Robot state
Robot observation records
Gripper commands, measured openings, and end-effector poses at each sampled frame. Joint positions and velocities are included at the initial frame and chunk ends. Values are serialized as JSON text.
Temporal alignment
Rollout recorder
Frame and chunk indices, within-chunk progress, and simulation timestamps.
Appendix
Table 5: VLM inputs. Sources and representation for a single rollout request.
Video type
Training
Validation
Test
Total
Failed rollouts
640
128
256
1,024
Successful references
118
5
4
127
Total
758
133
260
1,151
Appendix
Table 6: Annotated video corpus. Video counts by environment outcome and data split.
Figure 6: An annotated rollout. Selected head-camera frames and event labels from the video corpus. The interval identifies the placement error after the initial grasping difficulty is resolved. Chunk indices are zero-based.
Table 7: LoRA training configuration. Settings for the shared VLM event adapter.
Task
n
Base
Epoch 3
Epoch 6
Epoch 8
beat_block_hammer
32
34.0
21.9
57.8
60.9
handover_block
32
36.5
43.8
70.3
73.4
lift_pot
32
35.7
60.9
77.6
80.7
pick_dual_bottles
32
33.3
70.3
75.0
73.4
place_container_plate
31
31.7
58.1
79.0
77.4
place_empty_cup
32
24.8
53.1
59.4
72.9
Appendix
Table 8: Seven-task event localization. Interval IoU (%) against versioned reference intervals on 220 localizable, unrecovered test failures from the tasks in the policy comparison. Update 304 is the validation-selected checkpoint. Scores measure label agreement under the evaluation protocol in Appendix E.3 .
Figure 7: Event localization on a test rollout. Head-camera frames of a failed Handover. Blue outlines mark the interval identified by both the reference annotation and the selected model. Chunk indices are zero-based.
Figure 8: Predicted intervals across checkpoints. Interval endpoints for the Handover rollout in Figure 7 . The shaded band marks the reference interval [4,5] .
Hammer
Handover
Lift pot
Dual bottles
Container
Empty cup
Stack bowls
Update
Full
MART
Full
MART
Full
MART
Full
MART
Full
MART
Full
MART
Full
MART
0
30.5
30.9
21.1
—
5.5
5.9
20.7
20.3
87.1
86.3
80.9
80.9
39.1
37.1
5
35.5
42.6
24.6
25.8
20.7
24.6
21.5
23.0
91.4
88.7
82.8
84.4
46.1
40.2
10
48.0
49.6
32.0
25.4
29.3
31.2
32.0
25.0
91.4
91.4
86.7
87.5
47.7
52.3
15
46.9
63.3
30.5
39.1
35.9
38.3
30.1
34.4
93.8
94.9
83.6
89.5
56.2
56.2
20
53.9
69.1
31.2
35.2
44.5
42.2
35.9
37.1
92.6
94.1
85.9
89.5
57.0
59.0
Appendix
Table 9: Scheduled policy evaluations. Combined success (%) at each scheduled update. Table 1 reports update 40 and the per-run maximum over updates 5,10,…,40 . SFT evaluations at update 0 use the same task checkpoint in separate simulator executions. The dash denotes the unavailable initial Handover MART evaluation.
Task
N−
Shortened (%)
Retained mass (%)
Fallback (%)
Hammer
8,304
90.6
49.2
8.3
Handover
12,955
99.4
32.7
0.5
Lift pot
11,577
99.7
47.4
0.2
Dual bottles
13,296
97.9
59.3
1.9
Container on Plate
4,364
97.8
51.8
2.0
Empty cup
5,612
94.7
53.6
4.8
Appendix
Table 10: What the event mask removes. Pooled over all 40 updates of each MART run. N− counts accepted failed trajectories. Shortened denotes a boundary before the final chunk. Retained mass is the ratio of summed negative coefficient mass after and before masking. Fallbacks retain full feedback. These statistics quantify loss weights.
Figure 9: Feedback ablations. Success at update 40, with dashed Full GRPO references and percentage-point changes relative to Full GRPO above the bars. Percentages inside the bars show each run’s observed retained negative coefficient mass, pooled over all 40 updates.
Outcome-driven reinforcement learning offers a scalable way to post-train vision-language-action (VLA) policies from sparse task-success feedback. In common GRPO-based VLA post-training, one rollout-level advantage is applied to every action in the trajectory. A rollout that completes several valid stages but fails later can therefore penalize the actions that produced its earlier progress. We call this trajectory-level credit aliasing. Temporal GRPO addresses this problem by constructing detectable task stages, aligning each rollout with stage-specific action intervals, and comparing only rollouts that have entered the same stage. The resulting stage advantages are applied to their corresponding intervals in a single policy update. On RoboTwin 2.0, Temporal GRPO improves task success and sample efficiency, with consistent gains across task horizons. Controlled updates on LIBERO-Long preserve shared prerequisite stages and concentrate improvement at the first stage where rollout outcomes diverge.
Yao Zhou, Hang Gao, Fengge Wu +2
Institute of Software Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Vision-Language-Action (VLA) models remain brittle in long-horizon, contact-rich manipulation because success-only imitation provides little supervision for execution drift, while failed rollouts are often discarded. We introduce RePO-VLA, a recovery-driven policy optimization framework that assigns distinct roles to success, recovery, and failure trajectories. RePO-VLA first applies Recovery-Aware Initialization (RAI), slicing recovery segments and resetting history so corrective actions depend on the current adverse state rather than the preceding failure. It then learns a Progress-Aware Semantic Value Function (PAS-VF), aligning spatiotemporal trajectory features with instructions and successful references. The resulting labels salvage useful failure prefixes via reliability decay, while low-value labels mark drift and terminal breakdowns, teaching differences among nominal, failed, and corrective actions. The data engine turns adverse states into planner-generated or human-collected corrective rollouts, teaching recovery to the success manifold. Value-Conditioned Refinement (VCR) trains the policy to prefer high-progress actions. At deployment, a fixed high value (v=1.0) biases actions toward the learned success manifold without online failure detectors or heuristic retries. We introduce FRBench, with standardized error injection and recovery-focused evaluation. Across simulated and real-world bimanual tasks, RePO-VLA improves robustness, raising adversarial success from 20% to 75% on average and up to 80% in scaled real-world trials.
Weijia Liufu, Xiaoyu Guo, Ruiyi Chen +16
1Sun Yat-sen University · 2South China University of Technology · 3Peng Cheng Laboratory +3
GRPO is increasingly used for reinforcement learning of vision-language-action (VLA) policies because, unlike PPO, it does not require training a critic. This simplification comes with a sampling cost: group-relative advantages require multiple rollouts from each scene. Under binary success rewards, groups whose rollouts all succeed or all fail have zero advantage and are discarded by dynamic sampling. These groups are especially common early in training, when most rollouts fail, wasting much of the expensive robotic rollout budget. We introduce Prism-GRPO, which augments binary outcome reward with a weighted trajectory-level execution-quality score. By splitting same-outcome groups into a quality spectrum, Prism-GRPO recovers training signal while ensuring that every success still outranks every failure. Quality scores can be derived from simulator contacts, executed actions, or visual observations, avoiding task-specific progress rewards. We prove that Prism-GRPO never increases the probability that a sampled group is discarded for having zero advantages, and derive a gradient-alignment condition under which its combined update remains a local ascent direction for task success. Across four RoboTwin tasks spanning different horizons and coordination patterns, Prism-GRPO improves success and quality at matched rollout budgets and reaches target success rates with up to 56% fewer rollouts. It also suppresses a reward-hacking shortcut, with the cleaner behavior transferring under direct deployment to a real robot. Through ablations, we show consistent gains across contact-, smoothness-, and VLM-derived quality signals.