When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning

Zongyao Zhao; Cheng-Jie Zhao; Ruihang Jiang; Bing Zheng; Zhaolin Wang; Yuanwei Liu

cs.AIOct 23, 2026

When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning

Authors: Zongyao Zhao, Cheng-Jie Zhao, Ruihang Jiang, Bing Zheng, Zhaolin Wang, Yuanwei Liu

Organizations: The University of Hong Kong · Tsinghua University

Abstract

A failed vision-language-action (VLA) rollout does not imply that every action contributed to failure. Yet in group-relative policy optimization (GRPO), a single trajectory-level advantage is assigned to all actions in a rollout. Potentially useful recovery attempts can thus be penalized alongside the errors they aim to correct. Ideal stopping conditions for failure feedback are characterized using success-probability and policy-score martingales. At these stops, the removed gradient contribution has zero conditional expectation given the retained information. Truncation therefore preserves the expectation of a reference GRPO group-gradient estimator and can reduce its covariance. One such stop occurs when the conditional probability of success is zero, motivating error and recovery cues for selecting candidate feedback boundaries. Martingale GRPO (MART-GRPO) is introduced with a vision-language model (VLM) for event-guided truncation. The VLM is fine-tuned on our annotated training set using low-rank adaptation (LoRA). The last visible error without observed recovery is localized by the frozen VLM in complete rollout videos. Failure feedback is retained through the selected error interval and removed afterward, while successful-rollout loss terms and original loss normalization are preserved. For retrospective boundaries, changes in expectation and variance are analyzed to assess departures from ideal stopping. Experiments with OpenVLA-OFT on RoboTwin 2.0 show that MART-GRPO improves success rates over full-trajectory GRPO across multiple tasks.

Figures & tables

Figure 1: Errors and recovery attempts within a failed rollout. Selected RoboTwin hammer frames from two scenes, a success (top) and a failure (bottom). The red outline marks the hammer after it is lost at chunk 2, and the later frames show recovery attempts. In mixed-outcome groups, full GRPO assigns the error and subsequent recovery attempts the same negative advantage.

Figure 2: Feedback truncation and gradient estimation. Schematic views of (a) ideal conditional projection, (b) a finite-batch estimate before and after suffix masking, and (c) the trade-off between variance reduction and expectation shift.

Figure 3: MART-GRPO: from error localization to feedback truncation. A VLM is fine-tuned offline on event annotations, then frozen during RL. It localizes the last visible error without observed recovery in complete failed rollouts. The interval’s right endpoint sets $B$ : retain negative feedback through $B$ and mask the suffix. Success terms and original token denominators are preserved.

Figure 4: VLM event localization. Per-task interval IoU (a) and cumulative endpoint accuracy (b) on test failures. Epoch 8 is validation-selected.

Task	All	Seen	Held-out	All	Best	Seen	Held-out	All	Best	$\Delta_{40}$	$\Delta_{\mathrm{Best}}$
	SFT	Full GRPO ( Li et al., 2026a )				MART-GRPO
Hammer	30.47	78.12	70.31	74.22	74.61	82.81	79.69	81.25	81.25	+7.03	+6.64
Handover	21.09	38.28	31.25	34.77	35.55	40.62	46.88	43.75	44.53	+8.98	+8.98
Lift pot	5.47	60.16	56.25	58.20	58.20	57.03	60.16	58.59	58.59	+0.39	+0.39
Dual bottles	20.70	50.00	39.84	44.92	44.92	53.91	43.75	48.83	51.17	+3.91	+6.25
Container on Plate	87.11	92.97	94.53	93.75	94.92	92.97	90.62	91.80	94.92	-1.95	+0.00

Table 1: Seven-task policy performance. Success (%) on 128 training-seed (Seen) and 128 held-out-seed (Held-out) episodes. All pools both splits. RL scores are at update 40 except Best, the maximum All over updates

5,10,\ldots,40

. SFT is the initial score.

\Delta_{40}

and

\Delta_{\mathrm{Best}}

give MART-GRPO gains over Full GRPO (percentage points). Means and differences are computed before rounding.

Method	Failure feedback	Hammer				Handover
		10	20	30	40	10	20	30	40
Full GRPO	Full trajectory	48.05	53.91	65.23	74.22	32.03	31.25	35.55	34.77
Uniform scaling	Uniformly scaled	51.56	63.28	65.23	77.73	29.69	33.98	37.89	40.62
Random prefix	Random prefix	51.17	67.19	75.00	71.88	25.78	32.03	38.67	41.41
MART-GRPO	Event-guided prefix	49.61	69.14	78.12	81.25	25.39	35.16	42.58	43.75

Table 2: Feedback ablations. Success (%) pooled across both seed splits at updates 10, 20, 30, and 40. Column bests are bold. Reported differences are computed before rounding.

Figure 5: Final performance on Hammer and Handover. Success after 40 updates, with SFT references and task-specific horizontal scales.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Setting
Policy and rollout collection
Server capacity	8 NVIDIA H100 GPUs
Initialization	Task-specific SFT checkpoint
Actor observations	Head-camera view and proprioception
Environment	Randomized RoboTwin 2.0 scenes, seen instructions
Training initial conditions	1,000 per task

Appendix

Table 3: Policy training and evaluation settings. Configuration for the reported policy comparisons and feedback ablations.

Task	Episode budget		Full GRPO success (%) at update
	Max. action steps	Max. chunks	0	5	10	15	20	25	30	35	40
Hammer	200	8	30.5	35.5	48.0	46.9	53.9	63.7	65.2	74.6	74.2
Handover	400	16	21.1	24.6	32.0	30.5	31.2	34.4	35.5	33.6	34.8
Lift pot	200	8	5.5	20.7	29.3	35.9	44.5	46.1	52.0	54.3	58.2
Dual bottles	100	4	20.7	21.5	32.0	30.1	35.9	44.1	43.0	41.4	44.9
Container on Plate	150	6	87.1	91.4	91.4	93.8	92.6	93.0	93.8	94.9	93.8

Appendix

Table 4: Task budgets and Full GRPO learning progress. Success (%) pooled over 128 seen-seed and 128 held-out-seed episodes at each update. Update 0 is the SFT evaluation. Episode budgets are maxima, with 25 action steps per chunk.

Input	Source	Representation
Task context	Task configuration and rollout metadata	Natural-language instruction, task clarification, executed chunk count, and initial action budget.
Camera streams	Head and two wrist cameras	Three synchronized video sequences. Native $320\times 240$ RGB frames are extended to $320\times 288$ with camera, frame, and chunk labels. Sampling uses the initial frame and up to six frames per chunk.
Robot state	Robot observation records	Gripper commands, measured openings, and end-effector poses at each sampled frame. Joint positions and velocities are included at the initial frame and chunk ends. Values are serialized as JSON text.
Temporal alignment	Rollout recorder	Frame and chunk indices, within-chunk progress, and simulation timestamps.

Appendix

Table 5: VLM inputs. Sources and representation for a single rollout request.

Video type	Training	Validation	Test	Total
Failed rollouts	640	128	256	1,024
Successful references	118	5	4	127
Total	758	133	260	1,151

Appendix

Table 6: Annotated video corpus. Video counts by environment outcome and data split.

Panel of Figure 6 — Figure 6: An annotated rollout. Selected head-camera frames and event labels from the video corpus. The interval identifies the placement error after the initial grasping difficulty is resolved. Chunk indices are zero-based.

Parameter	Setting
Adapter
Adaptation scope	One shared adapter across tasks
Target layers	q_proj , k_proj , v_proj , o_proj , in_proj_qkv , in_proj_z , in_proj_b , in_proj_a , out_proj
Rank	8
LoRA scaling parameter	16
Dropout	0

Appendix

Table 7: LoRA training configuration. Settings for the shared VLM event adapter.

Task	$n$	Base	Epoch 3	Epoch 6	Epoch 8
beat_block_hammer	32	34.0	21.9	57.8	60.9
handover_block	32	36.5	43.8	70.3	73.4
lift_pot	32	35.7	60.9	77.6	80.7
pick_dual_bottles	32	33.3	70.3	75.0	73.4
place_container_plate	31	31.7	58.1	79.0	77.4
place_empty_cup	32	24.8	53.1	59.4	72.9

Appendix

Table 8: Seven-task event localization. Interval IoU (%) against versioned reference intervals on 220 localizable, unrecovered test failures from the tasks in the policy comparison. Update 304 is the validation-selected checkpoint. Scores measure label agreement under the evaluation protocol in Appendix E.3 .

Figure 7: Event localization on a test rollout. Head-camera frames of a failed Handover. Blue outlines mark the interval identified by both the reference annotation and the selected model. Chunk indices are zero-based.

Figure 8: Predicted intervals across checkpoints. Interval endpoints for the Handover rollout in Figure 7 . The shaded band marks the reference interval

[4,5]

	Hammer		Handover		Lift pot		Dual bottles		Container		Empty cup		Stack bowls
Update	Full	MART	Full	MART	Full	MART	Full	MART	Full	MART	Full	MART	Full	MART
0	30.5	30.9	21.1	—	5.5	5.9	20.7	20.3	87.1	86.3	80.9	80.9	39.1	37.1
5	35.5	42.6	24.6	25.8	20.7	24.6	21.5	23.0	91.4	88.7	82.8	84.4	46.1	40.2
10	48.0	49.6	32.0	25.4	29.3	31.2	32.0	25.0	91.4	91.4	86.7	87.5	47.7	52.3
15	46.9	63.3	30.5	39.1	35.9	38.3	30.1	34.4	93.8	94.9	83.6	89.5	56.2	56.2
20	53.9	69.1	31.2	35.2	44.5	42.2	35.9	37.1	92.6	94.1	85.9	89.5	57.0	59.0

Appendix

Table 9: Scheduled policy evaluations. Combined success (%) at each scheduled update. Table 1 reports update 40 and the per-run maximum over updates

5,10,\ldots,40

. SFT evaluations at update 0 use the same task checkpoint in separate simulator executions. The dash denotes the unavailable initial Handover MART evaluation.

Task	$N_{-}$	Shortened (%)	Retained mass (%)	Fallback (%)
Hammer	8,304	90.6	49.2	8.3
Handover	12,955	99.4	32.7	0.5
Lift pot	11,577	99.7	47.4	0.2
Dual bottles	13,296	97.9	59.3	1.9
Container on Plate	4,364	97.8	51.8	2.0
Empty cup	5,612	94.7	53.6	4.8

Appendix

Table 10: What the event mask removes. Pooled over all 40 updates of each MART run.

N_{-}

counts accepted failed trajectories. Shortened denotes a boundary before the final chunk. Retained mass is the ratio of summed negative coefficient mass after and before masking. Fallbacks retain full feedback. These statistics quantify loss weights.

Figure 9: Feedback ablations. Success at update 40, with dashed Full GRPO references and percentage-point changes relative to Full GRPO above the bars. Percentages inside the bars show each run’s observed retained negative coefficient mass, pooled over all 40 updates.

Policy Optimization

Policy optimization develops algorithms for improving decision rules in reinforcement learning, language models, and sequential decision systems.

See all 221 papers

Temporal GRPO: Beyond Trajectory-Level Credit in Vision-Language-Action Reinforcement Learning
Aug 13, 2026Yao Zhou, Hang Gao, Fengge Wu +2Group Relative Policy Optimization Step-Level Credit Assignment in RL
Institute of Software Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models
May 10, 2026Weijia Liufu, Xiaoyu Guo, Ruiyi Chen +16VLM Robustness Robot Failure Recovery
1Sun Yat-sen University · 2South China University of Technology · 3Peng Cheng Laboratory +3
Prism-GRPO: Faster VLA Policy Optimization via Splitting Same-outcome Groups
Aug 18, 2026Zeyun Deng, Yuzhe Lu, Yawei Wang +6Robotic RL Group Relative Policy Optimization
Purdue University · AWS AI

Aug 13, 2026J/K move · Enter open · S save

Temporal GRPO: Beyond Trajectory-Level Credit in Vision-Language-Action Reinforcement Learning

Yao Zhou, Hang Gao, Fengge Wu, Changwen Zheng+1

Institute of Software Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

Group Relative Policy Optimization Step-Level Credit Assignment in RL Temporal Credit Assignment RL Post-Training Policy Learning RL for VLMs

Open paper AskPDF arXiv

Outcome-driven reinforcement learning offers a scalable way to post-train vision-language-action (VLA) policies from sparse task-success feedback. In common GRPO-based VLA post-training, one rollout-level advantage is applied to every action in the trajectory. A rollout that completes several valid stages but fails later can therefore penalize the actions that produced its earlier progress. We call this trajectory-level credit aliasing. Temporal GRPO addresses this problem by constructing detectable task stages, aligning each rollout with stage-specific action intervals, and comparing only rollouts that have entered the same stage. The resulting stage advantages are applied to their corresponding intervals in a single policy update. On RoboTwin 2.0, Temporal GRPO improves task success and sample efficiency, with consistent gains across task horizons. Controlled updates on LIBERO-Long preserve shared prerequisite stages and concentrate improvement at the first stage where rollout outcomes diverge.

Abstract

Policy Optimization

Explore similar work

Temporal GRPO: Beyond Trajectory-Level Credit in Vision-Language-Action Reinforcement Learning

RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models

Prism-GRPO: Faster VLA Policy Optimization via Splitting Same-outcome Groups