Learning task progress from robot demonstrations provides a practical way to construct dense shaping rewards for robotic reinforcement learning. A common approach uses normalized demonstration time as progress labels, implicitly assuming monotonically increasing task progress. During policy exploration, however, actual task progress can stall or even regress. Such models may mistake regressions for progress, assigning positive shaping rewards to actions that cause progress loss. Real suboptimal and failed executions offer corrective experience, but assigning absolute progress values to their states is difficult. Our key observation is that stalls, regressions, and recoveries establish local relative progress relations even when absolute progress is unknown. We introduce Event2Reward, which uses normalized temporal labels to anchor the progress scale and these within-trajectory relations to constrain local changes, combining absolute and relative progress supervision. Its progress predictions serve as potentials for reward shaping. We curate an event-annotated dataset from approximately 100K raw real-robot execution records, covering successful, suboptimal, and failed executions, without requiring framewise absolute progress labels. Under the same training budgets, Event2Reward improves average policy success by 18.16 percentage points over a baseline using only sparse task rewards across 40 tasks in four LIBERO suites. On real robots, average success after 30 minutes of wall-clock training reaches 60.0% on three precise insertion tasks and 90.0% on three object manipulation tasks, exceeding the sparse-reward baseline under the same budget by 36.7 and 23.3 percentage points, respectively. Ablation studies further show that absolute progress supervision and relative progress relations are complementary, with joint supervision yielding better downstream policy learning.
Figures & tables
Figure 1: Overview of Event2Reward. Left: pretraining data and real robot rollouts containing both successful and failed executions. Task events provide within-trajectory progress relations that complement temporal labels from successful demonstrations. Top center: task-conditioned progress predictions support reward shaping. Right: six real-world downstream tasks. The performance panels summarize real-world and LIBERO policy success, plug-insertion learning time, and the effect of rollout-based joint training. Plug-insertion time is measured until a rolling window of 10 autonomous training episodes reaches 100% success.
Figure 2: Overview of progress supervision in Event2Reward. Left: normalized temporal labels from successful expert trajectories anchor the progress scale of a shared vision-language model. Right: execution events in robot rollouts support relative constraints, illustrated by a drop and subsequent recovery ( p8>p12 and p16>p12 ). These complementary signals supervise the same progress model; relation supervision uses both successful and failed executions.
Figure 3: Policy learning on four LIBERO-10 tasks with different reward supervision. Lines show mean independent-evaluation success rates across five environment seeds; shaded bands indicate ±1.96SEM . Horizontal axes show environment steps up to the shared 1.2M budget.
Supervision
Relation satisfaction rate (%)
Mean prediction range
Progress-only
41.62
0.5928
Relation-only
58.86
0.0021
Progress + Relation
54.47
0.5097
Table 1: Progress prediction diagnostics. Relation satisfaction is evaluated on 46 unrecovered regression trajectories and averaged equally across their nine tasks. Prediction range is averaged equally across all 103 held-out trajectories covering ten LIBERO-10 tasks.
Figure 4: Policy learning on eight LIBERO tasks with different reward models. Lines show mean independent-evaluation success rates across five environment seeds; shaded bands indicate ±1.96SEM . Horizontal axes show environment steps up to the shared 200k budget.
Figure 5: Real-world tasks for policy learning. Top: insertion tasks trained with SERL. Bottom: manipulation tasks trained with DSRL using a π0 base policy. Insets in (a) and (b) show oblique views of the socket and RAM slots. The inset in (c) shows the Ethernet plug and port in an earlier frame before insertion. In Press Buttons, the translucent overlay depicts an earlier end-effector pose during the second press, while the opaque pose shows contact with the pink button.
Figure 6: Success rate (%) after 30 minutes of wall-clock training, including initial buffer warm-up where applicable, resets, and reward computation. Bars show means and error bars show sample standard deviations across three independent training runs per task and reward condition. Each run’s checkpoint at 30 minutes is evaluated over 20 autonomous episodes.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Additional controls and reward-model baselines. Independent evaluation success of Progress + Relation, Relation-loss-off, Sparse, and Robometer on LIBERO-10 Tasks 1, 3, 7, and 9. Lines show means across five environment seeds; shaded bands indicate ±1.96SEM . The horizontal axis shows environment interaction steps.
Figure 8: A held-out example of progress-scale collapse. Predicted progress on a LIBERO-10 Task 7 trajectory, shown on a common, unnormalized scale. Each curve connects 25 sampled predictions with straight lines: 16 event-comparison points (filled circles), eight additional trajectory-diagnostic points (open squares), and one reference point (diamond). The dashed line marks the reference time, and shading indicates the progress-loss comparison interval. Only filled circles enter the reported decrease counts; the five open squares within the shaded interval are excluded.
Figure 4
Suite
Task ID
Instruction
Task 1
LIBERO-Object
Task02
pick up the salad dressing and place it in the basket
Task 2
LIBERO-Goal
Task06
put the cream cheese in the bowl
Task 3
LIBERO-Goal
Task09
put the wine bottle on the rack
Task 4
LIBERO-Object
Task08
pick up the chocolate pudding and place it in the basket
Task 5
LIBERO-Object
Task00
pick up the alphabet soup and place it in the basket
Task 6
LIBERO-Goal
Task05
push the plate to the front of the stove
Appendix
Table 17: Tasks shown in Figure 4 . Task IDs follow Figures 9 – 12 ; on each task, Event2Reward improves final success over the strongest baseline.
Figure 9: Independent evaluation on all ten LIBERO-Goal tasks. Task IDs follow the original suite ordering. The horizontal axis shows training environment steps. Curves extend to 240k steps; quantitative comparisons use 200k. Lines and shaded bands show the mean and ±1.96SEM across five environment seeds.
Figure 10: Independent evaluation on all ten LIBERO-Object tasks. Task IDs follow the original suite ordering. The horizontal axis shows training environment steps. Curves extend to 240k steps; quantitative comparisons use 200k. Lines and shaded bands show the mean and ±1.96SEM across five environment seeds.
Figure 11: Independent evaluation on all ten LIBERO-Spatial tasks. Task IDs follow the original suite ordering. The horizontal axis shows training environment steps. Curves extend to 240k steps; quantitative comparisons use 200k. Lines and shaded bands show the mean and ±1.96SEM across five environment seeds.
Figure 12: Independent evaluation on all ten LIBERO-10 tasks. Task IDs follow the original suite ordering. The horizontal axis shows training environment steps within the 1.2M-step budget; each task is plotted over its fixed common observation interval without extrapolation. Lines and shaded bands show the mean and ±1.96SEM across five environment seeds.
Figure 13: Rollout joint-training ablation: independent evaluation. Each task compares Event2Reward with Event2Reward w/o rollout joint training using 200 or 40 adaptation trajectories. Lines show means across five environment seeds and bands show ±1.96SEM , clipped to [0,1] . Curves extend to 240k training environment steps; the main quantitative comparison uses 200k. Curves are linearly interpolated only within the current runs’ common observed interval for each task, with no additional smoothing or extrapolation.
Reinforcement learning (RL) enables generalist robot policies to improve through trial-and-error interaction, yet its effectiveness is fundamentally constrained by sparse task rewards. Existing general-purpose reward models typically alleviate this issue by learning task progress from expert demonstrations, but introduce a distribution mismatch with the mixed-quality rollouts encountered during policy optimization, making their estimates unreliable on suboptimal and failed behaviors from which the policy must learn. In this work, we introduce a demonstration-free reward learning paradigm where dense reward feedback can be learned directly from sparse task outcomes and policy experience. We theoretically show that terminal task outcomes implicitly define dense success-probability feedback at intermediate timesteps, which can be recursively learned through bootstrapping. Based on this insight, we introduce eVTA0, which learns success probabilities from mixed-quality policy rollouts through temporal-difference-style bootstrapping, without expert demonstrations or intermediate annotations. We further introduce RL with Evolving Rewards (RLER), a closed-loop framework that adapts eVTA0 using newly collected rollouts as the policy evolves. Experiments show that eVTA0 provides more informative rewards than state-of-the-art reward models and achieves the best average policy performance across all LIBERO task suites under the same RL training budget, improving success rates by 5.4%-13.8% over the initial policy. In real-world manipulation, RLER further improves overall success rates by 20%-26%, with 35%-36% gains under out-of-distribution conditions. These results demonstrate the effectiveness of demonstration-free reward learning and adapting rewards as the policy evolves. Project webpage: https://duowuyms.github.io/evta0.
Duo Wu, Haifeng Wang, Rongwei Lu +6
Tsinghua University · Yuanxing Robotics · Pengcheng Laboratory
Learning robotic policies requires dense rewards that remain informative when behavior departs from successful demonstrations. Progress-based rewards estimate how far an observation has advanced along a nominal successful trajectory, but may remain high after an incorrect transition. We introduce Dream2Reward, which learns a language-conditioned successful latent transition field from positive demonstrations. Given the visual history up to a transition start, the model predicts the latent displacement associated with successful execution and scores the observed displacement through signed directional and symmetric magnitude agreement. This transition-level comparison penalizes wrong-direction, overshooting, and stagnant motion even when the resulting observation appears to show progress. Dream2Reward requires no failure annotations, progress labels, or synthetic negatives, and produces a dense causal reward. Across mechanism diagnostics and shared-trajectory evaluations, it provides stronger success-failure separation and more informative feedback on low-quality behavior than progress-based alternatives. Across online and offline policy learning, the same frozen reward model reduces reward hacking and supports stronger downstream performance, including in real-robot manipulation. These results show that comparing realized motion with predicted successful change provides an effective way to convert positive demonstrations into dense rewards for robot learning.
Haoyu Zhang, Zecui Zeng, Bin Wang +3
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · JD Explore Academy, China
General-purpose robot reward models are typically trained to predict absolute task progress from expert demonstrations, providing only local, frame-level supervision. While effective for expert demonstrations, this paradigm scales poorly to large-scale robotics datasets where failed and suboptimal trajectories are abundant and assigning dense progress labels is ambiguous. We introduce Robometer, a scalable reward modeling framework that combines intra-trajectory progress supervision with inter-trajectory preference supervision. Robometer is trained with a dual objective: a frame-level progress loss that anchors reward magnitude on expert data, and a trajectory-comparison preference loss that imposes global ordering constraints across trajectories of the same task, enabling effective learning from both real and augmented failed trajectories. To support this formulation at scale, we curate RBM-1M, a reward-learning dataset comprising over one million trajectories spanning diverse robot embodiments and tasks, including substantial suboptimal and failure data. Across benchmarks and real-world evaluations, Robometer learns more generalizable reward functions than prior methods and improves robot learning performance across a diverse set of downstream applications. Code, model weights, and videos at https://robometer.github.io/.