Event2Reward: Learning Robotic Rewards from Progress Relations

Junnan Nie; Zitian Fan; Jiachen Zhang; Junyi Lao; Likui Zhang; Chenghao Liu; Xiaoyang Lu; Yunbo Wang

cs.ROOct 29, 2026

Event2Reward: Learning Robotic Rewards from Progress Relations

Authors: Junnan Nie, Zitian Fan, Jiachen Zhang, Junyi Lao, Likui Zhang, Chenghao Liu, Xiaoyang Lu, Yunbo Wang,

Organizations: Peking University · JD Explore Academy

Abstract

Learning task progress from robot demonstrations provides a practical way to construct dense shaping rewards for robotic reinforcement learning. A common approach uses normalized demonstration time as progress labels, implicitly assuming monotonically increasing task progress. During policy exploration, however, actual task progress can stall or even regress. Such models may mistake regressions for progress, assigning positive shaping rewards to actions that cause progress loss. Real suboptimal and failed executions offer corrective experience, but assigning absolute progress values to their states is difficult. Our key observation is that stalls, regressions, and recoveries establish local relative progress relations even when absolute progress is unknown. We introduce Event2Reward, which uses normalized temporal labels to anchor the progress scale and these within-trajectory relations to constrain local changes, combining absolute and relative progress supervision. Its progress predictions serve as potentials for reward shaping. We curate an event-annotated dataset from approximately 100K raw real-robot execution records, covering successful, suboptimal, and failed executions, without requiring framewise absolute progress labels. Under the same training budgets, Event2Reward improves average policy success by 18.16 percentage points over a baseline using only sparse task rewards across 40 tasks in four LIBERO suites. On real robots, average success after 30 minutes of wall-clock training reaches 60.0% on three precise insertion tasks and 90.0% on three object manipulation tasks, exceeding the sparse-reward baseline under the same budget by 36.7 and 23.3 percentage points, respectively. Ablation studies further show that absolute progress supervision and relative progress relations are complementary, with joint supervision yielding better downstream policy learning.

Figures & tables

Figure 1: Overview of Event2Reward. Left: pretraining data and real robot rollouts containing both successful and failed executions. Task events provide within-trajectory progress relations that complement temporal labels from successful demonstrations. Top center: task-conditioned progress predictions support reward shaping. Right: six real-world downstream tasks. The performance panels summarize real-world and LIBERO policy success, plug-insertion learning time, and the effect of rollout-based joint training. Plug-insertion time is measured until a rolling window of 10 autonomous training episodes reaches 100% success.

Figure 2: Overview of progress supervision in Event2Reward. Left: normalized temporal labels from successful expert trajectories anchor the progress scale of a shared vision-language model. Right: execution events in robot rollouts support relative constraints, illustrated by a drop and subsequent recovery ( $p_{8}>p_{12}$ and $p_{16}>p_{12}$ ). These complementary signals supervise the same progress model; relation supervision uses both successful and failed executions.

Figure 3: Policy learning on four LIBERO-10 tasks with different reward supervision. Lines show mean independent-evaluation success rates across five environment seeds; shaded bands indicate

\pm 1.96\,\mathrm{SEM}

. Horizontal axes show environment steps up to the shared 1.2M budget.

Supervision	Relation satisfaction rate (%)	Mean prediction range
Progress-only	41.62	0.5928
Relation-only	58.86	0.0021
Progress + Relation	54.47	0.5097

Table 1: Progress prediction diagnostics. Relation satisfaction is evaluated on 46 unrecovered regression trajectories and averaged equally across their nine tasks. Prediction range is averaged equally across all 103 held-out trajectories covering ten LIBERO-10 tasks.

Figure 4: Policy learning on eight LIBERO tasks with different reward models. Lines show mean independent-evaluation success rates across five environment seeds; shaded bands indicate

\pm 1.96\,\mathrm{SEM}

. Horizontal axes show environment steps up to the shared 200k budget.

Panel of Figure 5 — Figure 5: Real-world tasks for policy learning. Top: insertion tasks trained with SERL. Bottom: manipulation tasks trained with DSRL using a $\pi_{0}$ base policy. Insets in (a) and (b) show oblique views of the socket and RAM slots. The inset in (c) shows the Ethernet plug and port in an earlier frame before insertion. In Press Buttons, the translucent overlay depicts an earlier end-effector pose during the second press, while the opaque pose shows contact with the pink button.

Figure 6: Success rate (%) after 30 minutes of wall-clock training, including initial buffer warm-up where applicable, resets, and reward computation. Bars show means and error bars show sample standard deviations across three independent training runs per task and reward condition. Each run’s checkpoint at 30 minutes is evaluated over 20 autonomous episodes.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Additional controls and reward-model baselines. Independent evaluation success of Progress + Relation, Relation-loss-off, Sparse, and Robometer on LIBERO-10 Tasks 1, 3, 7, and 9. Lines show means across five environment seeds; shaded bands indicate $\pm 1.96\,\mathrm{SEM}$ . The horizontal axis shows environment interaction steps.

Figure 8: A held-out example of progress-scale collapse. Predicted progress on a LIBERO-10 Task 7 trajectory, shown on a common, unnormalized scale. Each curve connects 25 sampled predictions with straight lines: 16 event-comparison points (filled circles), eight additional trajectory-diagnostic points (open squares), and one reference point (diamond). The dashed line marks the reference time, and shading indicates the progress-loss comparison interval. Only filled circles enter the reported decrease counts; the five open squares within the shaded interval are excluded.

Figure 4	Suite	Task ID	Instruction
Task 1	LIBERO-Object	Task02	pick up the salad dressing and place it in the basket
Task 2	LIBERO-Goal	Task06	put the cream cheese in the bowl
Task 3	LIBERO-Goal	Task09	put the wine bottle on the rack
Task 4	LIBERO-Object	Task08	pick up the chocolate pudding and place it in the basket
Task 5	LIBERO-Object	Task00	pick up the alphabet soup and place it in the basket
Task 6	LIBERO-Goal	Task05	push the plate to the front of the stove

Appendix

Table 17: Tasks shown in Figure 4 . Task IDs follow Figures 9 – 12 ; on each task, Event2Reward improves final success over the strongest baseline.

Figure 9: Independent evaluation on all ten LIBERO-Goal tasks. Task IDs follow the original suite ordering. The horizontal axis shows training environment steps. Curves extend to 240k steps; quantitative comparisons use 200k. Lines and shaded bands show the mean and

\pm 1.96\,\mathrm{SEM}

across five environment seeds.

Figure 10: Independent evaluation on all ten LIBERO-Object tasks. Task IDs follow the original suite ordering. The horizontal axis shows training environment steps. Curves extend to 240k steps; quantitative comparisons use 200k. Lines and shaded bands show the mean and

\pm 1.96\,\mathrm{SEM}

across five environment seeds.

Figure 11: Independent evaluation on all ten LIBERO-Spatial tasks. Task IDs follow the original suite ordering. The horizontal axis shows training environment steps. Curves extend to 240k steps; quantitative comparisons use 200k. Lines and shaded bands show the mean and

\pm 1.96\,\mathrm{SEM}

across five environment seeds.

Figure 12: Independent evaluation on all ten LIBERO-10 tasks. Task IDs follow the original suite ordering. The horizontal axis shows training environment steps within the 1.2M-step budget; each task is plotted over its fixed common observation interval without extrapolation. Lines and shaded bands show the mean and

\pm 1.96\,\mathrm{SEM}

across five environment seeds.

Figure 13: Rollout joint-training ablation: independent evaluation. Each task compares Event2Reward with Event2Reward w/o rollout joint training using 200 or 40 adaptation trajectories. Lines show means across five environment seeds and bands show

\pm 1.96\,\mathrm{SEM}

, clipped to

[0,1]

. Curves extend to 240k training environment steps; the main quantitative comparison uses 200k. Curves are linearly interpolated only within the current runs’ common observed interval for each task, with no additional smoothing or extrapolation.

Robotic RL

Robotic reinforcement learning uses trial-and-error learning to help robots control, navigate, and manipulate objects.

See all 307 papers

Demonstration-Free Success-Probability Reward Learning for Generalist Robot Policies
Sep 27, 2026Duo Wu, Haifeng Wang, Rongwei Lu +6Robotic RL Sparse-Reward RL
Tsinghua University · Yuanxing Robotics · Pengcheng Laboratory
Dream2Reward: Transition-Alignment Reward Models from Positive Demonstrations for Robotic Manipulation
Aug 19, 2026Haoyu Zhang, Zecui Zeng, Bin Wang +3Reward Design for RL Dense Reward RL
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · JD Explore Academy, China
Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons
Mar 2, 2026Anthony Liang, Yigit Korkmaz, Jiahui Zhang +14Reward Modeling
Univ. of Southern California · UT Dallas · MIT +4

Sep 27, 2026J/K move · Enter open · S save

Demonstration-Free Success-Probability Reward Learning for Generalist Robot Policies

Duo Wu, Haifeng Wang, Rongwei Lu, Jinghe Wang+5

Tsinghua University · Yuanxing Robotics · Pengcheng Laboratory

Robotic RL Sparse-Reward RL Policy Learning Reward Design for RL

Open paper AskPDF arXiv

Reinforcement learning (RL) enables generalist robot policies to improve through trial-and-error interaction, yet its effectiveness is fundamentally constrained by sparse task rewards. Existing general-purpose reward models typically alleviate this issue by learning task progress from expert demonstrations, but introduce a distribution mismatch with the mixed-quality rollouts encountered during policy optimization, making their estimates unreliable on suboptimal and failed behaviors from which the policy must learn. In this work, we introduce a demonstration-free reward learning paradigm where dense reward feedback can be learned directly from sparse task outcomes and policy experience. We theoretically show that terminal task outcomes implicitly define dense success-probability feedback at intermediate timesteps, which can be recursively learned through bootstrapping. Based on this insight, we introduce eVTA

_0

, which learns success probabilities from mixed-quality policy rollouts through temporal-difference-style bootstrapping, without expert demonstrations or intermediate annotations. We further introduce RL with Evolving Rewards (RLER), a closed-loop framework that adapts eVTA

_0

using newly collected rollouts as the policy evolves. Experiments show that eVTA

_0

provides more informative rewards than state-of-the-art reward models and achieves the best average policy performance across all LIBERO task suites under the same RL training budget, improving success rates by 5.4%-13.8% over the initial policy. In real-world manipulation, RLER further improves overall success rates by 20%-26%, with 35%-36% gains under out-of-distribution conditions. These results demonstrate the effectiveness of demonstration-free reward learning and adapting rewards as the policy evolves. Project webpage: https://duowuyms.github.io/evta0.

Abstract

Robotic RL

Explore similar work

Demonstration-Free Success-Probability Reward Learning for Generalist Robot Policies

Dream2Reward: Transition-Alignment Reward Models from Positive Demonstrations for Robotic Manipulation

Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons