Evo-RL: Iterative Robot Reinforcement Learning from Selective Experience

Kai Ye; Yifan Jin; Qianhang Luo; Hongru Zhang; Jiedong Zhang; Shuyuan Hu; Song-Yuan Wu; Jihai Xu

cs.ROOct 9, 2026

Evo-RL: Iterative Robot Reinforcement Learning from Selective Experience

Authors: Kai Ye, Yifan Jin, Qianhang Luo, Hongru Zhang, Jiedong Zhang, Shuyuan Hu, Song-Yuan Wu, Jihai Xu,

Organizations: School of AI, Shanghai Jiao Tong University · Central South University · South China Normal University · Boden AI

Abstract

Existing real-world reinforcement learning methods improve robot policies by combining human corrections with autonomous rollouts. However, accumulated experience varies in its value for policy improvement, and even successful trajectories can contain inefficient or redundant behavior, motivating more selective experience reuse. To this end, we propose Evo-RL, an iterative reinforcement learning method that combines human interventions with advantage-guided experience selection. Evo-RL uses human corrections and experience with positive estimated advantages from successful trajectories to iteratively refine the policy. To support learning from these different experience sources, we introduce RW-RL-Dataset, which spans over 1,000 hours across four robot embodiments. On a real-world high-precision manipulation task, Evo-RL improves attempt success rate over RECAP by 30.46%. Our code, model checkpoints, and dataset are publicly available.

Figures & tables

Fig. 1: Overview of the Evo-RL method. The current policy is deployed to collect demonstrations, human interventions, and autonomous rollouts. Complete trajectories are used to train a value model whose estimates guide policy refinement. The updated policy is then redeployed in the next iteration.

Fig. 2: Overview of RW-RL-Dataset, a large-scale real-world robot learning dataset spanning diverse embodiments and tasks, with hierarchical semantic annotations, synchronized multimodal observations and control signals, and over 1,000 hours of real-world interaction data.

Fig. 3: Annotation statistics of RW-RL-Dataset, summarizing task coverage, episode-level attributes, segment-level labels, and the counts and durations of annotated subtasks and key stages.

Fig. 4: Task sequences in RW-RL-Dataset: (A) bolt insertion and (B) garment folding. Each row shows one episode in chronological order from left to right.

		Successes / attempts
Budget	Method	$U_{1}$	$U_{2}$	$U_{3}$	$U_{4}$	$U_{5}$	ASR (%)	TP
2 h	SFT	0/18	0/0	0/0	0/0	0/0	0.00	0.00
2 h	SFT + Int.	5/12	1/1	0/0	0/0	0/0	46.15	7.20
4 h	SFT	5/23	4/7	1/7	0/0	0/0	27.03	12.00
4 h	SFT + Int.	7/13	5/14	3/3	2/2	0/1	51.52	20.40
8 h	SFT	9/21	9/14	8/23	8/19	4/10	43.68	50.17

TABLE I: Targeted interventions improve PiperX ASR under matched active human-control budgets, which include demonstration and intervention time. Each method uses

N=10

planned instances per unit.

Task	Method	$U_{1}$	$U_{2}$	$U_{3}$	$U_{4}$	$U_{5}$	ASR (%)	TP
		Successes / attempts
SO-101 folding	SFT	20/20	17/76	16/18	16/21	16/18	55.56	75.78
SO-101 folding	Evo-RL	20/20	19/36	18/22	18/22	17/20	76.67	90.00
PiperX insertion	SFT	7/20	3/15	0/9	0/0	0/0	22.73	12.00
	SFT + Int.	9/11	9/12	8/10	6/8	5/6	78.72	47.98
	SFT + RL	10/25	8/21	5/14	3/10	1/3	36.99	33.69

TABLE II: Combining corrections with RL improves PiperX ASR and throughput beyond either component alone; a second round yields further gains. Each method uses

N=10

planned instances per PiperX unit and

N=20

per SO-101 folding stage.

Fig. 5: Visual value estimation and segment selection. The visual value function predicts normalized remaining returns to evaluate task states and guide advantage-based training-data selection. Blue and gray curves show value and advantage estimates; gold dots denote selected starts, and the dashed line marks the cutoff $\tau_{A}$ . Gold regions indicate retained segments formed by merging overlapping or adjacent 20-frame forward windows, with bordered images showing corresponding observations.

Robotic RL

Robotic reinforcement learning uses trial-and-error learning to help robots control, navigate, and manipulate objects.

See all 297 papers

Demonstration-Free Success-Probability Reward Learning for Generalist Robot Policies
Sep 27, 2026Duo Wu, Haifeng Wang, Rongwei Lu +6Robotic RL Sparse-Reward RL
Tsinghua University · Yuanxing Robotics · Pengcheng Laboratory
EvoHIL: Self-Evolving Reward and Flow-Matched Policy Optimization for Robust Human-in-the-Loop Reinforcement Learning
Aug 4, 2026Shuoqin Zhang, Tongtong Cheng, Xiru Gao +7Reward Modeling Robot Policy Learning
National Elite Institute of Engineering, Chongqing University, Chongqing 401135, China · Chengdu Anu Intelligence, Chengdu, China · College of Computer Science, Chongqing University, Chongqing 400044, China +2
REVOLVE: An Automated Closed-Loop Framework for Evolving Robot Manipulation with Minimal Human Intervention
Sep 13, 2026Hanyu Liu, Qian Li, Yizhu Ding +8Continual Robot Learning Robot Failure Recovery
Jiangsu Key Laboratory of Advanced Food Manufacturing Equipment and Technology, School of Mechanical Engineering, Jiangnan University, Wuxi 214122, China.

Sep 27, 2026J/K move · Enter open · S save

Demonstration-Free Success-Probability Reward Learning for Generalist Robot Policies

Duo Wu, Haifeng Wang, Rongwei Lu, Jinghe Wang+5

Tsinghua University · Yuanxing Robotics · Pengcheng Laboratory

Robotic RL Sparse-Reward RL Policy Learning Reward Design for RL

Open paper AskPDF arXiv

Reinforcement learning (RL) enables generalist robot policies to improve through trial-and-error interaction, yet its effectiveness is fundamentally constrained by sparse task rewards. Existing general-purpose reward models typically alleviate this issue by learning task progress from expert demonstrations, but introduce a distribution mismatch with the mixed-quality rollouts encountered during policy optimization, making their estimates unreliable on suboptimal and failed behaviors from which the policy must learn. In this work, we introduce a demonstration-free reward learning paradigm where dense reward feedback can be learned directly from sparse task outcomes and policy experience. We theoretically show that terminal task outcomes implicitly define dense success-probability feedback at intermediate timesteps, which can be recursively learned through bootstrapping. Based on this insight, we introduce eVTA

_0

, which learns success probabilities from mixed-quality policy rollouts through temporal-difference-style bootstrapping, without expert demonstrations or intermediate annotations. We further introduce RL with Evolving Rewards (RLER), a closed-loop framework that adapts eVTA

_0

using newly collected rollouts as the policy evolves. Experiments show that eVTA

_0

provides more informative rewards than state-of-the-art reward models and achieves the best average policy performance across all LIBERO task suites under the same RL training budget, improving success rates by 5.4%-13.8% over the initial policy. In real-world manipulation, RLER further improves overall success rates by 20%-26%, with 35%-36% gains under out-of-distribution conditions. These results demonstrate the effectiveness of demonstration-free reward learning and adapting rewards as the policy evolves. Project webpage: https://duowuyms.github.io/evta0.

Abstract

Robotic RL

Explore similar work

Demonstration-Free Success-Probability Reward Learning for Generalist Robot Policies

EvoHIL: Self-Evolving Reward and Flow-Matched Policy Optimization for Robust Human-in-the-Loop Reinforcement Learning

REVOLVE: An Automated Closed-Loop Framework for Evolving Robot Manipulation with Minimal Human Intervention