Leveraging Human-In-The-Loop Demonstrations in Reinforcement Learning for Digital Twin-Driven Robot Flexibility

Yuzhu Sun; Mien Van; Nguyen Minh Nhat; Stephen McIlvanna; Sean McLoone

cs.ROOct 8, 2026

Leveraging Human-In-The-Loop Demonstrations in Reinforcement Learning for Digital Twin-Driven Robot Flexibility

Authors: Yuzhu Sun, Mien Van, Nguyen Minh Nhat, Stephen McIlvanna, Sean McLoone

Organizations: School of Electronics, Electrical Engineering and Computer Science, Queen’s University Belfast, Northern Ireland, UK

Abstract

Growing automation makes collaborative robots work in more variable environments, increasing the need for adaptation. We propose a human-in-the-loop online training framework combining a digital twin (DT), reinforcement learning (RL), and human demonstrations. Unlike DTs used mainly to generate synthetic data before task execution, our DT is synchronized with the physical system in real time through camera feeds, allowing the virtual robot to update its observations and policy from real-world feedback. A dual actor framework integrates imitation learning (IL) without adding a direct imitation loss to the RL actor, so demonstrations can guide adaptation instead of manual reprogramming. The proposed framework is demonstrated on the Ufactory Xarm5 collaborative robot, where the robot's end-effector aims to reach the target position while avoiding obstacles. The experiments show that the framework can resume training after a change in the physical workspace and that, with a fixed set of non-optimal demonstrations, the dual actor framework achieves a much higher final success rate than two methods that add an imitation loss to the actor. The same pattern holds with real human demonstrations collected in virtual reality (VR): with demonstrations that never reach the goal, the dual actor framework reached 83-100% mean deterministic evaluation success, against 0-17% for the two imitation-loss methods.

Figures & tables

Fig. 1 : The structure of the proposed system.

Approach	Online adapt.	Real-time DT sync	Robust to non-opt. demos
Synthetic-data DT [ 3 , 6 ]	–	partial	n/a
Synchronized DT [ 5 ]	–	✓	n/a
DT + continual learning [ 4 ]	partial	✓	n/a
Prioritized demo buffer [ 40 ]	–	partial	–
BC pre-train + demo-aug. PG [ 38 ]	–	–	–
Offline-to-online RL [ 11 ]	–	–	✓

TABLE I : Positioning of the proposed framework relative to representative approaches. Online adapt.: training resumed during operation when a change in the environment is detected; partial: holds only in part; –: not provided or not evaluated; n/a: not applicable.

Fig. 2 : The Overview of digital twin software setup.

Fig. 4 : The flowchart of proposed framework.

Panel of Figure 6 — Fig. 5 : Collecting human demonstrations by VR controllers.

Demonstration set	Source	Collection	Traj.	Unique transitions
Optimal	Expert policy	Deterministic rollout	10	330
Non-optimal	Trained policy	Stochastic rollout	1	50
Human, complete	Human operator	VR controller, DT	10	433
Human, partial	Human operator	VR controller, DT	10	500

TABLE II : Provenance of the demonstration sets.

Panel of Figure 8 — Fig. 6 : Retraining with optimal demonstrations (policy-generated in Table II ). Lines: mean over six paired seeds; shaded: $\pm$ one standard deviation across seeds, truncated to [0,100]% for success. (a) Return and (b) Terminal safe-success rate over the most recent 100 training episodes.

Panel of Figure 9 — Fig. 7 : Retraining with non-optimal demonstrations (policy-generated in Table II ). Lines: mean over six paired seeds; shaded: $\pm$ one standard deviation across seeds, truncated to [0,100]% for success. (a) Return and (b) Terminal safe-success rate over the most recent 100 training episodes.

Hyperparameter		Value
Shared
	Learning rate	0.0003
	Batch size	256
	Optimizer	Adam
	Actor network	$\left(256,256\right)$
	Critic network	$\left(256,256\right)$

TABLE III : Hyperparameters setup.

Demos	Method	Converged seeds; steps ( $10^{3}$ )	Training success (%)	Evaluation success (%)	Average return	Sampled contact episodes (of 3000)
None	SAC	5/6; $118.7\pm 27.8$	$93.8\pm 7.2$	$100.0\pm 0.0$	$293.5\pm 5.9$	$422\pm 159$
Optimal	SACBC	5/6; $95.1\pm 39.5$	$97.5\pm 4.7$	$100.0\pm 0.0$	$297.4\pm 4.4$	$671\pm 166$
	SACBC + Q-Filter	5/6; $114.9\pm 27.1$	$94.0\pm 8.0$	$83.3\pm 40.8$	$295.6\pm 5.2$	$539\pm 126$
	DA-SACBC (asym.)	6/6; $123.3\pm 23.3$	$97.7\pm 2.5$	$100.0\pm 0.0$	$297.9\pm 1.1$	$249\pm 154$
	DA-SACBC (sym.)	5/6; $121.6\pm 21.6$	$96.3\pm 6.1$	$100.0\pm 0.0$	$296.9\pm 4.4$	$220\pm 115$
Non-optimal	SACBC	0/6; –	$7.3\pm 10.5$	$0.0\pm 0.0$	$237.2\pm 53.2$	$728\pm 255$

TABLE IV : Retraining metrics in the nominal scene (six paired seeds; mean

\pm

standard deviation across seeds). The SAC baseline uses no demonstrations and is shared by both demonstration sets. DA-SACBC (asym.) and (sym.): asymmetric and symmetric target forms of Eq. ( 8 ).

Demonstrations	Method	Converged seeds; steps ( $10^{3}$ )	Training success (%)	Evaluation success (%)	Average return	Sampled contact episodes (of 3000)
None	SAC	5/6; $118.7\pm 27.8$	$93.8\pm 7.2$	$100.0\pm 0.0$	$293.5\pm 5.9$	$422\pm 159$
Human, complete	SACBC	0/6; –	$31.7\pm 38.0$	$33.3\pm 51.6$	$285.6\pm 12.4$	$514\pm 178$
	SACBC + Q-Filter	1/6; $149.35$	$36.8\pm 36.2$	$50.0\pm 54.8$	$291.1\pm 3.9$	$425\pm 108$
	DA-SACBC (asym.)	6/6; $94.6\pm 26.3$	$99.5\pm 0.8$	$100.0\pm 0.0$	$299.2\pm 0.8$	$124\pm 32$
	DA-SACBC (sym.)	6/6; $89.2\pm 30.1$	$99.0\pm 1.7$	$100.0\pm 0.0$	$299.2\pm 0.6$	$134\pm 36$
Human, partial	SACBC	0/6; –	$8.0\pm 18.6$	$16.7\pm 40.8$	$148.9\pm 123.0$	$1303\pm 398$

TABLE V : Retraining with human VR demonstrations (mean

\pm

SD over six seeds). Columns as in Table IV .

Scene	Demos	Method	Converged seeds; steps ( $10^{3}$ )	Training success (%)	Evaluation success (%)	Average return	Sampled contact episodes (of 3000)
S1 pillar	None	SAC	0/6; –	$15.0\pm 12.1$	$0.0\pm 0.0$	$274.7\pm 11.4$	$835\pm 501$
	Complete	SACBC	0/6; –	$19.5\pm 13.9$	$50.0\pm 54.8$	$273.1\pm 10.9$	$1004\pm 430$
	Complete	DA-SACBC	2/6; $133.4\pm 20.5$	$62.0\pm 34.6$	$83.3\pm 40.8$	$284.8\pm 11.3$	$644\pm 438$
S2 box	None	SAC	1/6; $140.9$	$24.7\pm 34.7$	$16.7\pm 40.8$	$254.6\pm 28.9$	$834\pm 261$
	Complete	SACBC	4/6; $115.2\pm 11.0$	$94.2\pm 7.7$	$100.0\pm 0.0$	$297.8\pm 1.3$	$134\pm 65$
	Complete	DA-SACBC	1/6; $137.8$	$31.7\pm 37.9$	$16.7\pm 40.8$	$258.7\pm 28.5$	$785\pm 332$

TABLE VI : Retraining in three additional fixed scenes (six paired seeds; mean

\pm

SD). Columns as in Table IV ; DA-SACBC uses the asymmetric target.

Explore similar work

One Demonstration Is Enough for Real-World Robotic Reinforcement Learning
Jul 2, 2026Yuwan Liu, Hongze Yu, Song Liu +5Reinforcement Learning Robot Skill Learning
National Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institution of Automation, Chinese Academy of Sciences, Beijing, China · Beijing Academy of Artificial Intelligence, Beijing, China · PKU-PsiBot Joint Lab, Beijing, China +2
Res-HIL: Human-Guided Residual Reinforcement Learning for Sample-Efficient Dexterous Manipulation
Sep 24, 2026Mariia Iavorskaia, Christian Dietz, Sebastian Albrecht +1Contact-Rich Robotic Manipulation Human-in-the-Loop AI
Siemens AG, Research and Predevelopment, Germany. · Technical University of Munich, Germany.
RoHIL: Robust Human-in-the-Loop Robotic Reinforcement Learning Against Illumination Variations
May 19, 2026Shuoqin Zhang, Yixin Xiong, Xiru Gao +4Distribution Shift Robustness Robot Skill Learning
Chongqing University · Chengdu Anu Intelligence

Jul 2, 2026, cs.ROJ/K move · Enter open · S save

One Demonstration Is Enough for Real-World Robotic Reinforcement Learning

Yuwan Liu, Hongze Yu, Song Liu, Yuhan Wang+4

National Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institution of Automation, Chinese Academy of Sciences, Beijing, China · Beijing Academy of Artificial Intelligence, Beijing, China · PKU-PsiBot Joint Lab, Beijing, China+2

Reinforcement Learning Robot Skill Learning Contact-Rich Robotic Manipulation Robotic RL Robotic Manipulation Data-Efficient Robot Learning+1

Open paper AskPDF arXiv

Learning effective robot control policies on physical hardware is challenging due to costly data collection and the difficulty of reward specification. Prior work has incorporated demonstrations into reinforcement learning (RL), yet existing approaches either require large numbers of demonstrations or depend on continuous human intervention during training. To address these limitations, we present AutoSERL, a framework that leverages a single demonstration to fully automate the intervention process in real-world robot RL. The framework includes three complementary mechanisms to accomplish certain tasks: a sliding window intervention mechanism that continuously guides exploration to prevent local optima and unsafe deviations, a safety recovery mechanism that detects and corrects failure states via predefined trajectory recovery points, and an intervention termination criterion that automatically disables guidance once the policy can independently complete the task, preserving its exploration advantage. We evaluate AutoSERL on six contact-intensive manipulation tasks across two robot platforms, spanning insertion, hanging, and hinge-based tasks. AutoSERL consistently outperforms SERL initialized with 20 demonstrations, behavior cloning, and MILES -- a dedicated one-shot imitation learning baseline -- across all tasks while matching HIL-SERL, achieves 100% success rate on insertion tasks, and demonstrates improved robustness to positional variations, all from a single demonstration. Code and videos are available on our project website: https://autoserl.github.io/.

Abstract

Explore similar work

One Demonstration Is Enough for Real-World Robotic Reinforcement Learning

Res-HIL: Human-Guided Residual Reinforcement Learning for Sample-Efficient Dexterous Manipulation

RoHIL: Robust Human-in-the-Loop Robotic Reinforcement Learning Against Illumination Variations