ExpertGen: Scalable Sim-to-Real Expert Policy Learning from Imperfect Behavior Priors

Zifan Xu; Ran Gong; Maria Vittoria Minniti; Kausik Sivakumar; Ahmet Salih Gundogdu; Eric Rosen; Riedana Yan; Tushar Kusnur

cs.ROMar 16, 2026

ExpertGen: Scalable Sim-to-Real Expert Policy Learning from Imperfect Behavior Priors

Authors: Zifan Xu, Ran Gong, Maria Vittoria Minniti, Kausik Sivakumar, Ahmet Salih Gundogdu, Eric Rosen, Riedana Yan, Tushar Kusnur, +5 more

Organizations: Robotics and AI Institute · University of Texas at Austin · Sony AI

Abstract

Learning generalizable and robust behavior cloning policies requires large volumes of high-quality robotics data. While human demonstrations (e.g., through teleoperation) serve as the standard source for expert behaviors, acquiring such data at scale in the real world is prohibitively expensive. This paper introduces ExpertGen, a framework that automates expert policy learning in simulation to enable scalable sim-to-real transfer. ExpertGen first initializes a behavior prior using a diffusion policy trained on imperfect demonstrations, which may be synthesized by large language models or provided by humans. Reinforcement learning is then used to steer this prior toward high task success by optimizing the diffusion model's initial noise while keep original policy frozen. By keeping the pretrained diffusion policy frozen, ExpertGen regularizes exploration to remain within safe, human-like behavior manifolds, while also enabling effective learning with only sparse rewards. Empirical evaluations on challenging manipulation benchmarks demonstrate that ExpertGen reliably produces high-quality expert policies with no reward engineering. On industrial assembly tasks, ExpertGen achieves a 90.5% overall success rate, while on long-horizon manipulation tasks it attains 85% overall success, outperforming all baseline methods. The resulting policies exhibit dexterous control and remain robust across diverse initial configurations and failure states. To validate sim-to-real transfer, the learned state-based expert policies are further distilled into visuomotor policies via DAgger and successfully deployed on real robotic hardware.

Figures & tables

Figure 2: ExpertGen training pipeline: generative modeling of imperfect behavior priors using a state-based diffusion policy (Phase 1); steering diffusion policy in massively parallel simulation using FastTD3 (phase 2); visual policy distillation using DAgger from expert teachers (phase 3).

Figure 3: Illustrations of the tasks in our experiments. (A) Real-world manipulation tasks. (B) Industrial assembly tasks from AutoMate. (C) Long-horizon tasks from AnyTask .

Panel of Table 1 — Table 1: The success rates (%) of the evaluated state-based policies on AnyTask benchmark. Bold number indicates the best number across all the approaches.

Figure 4: The success rates (%) of the evaluated approaches on assets from the AutoMate benchmark. ExpertGen outperforms all other baselines with an overall success of 91.1%. By introducing x-y noise, the diffusion policy demonstrates higher success rates compared to the no x-y noise.

Methods	Lift Banana		Open Drawer		Push Pear
Methods	Open	Force	Open	Force	Open	Force
Diffusion Policy	72.5 (7.6 $\downarrow$ )	1.9 (79.9 $\downarrow$ )	78.2 (8.7 $\downarrow$ )	58.0 (28.9 $\downarrow$ )	47.1 (1.9 $\downarrow$ )	15.8 (33.1 $\downarrow$ )
Residual RL	83.8 (1.0 $\downarrow$ )	46.7 (37.5 $\downarrow$ )	96.9 (2.6 $\downarrow$ )	99.7 (0.2 $\uparrow$ )	57.1( 0.7 $\uparrow$ )	38.4 (18.1 $\downarrow$ )
ExpertGen	99.4 (0.3 $\downarrow$ )	56.6 (43.1 $\downarrow$ )	99.9 (0.0 $\downarrow$ )	99.7 (0.2 $\downarrow$ )	85.4 (1.2 $\uparrow$ )	38.7 (45.3 $\downarrow$ )

Table 2: Success rates (%) with two perturbations: random gripper opening and random external force applied to the end-effector. Number in parentheses indicate the performance drop (

\downarrow

) / increase (

\uparrow

) compared to no perturbation. Bold number indicates the best success rates.

Panel of Table 3 — Table 3: The Wasserstein-2 distance between the evaluated policies and base diffusion policies.

Observation	Method	Number of Demos.	Lift Banana	Lift Brick	Push Pear to Center	Open Drawer
Point-cloud	ExpertGen	DAgger	75.0%	–	65.0%	85.0%
Point-cloud	AnyTask [ 8 ]	20k	73.3%	–	16.7%	42.5%
RGB	ExpertGen	20k	80.0%	65.0%	45.0%	65.0%
RGB	AnyTask [ 8 ]	20k	80.0%	55.0%	35.0%	10.0%

Table 4: Real-world evaluation results of visuomotor policies. Policies are trained from either ExpertGen or AnyTask [ 8 ] scripted-policy rollouts. While AnyTask achieve reasonable performance on simple pick tasks, they struggle in contact-rich manipulation, highlighting the importance of ExpertGen ’s RL-refined teacher policies for robust sim-to-real transfer.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Success rate (%) distribution of imperfect prior diffusion policy (left) and ExpertGen (right) over different initial object configurations for Push Pear to Center .

Deficiencies	Base Policy	ExpertGen
Limited state coverage	39.2 (30.6 $\downarrow$ )	76.8 (23.2 $\downarrow$ )
Joint dynamics mismatch	31.4 (41.2 $\downarrow$ )	99.2 (0.1 $\downarrow$ )

Appendix

Table 5: Success rates under prior deficiencies. Arrows indicate performance drops compared to the setting without prior deficiency.

Figure 6: Success rates of base diffusion policies (left) and the ExpertGen policies (right) trained with 50 , 200 , 500 , and 1000 imperfect demonstrations on (a) Lift Banana , (b) Open Drawer , (c) Push Pear to Center , and (d) Put Object in Closed Drawer .

Figure 7: Success rates (%) of the evaluated state-based policies on the Stack Banana on Can task. Diffusion Policy w/ SkillMimicGen trains a diffusion policy on human demonstrations augmented with SkillMimicGen [ 7 ] , while ExpertGen w/ SkillMimicGen further refines the diffusion policy trained on SkillMimicGen data. Using human motions as behavior priors significantly improves the performance of both the diffusion policy and ExpertGen .

Hyperparameters	AnyTask	AutoMate
Generative Behavior Prior Modeling
Architecture	U-Net [ 19 ]	U-Net
Number of groups	8	8
Kernel size	5	5
Number of channels	[128, 256]	[128, 256]
Diffusion step embedding dim.	16	16

Appendix

Table 6: ExpertGen hyperparameters. The right two columns indicate the hyerparameters for AnyTask and AutoMate, respectively.

Hyperparameters	AnyTask
Expert Policy Acquisition
Receding horizon	8
Number of steps	64
Learning rate	5e-4
Learning rate schedule	fixed
Discount factor	0.995

Appendix

Table 7: ExpertGen -PPO hyperparameters.

Hyperparameters	AnyTask
Residual RL
Warm-start steps	10240
Action scale	0.02
Receding horizon	8
FastTD3
Critic learning rate	3e-4

Appendix

Table 8: Residual RL hyperparameters.

Hyperparameters	AnyTask
Warm-start steps	10240
Action scale	0.02
Receding horizon	8

Appendix

Table 9: SMP hyperparameters.

Figure 8: An overview of the selected eight tasks from AnyTask benchmark.

State	Dimension
End-effector pose	16
robot arm joint positions	9
Two object positions	6
Two object 6D rotations	12
Articulated object position	3
Articulated object 6D rotation	6

Appendix

Table 10: State condition and action definitions for state-based diffusion policy for the AnyTask benchmark.

Figure 9: Visualization of the selected nine assets from AutoMate benchmark covering three difficulty levels (easy, medium, and hard).

State	Dimension
End-effector pose	16
Held positions	3
Held 6D rotation	6
Task embedding	32
Action	Dimension
End-effector position	3

Appendix

Table 11: State condition and action definitions for state-based diffusion policy for the AutoMate benchmark.

Panel of Figure 10 — Figure 10: Scripted policy for Lift Banana

Panel of Figure 11 — Figure 11: Scripted policy for Lift Brick

Panel of Figure 12 — Figure 12: Scripted policy for Lift Peach

Panel of Figure 13 — Figure 13: Scripted policy for Open Drawer

Panel of Figure 14 — Figure 14: Scripted policy for Push Pear to Center

Panel of Figure 15 — Figure 15: Scripted policy for Stack Banana on Can

Panel of Figure 16 — Figure 16: Scripted policy for Put Object in Closed Drawer

Panel of Figure 17 — Figure 17: Scripted policy for Put Object in Closed Drawer

Panel of Figure 18 — Figure 18: Scripted policy for AutoMate disassembly tasks.

Panel of Figure 19 — Figure 19: Scripted policy for AutoMate disassembly tasks.

Explore similar work

HyperSim: A Holistic Sim-To-Real Framework For Robust Robotic Manipulation
May 26, 2026Junyi Dong, Haotian Luo, Ziwei Xu +11Sim-To-Real Reinforcement Learning Robotic Manipulation Policies
CloudRobo Lab, Huawei Cloud Computing Technologies Co.,Ltd. · Shanghai Jiao Tong University · The University of Hong Kong
Efficient Sim-to-Real Transfer of World-Action Models from Synthetic Priors
Jun 30, 2026Zixing Wang, Kausik Sivakumar, Jinghuan Shang +5Sim-To-Real Reinforcement Learning Robotic Manipulation
1Purdue University · 2Robotics and AI Institute
Genie Sim 3.0 : A High-Fidelity Comprehensive Simulation Platform for Humanoid Robot
Jan 5, 2026Chenghao Yin, Da Huang, Di Yang +16Sim-To-Real Reinforcement Learning Scalable Robot Learning

May 26, 2026, cs.ROJ/K move · Enter open · S save

HyperSim: A Holistic Sim-To-Real Framework For Robust Robotic Manipulation

Junyi Dong, Haotian Luo, Ziwei Xu, Shengwei Bian+10

CloudRobo Lab, Huawei Cloud Computing Technologies Co.,Ltd. · Shanghai Jiao Tong University · The University of Hong Kong

Sim-To-Real Reinforcement Learning Robotic Manipulation Policies Sim-To-Real Gap Robotic Manipulation Nvidia Isaac Sim Synthetic Data+1

Open paper AskPDF arXiv

Scaling data volume and diversity is critical for generalizing embodied intelligence. While synthetic data generation offers a scalable alternative to expensive physical data acquisition, transferring robotic manipulation policies from simulation to the real world (sim-to-real) remains a formidable challenge due to the domain gap. This paper presents HyperSim, a holistic framework spanning from synthetic data generation to policy training and seamless real-world deployment. To systematically bridge the sim-to-real gap, HyperSim is realized through three core pillars: high-fidelity environment synthesis, adversarial trajectory generation, and sim-and-real co-training. Collectively, these modules address domain discrepancies by enhancing visual fidelity, expanding data coverage, and enforcing domain-invariant representations. We rigorously validate HyperSim through a large-scale empirical study involving 400 real-world task executions across two representative manipulation models. Assessed across three fine-grained metrics, our complete pipeline achieves remarkable sim-to-real success rates of 80% and 95% with ACT and π_{0}, respectively. Furthermore, policies trained on our adversarial trajectories exhibit significantly enhanced robustness against dynamic uncertainties, achieving a 35% higher completion rate under physical perturbations.

Abstract

Explore similar work

HyperSim: A Holistic Sim-To-Real Framework For Robust Robotic Manipulation

Efficient Sim-to-Real Transfer of World-Action Models from Synthetic Priors

Genie Sim 3.0 : A High-Fidelity Comprehensive Simulation Platform for Humanoid Robot