ScheduleGen: Demonstration Generation via Skill Scheduling for Multi-Arm Imitation Learning

Neel Jawale; Stephan Pleines; Soha Pouya; Yuke Zhu; Ajay Mandlekar; Caelan Garrett

cs.ROOct 29, 2026

ScheduleGen: Demonstration Generation via Skill Scheduling for Multi-Arm Imitation Learning

Authors: Neel Jawale, Stephan Pleines, Soha Pouya, Yuke Zhu, Ajay Mandlekar, Caelan Garrett

Organizations: NVIDIA · The University of Texas at Austin

Abstract

Multi-arm robot platforms, including bimanual and humanoid robots, have the capability to complete tasks with lower cycle times than single-arm systems, but they are more difficult to safely control due to the presence of multi-arm constraints. The paradigm of training generalist robot policies from massive datasets is a promising approach to address general-purpose multi-arm manipulation, but the acquisition of large multi-arm manipulation datasets remains a bottleneck. Automated demonstration generation can alleviate this burden by synthesizing new demonstrations from a handful of source demonstrations, but existing algorithms ignore multi-arm constraints and as a result often induce unsafe and unsuccessful behaviors. We address this limitation by introducing ScheduleGen, an integrated planning & scheduling algorithm for multi-arm demonstration generation. We evaluate ScheduleGen on 28 simulation task variants, including seven externally authored RoboDojo tasks and settings inaccessible to prior systems; it achieves 79.6% mean data-generation success and trains policies with 65.5% mean success. ScheduleGen policies outperform those trained on a state-of-the-art data generation baseline, and achieve 50-80% success across eight real-world variants through zero-shot sim-to-real and real-to-sim-to-real transfer. Videos and more at: https://schedulegen.github.io/

Figures & tables

Figure 1: ScheduleGen overview. Top: From a few source demonstrations, ScheduleGen adapts contact-rich skill demonstrations, plans collision-free connecting motions, and jointly schedules multiple arms to generate simulation data for multi-arm imitation learning that transfers to the real world. Bottom: ScheduleGen can handle long-horizon contact manipulation, challenging reset distributions, cluttered layouts, and multi-arm scenarios.

Figure 2: ScheduleGen methodology. (a) Contact-skill segments $\tau_{\psi}$ are adapted to the initial state and connected by planned motion $\tau_{m}$ , producing partially ordered paths for each arm. (b) Collision checking identifies conflicting path segments. (c) ScheduleGen implicitly inserts waits to form a joint schedule that preserves task order, avoids collisions, and enforces synchronization when required. It minimizes makespan with optional soft costs, like summed robot execution times. (d) Scheduled motions are replayed across parallel simulation environments, and successful rollouts are retained as demonstrations.

Figure 3: Simulation data generation and policy evaluation. Rows group tasks and embodiments; columns show data generation success, policy success, and mean steps over successful policy episodes. Methods share source demonstrations within each family. Policies use 50 evaluation episodes. Four-arm policies train separately for distributions I and II; DexMimicGen baselines are unsupported. Labels are rounded; bar heights retain the original values. Filled baseline markers denote zero; hollow markers denote unavailable steps. Unsupported four-arm baselines are omitted. Step axes use task-specific ranges.

Figure 4: Simulation task suite. Five Isaac Lab, seven RoboDojo, and three robosuite simulated task families are shown. Paired views show task execution ( left ) and completion ( right ). CubeClose is evaluated under D0 and D1.

Task	Method	DGR (%)	Policy SR (%)
CubeClose D0	DexMG	87.6	98
	ScheduleGen	84.0	100
CubeClose D1	DexMG	0.0	—
	ScheduleGen	72.3	94
Threading	DexMG	20.7	60
	ScheduleGen	24.7	60

Table 1: Robosuite experiments. ScheduleGen improves data generation success rate on three of four variants, improves or matches policy success wherever DexMimicGen (DexMG) trains a policy, and alone generates successful data on CubeClose D1.

Scheduling cost	Data generation success (%)	Policy success (%)	Mean policy steps $\downarrow$
Makespan Minimize just the schedule duration.	94.16	43.3	239.8
Makespan + continuity Minimize the longest internal pause.	93.46	43.3	240.2
Makespan + max-min velocity Maximize the slowest moving-edge velocity.	92.08	40.0	237.8
Makespan + minimum active span Minimize summed arm-active spans.	93.46	50.0	240.3

Table 2: Scheduling cost ablation. Effect of scheduling costs on data generation and policy learning. Each cost generates 500 successful demonstrations. Policy success is measured over 60 Nominal episodes per cost; mean policy steps are calculated over successful rollouts. Appendix B defines the secondary costs.

Figure 5: RoboDojo results. (a) Data generation success, reduction in mean steps relative to released demonstrations, and concurrent arm motion for 500 generated trajectories per task. (b) Policy success over 20 Nominal and 20 Perturbed episodes per task. (c) Scheduling success and mean per-trajectory step reduction for 700 fixed released trajectories (Table 4 ). Concurrency is the fraction of steps when both arms move; labels are rounded. A short baseline marker denotes zero.

Figure 6: ScheduleGen policies transfer to the real world. Top: successful sim-to-real Bowl and Dishware Stacking rollouts (Normal and Cross). Bottom: real-to-sim-to-real Cube Stacking and Trash rollouts (Normal, Cup cross, and Cup + bin cross). Each policy is evaluated over 20 real-world episodes; Table 3 reports success rates.

Experiment	Task	Sim DGR (%) $\uparrow$	Real SR (%) $\uparrow$
Sim-to-real	Bowl Stacking (Normal)	96.5	75
	Bowl Stacking (Cross-Arm)	66.7	80
	Dishware Stacking (Normal)	71.2	80
	Dishware Stacking (Cross-Arm)	68.0	75
Real-to-sim-to-real	Cube Stacking	64.6	65
	Trash (Normal)	74.9	75

Table 3: Simulation data-generation and real-world policy success. Each policy is evaluated over 20 real-world episodes without real-world fine-tuning. Each variant uses 1,000 ScheduleGen-generated simulation demonstrations. Sim-to-real uses no real sources; real-to-sim-to-real uses calibrated real sources, collected only in the Normal layout for Trash. DGR: data generation success rate in simulation. SR: real-world policy success rate.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Source layout and Cross-Arm performance. DexMimicGen receives newly collected Cross-Arm sources; ScheduleGen reuses Normal sources. The comparison reports data generation success and learned-policy success on the same two task variants. Each policy is evaluated over 50 episodes.

Figure 8: RoboDojo demonstration transfer. Insert Tubes illustrates scene calibration, skill validation through replay, and trajectory generation. The same procedure supplies the five source demonstrations used for each RoboDojo task.

Figure 9: Scheduling costs change motion timing at the same completion time. Both measures use seconds; lower is better. Summed spans can exceed completion time because arms move in parallel. Pauses and spans use a

10^{-3}

-rad joint-motion threshold.

Task	Scheduled/ attempted	Source steps	Scheduled steps	Mean reduction (%)
Swap T	98/100	276.88	183.55	33.71
Build Tower	100/100	726.78	622.51	14.34
Insert Tubes	100/100	322.39	267.66	16.99
Arrange Largest Number	100/100	583.33	523.59	9.92
Press by Number	100/100	389.21	346.87	11.14
Sort Nesting Dolls	100/100	590.13	522.82	11.20

Appendix

Table 4: Retiming released RoboDojo demonstrations. Mean step counts use the source and scheduled versions of each successfully scheduled trajectory. Paths and task order are fixed. The last column gives the mean percentage reduction per trajectory.

Component	Setting
Image encoder	DINOv3 ViT-L/16, fine-tuned
Inputs	Three RGB views and 14-D state
Output	14-D absolute action, horizon 50
DiT	32 layers; 16 heads; head dim. 64; hidden dim. 1024; dropout 0.2
Flow noise	$\mathrm{Beta}(1.5,1.0)$ ; $s=0.999$ ; 1,000 time buckets
Inference	Four flow integration steps

Appendix

Table 5: YAM visuomotor policy hyperparameters.

Robot Demonstration Generation

See all 14 papers

KnowDemo: Knowledge-Guided Robot Demonstration Generation from Human Videos
Sep 18, 2026Zhiyuan Gao, Yanxiang Zhan, Mohammad Khoshnazar +2Robot Demonstration Generation Synthetic Data Generation for Robotics
University of Bremen · Robotics Institute Germany (RIG)
DynaForge: Planning-Guided Residual Learning for Dynamic Manipulation Demonstration Generation
Sep 22, 2026Yiyang Jin, Yu Zheng, Xiao He +1Synthetic Data Generation for Robotics Robot Motion Generation
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai 200240, China. · LandSpace Technology Co.,Ltd., China
OneDemo: Unified Policy Learning and Data Generation from a Single Demonstration
Oct 13, 2026Ying Qian, Jianshu Hu, Han Fang +2Robot Policy Learning Learning from Demonstration
Stanford University · Shanghai Jiao Tong University · Tsinghua University

Sep 18, 2026J/K move · Enter open · S save

KnowDemo: Knowledge-Guided Robot Demonstration Generation from Human Videos

Zhiyuan Gao, Yanxiang Zhan, Mohammad Khoshnazar, Jeroen Schäfer+1

University of Bremen · Robotics Institute Germany (RIG)

Robot Demonstration Generation Synthetic Data Generation for Robotics Robotic Manipulation Learning from Demonstration

Open paper AskPDF arXiv

Learning robot manipulation policies typically requires substantial demonstration data, which are costly to collect on real robots. Recent methods generate robot demonstrations from human videos by adapting recovered motion and validating the resulting trajectories in simulation. However, methods centered on motion-reference adaptation can limit behavioral diversity by retaining the demonstrated contact strategies and subtask orders, while insufficient understanding of task requirements and scene relations can reduce demonstration generation efficiency by generating invalid candidates. To address these limitations, we propose KnowDemo, a framework that uses structured manipulation knowledge from human videos to generate diverse robot demonstrations for a target workspace. To distinguish task requirements from demonstration-specific choices, we develop a knowledge extraction and reasoning module based on a vision-language model (VLM) that associates object and action descriptions with inferred task conditions, demonstration references, and permissible execution variations. To translate this knowledge into executable demonstrations, we resolve the descriptions against target-scene entities and geometry to guide candidate generation and screening before motion planning and simulation. The resulting demonstrations exhibit multimodal behavior through alternative contact strategies and valid subtask orders, with structured execution labels. Experiments demonstrate additional verified execution modes beyond a reference-only configuration and improved candidate planning success through task-guided grasp sampling. To validate the generated data for policy learning, we fine-tune the pretrained

π_{0.5}

model on simulation data, achieving sim-to-real transfer across three tasks. Project page: https://zhiyuan-gao.github.io/knowdemo/

Abstract

Robot Demonstration Generation

Explore similar work

KnowDemo: Knowledge-Guided Robot Demonstration Generation from Human Videos

DynaForge: Planning-Guided Residual Learning for Dynamic Manipulation Demonstration Generation

OneDemo: Unified Policy Learning and Data Generation from a Single Demonstration