TUCO: Curating Simulation Demonstrations for Sim-to-Real Robot Policy Co-Training

Ning Zhu; Mengfei Zhao; Yikai Tang; Zhangyujie Sun; Peihao Li; Dongyue Ni; Jindou Jia; Jianfei Yang

cs.ROOct 4, 2026

TUCO: Curating Simulation Demonstrations for Sim-to-Real Robot Policy Co-Training

Authors: Ning Zhu, Mengfei Zhao, Yikai Tang, Zhangyujie Sun, Peihao Li, Dongyue Ni, Jindou Jia, Jianfei Yang

Organizations: Stanford University · Nanyang Technological University · AXIS Robotics · Carnegie Mellon University · Fudan University · University of California, Berkeley · Shanghai Jiao Tong University

Abstract

Simulation demonstrations can supplement scarce real-world data for robot policy co-training. However, the value of using data curation to actively select these demonstrations for sim-to-real co-training remains underexplored. Existing curation methods also lack a unified criterion for measuring trajectory-level utility and set-level coverage from closed-loop target behavior. To address these gaps, we present the first systematic study of data curation for sim-to-real robot policy co-training and propose Trajectory-level Utility and set-level Coverage Optimization (TUCO). TUCO uses influence functions to trace how each source demonstration affects target-domain scoring rollouts. Our key insight is that these effects can be decomposed into an overall contribution to target return and variation across rollouts, providing a common closed-loop basis for measuring trajectory utility and set coverage. We further propose a performance-aligned subset optimizer that combines these measures in a unified curation objective to reduce redundancy and select complementary demonstrations. Extensive experiments on RoboMimic and OmniReset establish the value of active simulation data curation for sim-to-real policy co-training and show that TUCO achieves state-of-the-art performance across single-simulator, sim-to-sim, and sim-to-real settings.

Figures & tables

Figure 1: Overview of TUCO. ( A ) Influence functions estimate source demonstration effects on target-domain scoring rollouts at a fixed base policy, yielding the influence matrix

\mathbf{X}

and return vector

\mathbf{y}

. ( B ) Each influence profile is decomposed for trajectory utility and set coverage. ( C ) TUCO filters candidates by performance alignment and iteratively selects trajectories using utility and marginal coverage. ( D ) The curated source subset is combined with target demonstrations for co-training and target-domain evaluation.

Figure 2: Single-simulator data curation on RoboMimic under demonstration filtering and selection. Results are averaged over 3 random seeds. Success rates are computed over 50 evaluation rollouts from the last 10 checkpoints (500 rollouts total).

Figure 3: Sim-to-Sim co-training with curated IsaacSim demonstrations and fixed MuJoCo target data across different source-data budgets. Results are averaged over 3 random seeds. Success rates are computed over 50 evaluation rollouts from the last 5 checkpoints (250 rollouts total).

Training-data baselines
Method / training data	Peg	StackCube	CupCake	Avg.
10 real demos + all sim data	17.5	40.0	20.0	25.8
10 real demos	15.0	35.0	0.0	16.7
Compared methods
Random	7.5	57.5	12.5	25.8
CUPID	75.0	65.0	80.0	73.3

Table 1: Success rates (%) across three real-world tasks. Each method is evaluated over 40 trials per task (120 real-robot trials in total). Avg. denotes the macro-average across tasks.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Example simulated executions of CupCake , Peg , and StackCube . Overlaid frames show successive stages of each task, with blue paths tracing gripper motion.

Figure 6: Example real-world executions of StackCube , Peg , and CupCake , shown from top to bottom. Frames progress from left to right through grasping, transfer, and final placement or insertion.

Parameter	Value
Observation history	2 steps
Prediction horizon	16 steps
Image resolution	$84\times 84$
Training crop	$76\times 76$
Visual encoder	ResNet-18 per camera
U-Net channel widths	256, 512, 1024

Appendix

Table 3: Policy and training configuration for sim-to-real co-training.

Figure 7: Influence-space visualization of StackCube subsets selected at a $10\%$ source budget. Gray points denote the source pool and colored points denote selected demonstrations by utility percentile. Ellipses summarize the PCA-projected spread of each subset.

Protocol	Task	$\lambda$	$10\%$	$25\%$	$50\%$	$75\%$	$90\%$
Filtering	Lift	$0.1$	$1.000{\pm}0.000$	$1.000{\pm}0.000$	$1.000{\pm}0.000$	$0.986{\pm}0.013$	$0.960{\pm}0.023$
		$\mathbf{0.2}$	$1.000{\pm}0.000$	$1.000{\pm}0.000$	$1.000{\pm}0.000$	$1.000{\pm}0.000$	$0.988{\pm}0.016$
		$0.5$	$1.000{\pm}0.000$	$1.000{\pm}0.000$	$1.000{\pm}0.000$	$0.988{\pm}0.009$	$1.000{\pm}0.000$
Filtering	Square	$0.1$	$0.760{\pm}0.006$	$0.710{\pm}0.024$	$0.730{\pm}0.013$	$0.534{\pm}0.029$	$0.364{\pm}0.026$
		$\mathbf{0.2}$	$0.762{\pm}0.005$	$0.732{\pm}0.018$	$0.734{\pm}0.014$	$0.528{\pm}0.021$	$0.360{\pm}0.024$
		$0.5$	$0.738{\pm}0.011$	$0.700{\pm}0.019$	$0.706{\pm}0.020$	$0.536{\pm}0.031$	$0.388{\pm}0.021$

Appendix

Table 4: Sensitivity to

\lambda

in the single-simulator setting with

\gamma=0.01

. Entries are target-domain success rates reported as mean

\pm

standard deviation. Bold parameter values denote the default configuration.

Task	$\lambda$	$5\%$	$10\%$	$15\%$	$25\%$	$50\%$	$75\%$
Peg	$0.1$	$0.572{\pm}0.044$	$0.588{\pm}0.045$	$0.634{\pm}0.052$	$0.678{\pm}0.010$	$0.709{\pm}0.040$	$0.674{\pm}0.013$
	$\mathbf{0.2}$	$0.567{\pm}0.033$	$0.614{\pm}0.026$	$0.633{\pm}0.031$	$0.716{\pm}0.023$	$0.709{\pm}0.040$	$0.674{\pm}0.013$
	$0.5$	$0.564{\pm}0.069$	$0.591{\pm}0.030$	$0.632{\pm}0.045$	$0.704{\pm}0.051$	$0.709{\pm}0.040$	$0.674{\pm}0.013$
StackCube	$0.1$	$0.373{\pm}0.036$	$0.329{\pm}0.054$	$0.387{\pm}0.031$	$0.369{\pm}0.031$	$0.429{\pm}0.038$	$0.426{\pm}0.029$
	$\mathbf{0.2}$	$0.399{\pm}0.040$	$0.423{\pm}0.044$	$0.432{\pm}0.028$	$0.421{\pm}0.021$	$0.455{\pm}0.015$	$0.426{\pm}0.029$
	$0.5$	$0.387{\pm}0.035$	$0.387{\pm}0.050$	$0.375{\pm}0.029$	$0.376{\pm}0.010$	$0.429{\pm}0.038$	$0.426{\pm}0.029$

Appendix

Table 5: Sensitivity to

\lambda

in Sim-to-Sim co-training with

\gamma=0.01

. Entries are target-domain success rates reported as mean

\pm

standard deviation. Bold parameter values denote the default configuration.

Task	$\gamma$	$5\%$	$10\%$	$15\%$	$25\%$	$50\%$	$75\%$
Peg	$\mathbf{0.01}$	$0.567{\pm}0.033$	$0.614{\pm}0.026$	$0.633{\pm}0.031$	$0.716{\pm}0.023$	$0.709{\pm}0.040$	$0.674{\pm}0.013$
	$0.1$	$0.559{\pm}0.007$	$0.596{\pm}0.044$	$0.610{\pm}0.054$	$0.659{\pm}0.036$	$0.709{\pm}0.040$	$0.674{\pm}0.013$
StackCube	$\mathbf{0.01}$	$0.399{\pm}0.040$	$0.423{\pm}0.044$	$0.432{\pm}0.028$	$0.421{\pm}0.021$	$0.455{\pm}0.015$	$0.426{\pm}0.029$
	$0.1$	$0.364{\pm}0.026$	$0.360{\pm}0.021$	$0.421{\pm}0.039$	$0.404{\pm}0.024$	$0.440{\pm}0.015$	$0.426{\pm}0.029$
CupCake	$\mathbf{0.01}$	$0.525{\pm}0.033$	$0.522{\pm}0.045$	$0.541{\pm}0.026$	$0.545{\pm}0.033$	$0.522{\pm}0.040$	$0.495{\pm}0.035$
	$0.1$	$0.475{\pm}0.035$	$0.527{\pm}0.031$	$0.523{\pm}0.034$	$0.540{\pm}0.052$	$0.522{\pm}0.040$	$0.495{\pm}0.035$

Appendix

Table 6: Sensitivity to

\gamma

in Sim-to-Sim co-training with

\lambda=0.2

. Entries are target-domain success rates reported as mean

\pm

standard deviation. Bold parameter values denote the default configuration.

Explore similar work

Sep 30, 2026cs.RO

Getting Out and Getting Back: World and Behavior Grounding in Real2Sim2Real Co-Training

Simulation can expand scarce real demonstrations for co-training, yet how world fidelity and similarity to human behavior affect policy performance remains unclear. We distinguish world grounding, which aligns simulation with the real system, and behavior grounding, which aligns simulated trajectories with human motion. We build a real2sim2real pipeline that varies these axes independently to generate data for co-training. On a dynamic dexterous pick-and-sort task, fully grounded co-training raises success from 52% to 86%; averaged across configurations, world grounding improves success by 18 percentage points and behavior grounding by 10. Deployed policies behave like a mixture of real-derived and simulation-derived policies, imitating real demonstrations in covered states and relying on simulated behavior elsewhere, which we examine through latent-space analysis. Together, these results suggest complementary roles: world grounding lets policies use simulated experience beyond real-data coverage, while behavior grounding matters mainly when world grounding is imperfect. Grounded simulation remains beneficial when co-training foundation models.

Samuel Liu, Youngsun Kim, Martin Matak +1

University of Cambridge · Industrial Next

Simulation-Based Reinforcement Learning Contextual Grounding Agent Behavior Modeling Policy Evaluation World

Ask PDF

Sep 21, 2026cs.RO

ARSTAG: An Agentic Real2Sim2Real System for Task-Specific Robot Data Generation

Adapting visuomotor policies to new manipulation tasks often requires substantial manual engineering or teleoperated data collection. Simulation can provide task-specific data at scale, but constructing the scene, designing expert behavior, and configuring data generation still require significant per-task effort. We present ARSTAG, an agentic Real2Sim2Real system that turns a single RGB image and a natural-language instruction directly into robot policy-learning data. A hierarchy of language agents constructs a task-scoped simulation scene, generates robot-feasible demonstrations, and expands the training distribution through task-consistent randomization, while a coordinator agent manages cross-stage feedback and recovery. Across seven manipulation tasks spanning grasping, placement, and stacking, the ARSTAG-generated demonstrations enable sim-to-real transfer of three visuomotor policy architectures to a dual-arm robot, with pi0.5 achieving an average real-world success rate of 74.6%. Ablations show that task-consistent randomization substantially improves robustness, and policy performance increases with generated dataset size. Project webpage: https://boweili666.github.io/ARSTAG/.

Bowei Li, Yuner Zhang, Changliu Liu

Carnegie Mellon University, Pittsburgh, PA 15213, USA

Robot-Free Demonstrations Visuomotor Policy Robot Systems Data Generation Random

Ask PDF

Jan 27, 2026cs.RO

Sim-and-Human Co-training for Data-Efficient and Scene-Generalizable Bimanual Manipulation

Real-robot demonstrations are prohibitively expensive, while simulation data and real-world human demonstrations are both scalable but each leaves a distinct gap: simulation suffers from a sim-to-real visual gap, and human data suffers from a human-to-robot embodiment gap. In this work, we identify a natural yet underexplored complementarity between these sources: simulation contributes robot-valid actions absent in human data, while human data provides real-world observations that simulation struggles to render. Building on this insight, we present SimHum, a co-training recipe that extracts kinematic priors from simulation and visual priors from human observations, then fine-tunes on a small real-robot dataset. SimHum exhibits strong scene-generalizable and data-efficient capabilities. With only 80 real-robot episodes per task, it achieves 62.5% success on held-out OOD scenes across four bimanual tabletop tasks, 53.7% higher than Real only in absolute success rate. Moreover, in a controlled data-collection study with matched collection time, SimHum improves over the best single-source pre-training baseline by 35.0% in absolute success rate. Project page: https://kaipengfang.github.io/sim-and-human/

Kaipeng Fang, Weiqing Liang, Yuyang Li +5

University of Electronic Science and Technology of China · Southwest Jiaotong University · Tongji University +1

Bimanual Manipulation Large-Scale Robot Demonstration Datasets Human-Robot Interaction Sim-To-Real Gap Domain Adaptation

Ask PDF