TUCO: Curating Simulation Demonstrations for Sim-to-Real Robot Policy Co-Training

Ning Zhu; Mengfei Zhao; Yikai Tang; Zhangyujie Sun; Peihao Li; Dongyue Ni; Jindou Jia; Jianfei Yang

cs.ROOct 4, 2026

TUCO: Curating Simulation Demonstrations for Sim-to-Real Robot Policy Co-Training

Authors: Ning Zhu, Mengfei Zhao, Yikai Tang, Zhangyujie Sun, Peihao Li, Dongyue Ni, Jindou Jia, Jianfei Yang

Organizations: Stanford University · Nanyang Technological University · AXIS Robotics · Carnegie Mellon University · Fudan University · University of California, Berkeley · Shanghai Jiao Tong University

Abstract

Simulation demonstrations can supplement scarce real-world data for robot policy co-training. However, the value of using data curation to actively select these demonstrations for sim-to-real co-training remains underexplored. Existing curation methods also lack a unified criterion for measuring trajectory-level utility and set-level coverage from closed-loop target behavior. To address these gaps, we present the first systematic study of data curation for sim-to-real robot policy co-training and propose Trajectory-level Utility and set-level Coverage Optimization (TUCO). TUCO uses influence functions to trace how each source demonstration affects target-domain scoring rollouts. Our key insight is that these effects can be decomposed into an overall contribution to target return and variation across rollouts, providing a common closed-loop basis for measuring trajectory utility and set coverage. We further propose a performance-aligned subset optimizer that combines these measures in a unified curation objective to reduce redundancy and select complementary demonstrations. Extensive experiments on RoboMimic and OmniReset establish the value of active simulation data curation for sim-to-real policy co-training and show that TUCO achieves state-of-the-art performance across single-simulator, sim-to-sim, and sim-to-real settings.

Figures & tables

Figure 1: Overview of TUCO. ( A ) Influence functions estimate source demonstration effects on target-domain scoring rollouts at a fixed base policy, yielding the influence matrix

\mathbf{X}

and return vector

\mathbf{y}

. ( B ) Each influence profile is decomposed for trajectory utility and set coverage. ( C ) TUCO filters candidates by performance alignment and iteratively selects trajectories using utility and marginal coverage. ( D ) The curated source subset is combined with target demonstrations for co-training and target-domain evaluation.

Figure 2: Single-simulator data curation on RoboMimic under demonstration filtering and selection. Results are averaged over 3 random seeds. Success rates are computed over 50 evaluation rollouts from the last 10 checkpoints (500 rollouts total).

Figure 3: Sim-to-Sim co-training with curated IsaacSim demonstrations and fixed MuJoCo target data across different source-data budgets. Results are averaged over 3 random seeds. Success rates are computed over 50 evaluation rollouts from the last 5 checkpoints (250 rollouts total).

Training-data baselines
Method / training data	Peg	StackCube	CupCake	Avg.
10 real demos + all sim data	17.5	40.0	20.0	25.8
10 real demos	15.0	35.0	0.0	16.7
Compared methods
Random	7.5	57.5	12.5	25.8
CUPID	75.0	65.0	80.0	73.3

Table 1: Success rates (%) across three real-world tasks. Each method is evaluated over 40 trials per task (120 real-robot trials in total). Avg. denotes the macro-average across tasks.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Example simulated executions of CupCake , Peg , and StackCube . Overlaid frames show successive stages of each task, with blue paths tracing gripper motion.

Figure 6: Example real-world executions of StackCube , Peg , and CupCake , shown from top to bottom. Frames progress from left to right through grasping, transfer, and final placement or insertion.

Parameter	Value
Observation history	2 steps
Prediction horizon	16 steps
Image resolution	$84\times 84$
Training crop	$76\times 76$
Visual encoder	ResNet-18 per camera
U-Net channel widths	256, 512, 1024

Appendix

Table 3: Policy and training configuration for sim-to-real co-training.

Figure 7: Influence-space visualization of StackCube subsets selected at a $10\%$ source budget. Gray points denote the source pool and colored points denote selected demonstrations by utility percentile. Ellipses summarize the PCA-projected spread of each subset.

Protocol	Task	$\lambda$	$10\%$	$25\%$	$50\%$	$75\%$	$90\%$
Filtering	Lift	$0.1$	$1.000{\pm}0.000$	$1.000{\pm}0.000$	$1.000{\pm}0.000$	$0.986{\pm}0.013$	$0.960{\pm}0.023$
		$\mathbf{0.2}$	$1.000{\pm}0.000$	$1.000{\pm}0.000$	$1.000{\pm}0.000$	$1.000{\pm}0.000$	$0.988{\pm}0.016$
		$0.5$	$1.000{\pm}0.000$	$1.000{\pm}0.000$	$1.000{\pm}0.000$	$0.988{\pm}0.009$	$1.000{\pm}0.000$
Filtering	Square	$0.1$	$0.760{\pm}0.006$	$0.710{\pm}0.024$	$0.730{\pm}0.013$	$0.534{\pm}0.029$	$0.364{\pm}0.026$
		$\mathbf{0.2}$	$0.762{\pm}0.005$	$0.732{\pm}0.018$	$0.734{\pm}0.014$	$0.528{\pm}0.021$	$0.360{\pm}0.024$
		$0.5$	$0.738{\pm}0.011$	$0.700{\pm}0.019$	$0.706{\pm}0.020$	$0.536{\pm}0.031$	$0.388{\pm}0.021$

Appendix

Table 4: Sensitivity to

\lambda

in the single-simulator setting with

\gamma=0.01

. Entries are target-domain success rates reported as mean

\pm

standard deviation. Bold parameter values denote the default configuration.

Task	$\lambda$	$5\%$	$10\%$	$15\%$	$25\%$	$50\%$	$75\%$
Peg	$0.1$	$0.572{\pm}0.044$	$0.588{\pm}0.045$	$0.634{\pm}0.052$	$0.678{\pm}0.010$	$0.709{\pm}0.040$	$0.674{\pm}0.013$
	$\mathbf{0.2}$	$0.567{\pm}0.033$	$0.614{\pm}0.026$	$0.633{\pm}0.031$	$0.716{\pm}0.023$	$0.709{\pm}0.040$	$0.674{\pm}0.013$
	$0.5$	$0.564{\pm}0.069$	$0.591{\pm}0.030$	$0.632{\pm}0.045$	$0.704{\pm}0.051$	$0.709{\pm}0.040$	$0.674{\pm}0.013$
StackCube	$0.1$	$0.373{\pm}0.036$	$0.329{\pm}0.054$	$0.387{\pm}0.031$	$0.369{\pm}0.031$	$0.429{\pm}0.038$	$0.426{\pm}0.029$
	$\mathbf{0.2}$	$0.399{\pm}0.040$	$0.423{\pm}0.044$	$0.432{\pm}0.028$	$0.421{\pm}0.021$	$0.455{\pm}0.015$	$0.426{\pm}0.029$
	$0.5$	$0.387{\pm}0.035$	$0.387{\pm}0.050$	$0.375{\pm}0.029$	$0.376{\pm}0.010$	$0.429{\pm}0.038$	$0.426{\pm}0.029$

Appendix

Table 5: Sensitivity to

\lambda

in Sim-to-Sim co-training with

\gamma=0.01

. Entries are target-domain success rates reported as mean

\pm

standard deviation. Bold parameter values denote the default configuration.

Task	$\gamma$	$5\%$	$10\%$	$15\%$	$25\%$	$50\%$	$75\%$
Peg	$\mathbf{0.01}$	$0.567{\pm}0.033$	$0.614{\pm}0.026$	$0.633{\pm}0.031$	$0.716{\pm}0.023$	$0.709{\pm}0.040$	$0.674{\pm}0.013$
	$0.1$	$0.559{\pm}0.007$	$0.596{\pm}0.044$	$0.610{\pm}0.054$	$0.659{\pm}0.036$	$0.709{\pm}0.040$	$0.674{\pm}0.013$
StackCube	$\mathbf{0.01}$	$0.399{\pm}0.040$	$0.423{\pm}0.044$	$0.432{\pm}0.028$	$0.421{\pm}0.021$	$0.455{\pm}0.015$	$0.426{\pm}0.029$
	$0.1$	$0.364{\pm}0.026$	$0.360{\pm}0.021$	$0.421{\pm}0.039$	$0.404{\pm}0.024$	$0.440{\pm}0.015$	$0.426{\pm}0.029$
CupCake	$\mathbf{0.01}$	$0.525{\pm}0.033$	$0.522{\pm}0.045$	$0.541{\pm}0.026$	$0.545{\pm}0.033$	$0.522{\pm}0.040$	$0.495{\pm}0.035$
	$0.1$	$0.475{\pm}0.035$	$0.527{\pm}0.031$	$0.523{\pm}0.034$	$0.540{\pm}0.052$	$0.522{\pm}0.040$	$0.495{\pm}0.035$

Appendix

Table 6: Sensitivity to

\gamma

in Sim-to-Sim co-training with

\lambda=0.2

. Entries are target-domain success rates reported as mean

\pm

standard deviation. Bold parameter values denote the default configuration.

Explore similar work

Getting Out and Getting Back: World and Behavior Grounding in Real2Sim2Real Co-Training
Sep 30, 2026Samuel Liu, Youngsun Kim, Martin Matak +1Simulation-Based Reinforcement Learning Contextual Grounding
University of Cambridge · Industrial Next
ARSTAG: An Agentic Real2Sim2Real System for Task-Specific Robot Data Generation
Sep 21, 2026Bowei Li, Yuner Zhang, Changliu LiuRobot-Free Demonstrations Visuomotor Policy
Carnegie Mellon University, Pittsburgh, PA 15213, USA
Sim-and-Human Co-training for Data-Efficient and Scene-Generalizable Bimanual Manipulation
Jan 27, 2026Kaipeng Fang, Weiqing Liang, Yuyang Li +5Bimanual Manipulation Large-Scale Robot Demonstration Datasets
University of Electronic Science and Technology of China · Southwest Jiaotong University · Tongji University +1

Sep 30, 2026, cs.ROJ/K move · Enter open · S save

Getting Out and Getting Back: World and Behavior Grounding in Real2Sim2Real Co-Training

Samuel Liu, Youngsun Kim, Martin Matak, Gilwoo Lee

University of Cambridge · Industrial Next

Simulation-Based Reinforcement Learning Contextual Grounding Agent Behavior Modeling Policy Evaluation World

Open paper AskPDF arXiv

Simulation can expand scarce real demonstrations for co-training, yet how world fidelity and similarity to human behavior affect policy performance remains unclear. We distinguish world grounding, which aligns simulation with the real system, and behavior grounding, which aligns simulated trajectories with human motion. We build a real2sim2real pipeline that varies these axes independently to generate data for co-training. On a dynamic dexterous pick-and-sort task, fully grounded co-training raises success from 52% to 86%; averaged across configurations, world grounding improves success by 18 percentage points and behavior grounding by 10. Deployed policies behave like a mixture of real-derived and simulation-derived policies, imitating real demonstrations in covered states and relying on simulated behavior elsewhere, which we examine through latent-space analysis. Together, these results suggest complementary roles: world grounding lets policies use simulated experience beyond real-data coverage, while behavior grounding matters mainly when world grounding is imperfect. Grounded simulation remains beneficial when co-training foundation models.

Abstract

Explore similar work

Getting Out and Getting Back: World and Behavior Grounding in Real2Sim2Real Co-Training

ARSTAG: An Agentic Real2Sim2Real System for Task-Specific Robot Data Generation

Sim-and-Human Co-training for Data-Efficient and Scene-Generalizable Bimanual Manipulation