ComSim: Building Scalable Real-World Robot Data Generation via Compositional Simulation

Yiran Qin; Jiahua Ma; Li Kang; Wenzhan Li; Yihang Jiao; Xin Wen; Xiufeng Song; Heng Zhou

cs.ROApr 13, 2026

ComSim: Building Scalable Real-World Robot Data Generation via Compositional Simulation

Authors: Yiran Qin, Jiahua Ma, Li Kang, Wenzhan Li, Yihang Jiao, Xin Wen, Xiufeng Song, Heng Zhou,

Organizations: CUHK-Shenzhen · University of Oxford · Sun Yat-sen University · Shanghai Jiao Tong University ·

Abstract

Recent advancements in foundational models, such as large language models and world models, have greatly enhanced the capabilities of robotics, enabling robots to autonomously perform complex tasks. However, acquiring large-scale, high-quality training data for robotics remains a challenge, as it often requires substantial manual effort and is limited in its coverage of diverse real-world environments. To address this, we propose a novel hybrid approach called Compositional Simulation, which combines classical simulation and neural simulation to generate accurate action-video pairs while maintaining real-world consistency. Our approach utilizes a closed-loop real-sim-real data augmentation pipeline, leveraging a small amount of real-world data to generate diverse, large-scale training datasets that cover a broader spectrum of real-world scenarios. We train a neural simulator to transform classical simulation videos into real-world representations, improving the accuracy of policy models trained in real-world environments. Through extensive experiments, we demonstrate that our method significantly reduces the sim2real domain gap, resulting in higher success rates in real-world policy model training. Our approach offers a scalable solution for generating robust training data and bridging the gap between simulated and real-world robotics.

Figures & tables

Figure 1: There are three main sources of real-world robotic data: (1) direct human collection, which yields high-quality samples but cannot scale; (2) classical simulators, which generate large datasets but suffer from appearance and physics gaps to reality; and (3) neural simulators trained on real data, which reduce these gaps but struggle with action-conditioned video generation, leading to weak action–video consistency. We introduce the concept of Compositional Simulation , a flexible and scalable approach that bridges the gap between classical simulation and real-world dynamics via compositional simulation.

Figure 2: (Left) Alignment between real-world and simulation: trajectories collected in the real world are replayed in simulation to generate paired video data for training the sim-to-real neural simulator. (Right) A DiT can be used to estimate scores conditioned on different dynamics, including Control Dynamics (actions) and Visual Dynamics (simulated observations). These scores can be composed during sampling to enable Dynamic Guidance methods.

Figure 3: Real World Deployment with Compositional Simulation . Large volumes of $(\mathcal{V}_{\text{sim}},\mathcal{A})$ pairs are collected from the classical simulator and transformed into corresponding $(\mathcal{V}_{\text{real}},\mathcal{A})$ pairs, referred to as Pseudo Real Data . These data, together with a small amount of real-world data, are used to train policies with improved success rates and generalization.

	PSNR $\uparrow$	SSIM $\uparrow$	CLIP Score $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$
Sim	16.443	0.4342	0.7564	0.3629	187.40	61.048
Baseline	16.849	0.5129	0.7526	0.3494	254.59	50.369
Zero-Shot	13.093	0.5487	0.7308	0.4756	219.74	163.83
Ours-CD	8.4640	0.1486	0.7216	0.8130	434.44	239.13
Ours-VD	18.153	0.5916	0.7884	0.2813	153.12	22.311
Ours-Full	19.577	0.6484	0.8102	0.2647	147.90	15.765

Table 1: Comparison of the realism quality of generated videos across different methods.

Figure 4: Visual comparison of generated results across four different tasks.

Real World Task	Spatial Distribution	10 Real	20 Real	200 Sim Pretrain + 10 Real
Shake Bottle	In Domain	9/30	17/30	12/30
	OOD	0/30	1/30	0/30
Stack Blocks Two	In Domain	5/30	13/30	8/30
	OOD	0/30	0/30	0/30
Move Playing-Card Away	In Domain	12/30	24/30	15/30
	OOD	0/30	3/30	2/30

Table 2: Quantitative Results of Our Real-world Experiments. The compared methods include DP trained under six data-mixture regimes: 10 Real , 20 Real , 200 Sim Pretrain + 10 Real , 10 Real + 200 Sim , 10 Real + 200 Pseudo Real , and 200 Pseudo Real . “OOD” abbreviates out-of-distribution.

Figure 5: Visualization of DP performance on Move Playing-Card Away . Top two rows: objects lie initially within the region predefined in collected real-world demonstrations (in-domain spatial distribution). Middle two rows: initial positions are outside the region (out-of-domain spatial distribution). Bottom four rows: introduce a colored background with varying levels of object clustering. Policies shown are trained under 20 Real and 10 Real + 200 Pseudo Real , respectively.

Real World Task	10 Real	20 Real	200 Sim Pretrain + 10 Real
Shake Bottle	0/30	0/30	0/30
Move Playing-Card Away	1/30	2/30	1/30
Real World Task	10 Real + 200 Sim	10 Real + 200 Pseudo Real	200 Pseudo Real (Zero Shot)
Shake Bottle	0/30	15/30	9/30
Move Playing-Card Away	0/30	21/30	11/30

Table 3: Quantitative evaluation of DP new-object generalization across six data mixtures.

Figure 6: Generalization visualization of DP on Shake Bottle under out-of-domain object distributions. Top: policy trained with 20 Real . Bottom: policy trained with 10 Real + 200 Pseudo Real .

Figure 7: Definition of in-domain and out-of-domain spatial distributions in different tasks. Both terms refer exclusively to the initial position of objects before being manipulated. Positions are labeled in-domain if and only if they appear in the collected real-world demonstrations; all others are deemed out-of-domain.

Figure 8: Real-world evaluation platform.

Figure 9: Generalization visualization of DP on new objects. The top two rows are corresponds to Move Playing-Card Away , and the bottom two rows are corresponds to Shake Bottle . respectively.

Figure 10: Real2Sim alignment on Move Playing-Card Away . From top to bottom: Nongfu Spring Oriental Leaf Tea, Coca-Cola, Sprite, and Fanta.

Figure 11: Real2Sim alignment on additional tasks. From top to bottom: Ranking Blocks RGB , Stack Blocks Three and Stack Blocks Two .

Figure 12: Sim2Real visualization on various tasks. From top to bottom: Adjust Bottle , Moving PlayingCard Away , Stack Blocks Three and Ranking Blocks RGB .

Synthetic Data Generation for Robotics

Synthetic data generation for robotics creates simulated scenes, sensor readings, and action trajectories to train and evaluate robots.

See all 303 papers

HyperSim: A Holistic Sim-To-Real Framework For Robust Robotic Manipulation
May 26, 2026Junyi Dong, Haotian Luo, Ziwei Xu +11Synthetic-to-Real Domain Adaptation Sim-to-Real Transfer
CloudRobo Lab, Huawei Cloud Computing Technologies Co.,Ltd. · Shanghai Jiao Tong University · The University of Hong Kong
Active Generative Simulation: Meta-Learning What to Simulate for Sim-and-Real Policy Co-Training
Oct 14, 2026Liqian Ma, Vaibhav Saxena, Utkarsh Aashu Mishra +3Synthetic Data Generation for Robotics Sim-to-Real Transfer
Georgia Institute of Technology
RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real Transfer
Oct 7, 2026Huang Huang, Wensi Ai, Ziyu Chen +6Sim-to-Real Transfer Synthetic Data Generation for Robotics
Stanford University · Nvidia · Microsoft Research

May 26, 2026J/K move · Enter open · S save

HyperSim: A Holistic Sim-To-Real Framework For Robust Robotic Manipulation

Junyi Dong, Haotian Luo, Ziwei Xu, Shengwei Bian+10

CloudRobo Lab, Huawei Cloud Computing Technologies Co.,Ltd. · Shanghai Jiao Tong University · The University of Hong Kong

Synthetic-to-Real Domain Adaptation Sim-to-Real Transfer Generalization in Robotic Manipulation Synthetic Data Generation for Robotics Robotic Manipulation Robot Learning

Open paper AskPDF arXiv

Scaling data volume and diversity is critical for generalizing embodied intelligence. While synthetic data generation offers a scalable alternative to expensive physical data acquisition, transferring robotic manipulation policies from simulation to the real world (sim-to-real) remains a formidable challenge due to the domain gap. This paper presents HyperSim, a holistic framework spanning from synthetic data generation to policy training and seamless real-world deployment. To systematically bridge the sim-to-real gap, HyperSim is realized through three core pillars: high-fidelity environment synthesis, adversarial trajectory generation, and sim-and-real co-training. Collectively, these modules address domain discrepancies by enhancing visual fidelity, expanding data coverage, and enforcing domain-invariant representations. We rigorously validate HyperSim through a large-scale empirical study involving 400 real-world task executions across two representative manipulation models. Assessed across three fine-grained metrics, our complete pipeline achieves remarkable sim-to-real success rates of 80% and 95% with ACT and π_{0}, respectively. Furthermore, policies trained on our adversarial trajectories exhibit significantly enhanced robustness against dynamic uncertainties, achieving a 35% higher completion rate under physical perturbations.

Abstract

Synthetic Data Generation for Robotics

Explore similar work

HyperSim: A Holistic Sim-To-Real Framework For Robust Robotic Manipulation

Active Generative Simulation: Meta-Learning What to Simulate for Sim-and-Real Policy Co-Training

RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real Transfer