ComSim: Building Scalable Real-World Robot Data Generation via Compositional Simulation
Authors: Yiran Qin, Jiahua Ma, Li Kang, Wenzhan Li, Yihang Jiao, Xin Wen, Xiufeng Song, Heng Zhou, Jiwen Yu, Zhenfei Yin, Xihui Liu, Philip Torr, Yilun Du, Ruimao Zhang
Organizations: CUHK-Shenzhen · University of Oxford · Sun Yat-sen University · Shanghai Jiao Tong University · USTC · The University of Hong Kong · Harvard University
Recent advancements in foundational models, such as large language models and world models, have greatly enhanced the capabilities of robotics, enabling robots to autonomously perform complex tasks. However, acquiring large-scale, high-quality training data for robotics remains a challenge, as it often requires substantial manual effort and is limited in its coverage of diverse real-world environments. To address this, we propose a novel hybrid approach called Compositional Simulation, which combines classical simulation and neural simulation to generate accurate action-video pairs while maintaining real-world consistency. Our approach utilizes a closed-loop real-sim-real data augmentation pipeline, leveraging a small amount of real-world data to generate diverse, large-scale training datasets that cover a broader spectrum of real-world scenarios. We train a neural simulator to transform classical simulation videos into real-world representations, improving the accuracy of policy models trained in real-world environments. Through extensive experiments, we demonstrate that our method significantly reduces the sim2real domain gap, resulting in higher success rates in real-world policy model training. Our approach offers a scalable solution for generating robust training data and bridging the gap between simulated and real-world robotics.
Figures & tables
Figure 1: There are three main sources of real-world robotic data: (1) direct human collection, which yields high-quality samples but cannot scale; (2) classical simulators, which generate large datasets but suffer from appearance and physics gaps to reality; and (3) neural simulators trained on real data, which reduce these gaps but struggle with action-conditioned video generation, leading to weak action–video consistency. We introduce the concept of Compositional Simulation , a flexible and scalable approach that bridges the gap between classical simulation and real-world dynamics via compositional simulation.
Figure 2: (Left) Alignment between real-world and simulation: trajectories collected in the real world are replayed in simulation to generate paired video data for training the sim-to-real neural simulator. (Right) A DiT can be used to estimate scores conditioned on different dynamics, including Control Dynamics (actions) and Visual Dynamics (simulated observations). These scores can be composed during sampling to enable Dynamic Guidance methods.
Figure 3: Real World Deployment with Compositional Simulation . Large volumes of (Vsim,A) pairs are collected from the classical simulator and transformed into corresponding (Vreal,A) pairs, referred to as Pseudo Real Data . These data, together with a small amount of real-world data, are used to train policies with improved success rates and generalization.
PSNR ↑
SSIM ↑
CLIP Score ↑
LPIPS ↓
FID ↓
FVD ↓
Sim
16.443
0.4342
0.7564
0.3629
187.40
61.048
Baseline
16.849
0.5129
0.7526
0.3494
254.59
50.369
Zero-Shot
13.093
0.5487
0.7308
0.4756
219.74
163.83
Ours-CD
8.4640
0.1486
0.7216
0.8130
434.44
239.13
Ours-VD
18.153
0.5916
0.7884
0.2813
153.12
22.311
Ours-Full
19.577
0.6484
0.8102
0.2647
147.90
15.765
Table 1: Comparison of the realism quality of generated videos across different methods.
Figure 4: Visual comparison of generated results across four different tasks.
Real World Task
Spatial Distribution
10 Real
20 Real
200 Sim Pretrain + 10 Real
Shake Bottle
In Domain
9/30
17/30
12/30
OOD
0/30
1/30
0/30
Stack Blocks Two
In Domain
5/30
13/30
8/30
OOD
0/30
0/30
0/30
Move Playing-Card Away
In Domain
12/30
24/30
15/30
OOD
0/30
3/30
2/30
Table 2: Quantitative Results of Our Real-world Experiments. The compared methods include DP trained under six data-mixture regimes: 10 Real , 20 Real , 200 Sim Pretrain + 10 Real , 10 Real + 200 Sim , 10 Real + 200 Pseudo Real , and 200 Pseudo Real . “OOD” abbreviates out-of-distribution.
Figure 5: Visualization of DP performance on Move Playing-Card Away . Top two rows: objects lie initially within the region predefined in collected real-world demonstrations (in-domain spatial distribution). Middle two rows: initial positions are outside the region (out-of-domain spatial distribution). Bottom four rows: introduce a colored background with varying levels of object clustering. Policies shown are trained under 20 Real and 10 Real + 200 Pseudo Real , respectively.
Real World Task
10 Real
20 Real
200 Sim Pretrain + 10 Real
Shake Bottle
0/30
0/30
0/30
Move Playing-Card Away
1/30
2/30
1/30
Real World Task
10 Real + 200 Sim
10 Real + 200 Pseudo Real
200 Pseudo Real (Zero Shot)
Shake Bottle
0/30
15/30
9/30
Move Playing-Card Away
0/30
21/30
11/30
Table 3: Quantitative evaluation of DP new-object generalization across six data mixtures.
Figure 6: Generalization visualization of DP on Shake Bottle under out-of-domain object distributions. Top: policy trained with 20 Real . Bottom: policy trained with 10 Real + 200 Pseudo Real .
Figure 7: Definition of in-domain and out-of-domain spatial distributions in different tasks. Both terms refer exclusively to the initial position of objects before being manipulated. Positions are labeled in-domain if and only if they appear in the collected real-world demonstrations; all others are deemed out-of-domain.
Figure 8: Real-world evaluation platform.
Figure 9: Generalization visualization of DP on new objects. The top two rows are corresponds to Move Playing-Card Away , and the bottom two rows are corresponds to Shake Bottle . respectively.
Figure 10: Real2Sim alignment on Move Playing-Card Away . From top to bottom: Nongfu Spring Oriental Leaf Tea, Coca-Cola, Sprite, and Fanta.
Figure 11: Real2Sim alignment on additional tasks. From top to bottom: Ranking Blocks RGB , Stack Blocks Three and Stack Blocks Two .
Figure 12: Sim2Real visualization on various tasks. From top to bottom: Adjust Bottle , Moving PlayingCard Away , Stack Blocks Three and Ranking Blocks RGB .
Scaling data volume and diversity is critical for generalizing embodied intelligence. While synthetic data generation offers a scalable alternative to expensive physical data acquisition, transferring robotic manipulation policies from simulation to the real world (sim-to-real) remains a formidable challenge due to the domain gap. This paper presents HyperSim, a holistic framework spanning from synthetic data generation to policy training and seamless real-world deployment. To systematically bridge the sim-to-real gap, HyperSim is realized through three core pillars: high-fidelity environment synthesis, adversarial trajectory generation, and sim-and-real co-training. Collectively, these modules address domain discrepancies by enhancing visual fidelity, expanding data coverage, and enforcing domain-invariant representations. We rigorously validate HyperSim through a large-scale empirical study involving 400 real-world task executions across two representative manipulation models. Assessed across three fine-grained metrics, our complete pipeline achieves remarkable sim-to-real success rates of 80% and 95% with ACT and π_{0}, respectively. Furthermore, policies trained on our adversarial trajectories exhibit significantly enhanced robustness against dynamic uncertainties, achieving a 35% higher completion rate under physical perturbations.
Junyi Dong, Haotian Luo, Ziwei Xu +11
CloudRobo Lab, Huawei Cloud Computing Technologies Co.,Ltd. · Shanghai Jiao Tong University · The University of Hong Kong
Sim-and-real co-training is an effective way to scale robot learning by augmenting limited real-world demonstrations with simulation data. However, existing approaches typically treat simulators as fixed data sources, relying on uniform generation, offline retrieval, or manual parameter alignment by domain experts. We formulate Active Generative Simulation: optimizing a simulator's data-generation distribution to maximize downstream real-world performance. To solve this problem, we propose MetaGenSim, a meta-learning framework that trains an auxiliary rater network via meta-gradients to automatically discover high-utility data-generation distributions within a user-specified simulation parameter space, without manual parameter alignment. Across simulation and real-world manipulation tasks, MetaGenSim outperforms uniform generation and retrieval baselines, achieves a 150% relative improvement in real-world full-task success over uniform generation, and matches expert manual alignment. We further show that distributions curated with a lightweight proxy policy transfer to VLA fine-tuning, and that MetaGenSim can exploit partial alignment through atomic skill stitching and object-receptacle composition when exact sim-and-real task alignment is unavailable. Project website: https://meta-gensim.github.io
Simulation enables large-scale, low-cost robot data generation, but policies trained in simulation often fail to transfer to the real world due to the sim-to-real visual discrepancies. Existing approaches often rely on intermediate representations, which can discard rich semantic information or require additional perception modules at deployment. We address this visual sim-to-real gap with RoboRender, a framework that converts simulated trajectories into photorealistic RGB videos for policy learning. RoboRender trains a robot-oriented video generation model conditioned on simulated depth videos, language instructions, and robot RGB mask videos, preserving simulator geometry, robot motion, and action labels while synthesizing realistic textures, backgrounds, and distractors. The generated RGB videos are paired with simulator-provided states and actions to train policies for zero-shot real-world deployment. On robot video test sets, our video model outperforms depth-conditioned video generation baselines in generation quality. In real-world experiments across pick-and-place, articulated-object manipulation, and mobile manipulation tasks, policies trained on RoboRender-generated data achieve a 71% average success rate, outperforming raw simulation renderings and conventional visual domain randomization by approximately 7.1x and 3.6x, respectively. We further show that policy performance improves with more generated videos per simulation trajectory, increasing opening-task success by 65 percentage points. These results demonstrate that generative video rendering mitigates the visual sim-to-real gap for zero-shot policy transfer. Project website: https://robo-render.github.io/.