Large-scale robot datasets have facilitated the learning of a wide range of robot manipulation skills, but these datasets remain difficult to collect and scale further, owing to the intractable amount of human time, effort, and cost required. Simulation and synthetic data generation have proven to be an effective alternative to fuel this need for data, especially with the advent of recent work showing that such synthetic datasets can dramatically reduce real-world data requirements and facilitate generalization to novel scenarios unseen in real-world demonstrations. However, this paradigm has been limited to rigid-body tasks, which are easy to simulate. Deformable object manipulation encompasses a large portion of real-world manipulation and remains a crucial gap to address towards increasing adoption of the synthetic simulation data paradigm. In this paper, we introduce SoftMimicGen, an automated data generation pipeline for deformable object manipulation tasks. We introduce a suite of high-fidelity simulation environments that encompasses a wide range of deformable objects (stuffed animal, rope, tissue, towel) and manipulation behaviors (high-precision threading, dynamic whipping, folding, pick-and-place), across four robot embodiments: a single-arm manipulator, bimanual arms, a humanoid, and a surgical robot. We apply SoftMimicGen to generate datasets across the task suite, train high-performing policies from the data, and systematically analyze the data generation system. Project website: softmimicgen.github.io.
Figures & tables
Figure 1: SoftMimicGen System Pipeline. (Left) Human-teleoperated demonstrations are segmented into object-centric subtasks using manual annotations or heuristic signals, forming a library of source segments. (Right) Given a new target scene, SoftMimicGen (i) observes the current deformable state, (ii) performs non-rigid registration to establish correspondence and a warp field between the source and target geometries ( correspondence + warp field ), (iii) selects the source segment with the lowest registration cost and applies the resulting warp field to the end-effector trajectory ( warp trajectory; source end-effector trajectory in light arrows, warped trajectory in darker arrows ), and (iv) executes the warped trajectory in the target environment.
Figure 2: Simulation Tasks. SoftMimicGen is used to generate new demonstrations across 10 challenging tasks involving 4 distinct robot embodiments: (a-b) GR1 humanoid, (c-f) Franka arm, (g-h) dVRK surgical robot, and (i-j) bimanual YAM arms. These tasks demand high precision and fine-grained manipulation to be successfully executed.
Figure 3: Real-World Deployment. Example rollouts of real-world deformable manipulation tasks using policies trained on SoftMimicGen -generated datasets.
Task
Source Demo BC-RNN-GMM
Generated Demo BC-RNN-GMM
Generated Demo Diffusion Policy
Humanoid - Teddy
0.0 ± 0.0
32.0 ± 3.3
42.0 ± 2.0
Humanoid - Towel
1.3 ± 1.9
50.7 ± 1.9
56.0 ± 5.7
Franka - Rope
2.0 ± 2.0
99.3 ± 0.9
100.0 ± 0.0
Franka - Jenga
4.0 ± 3.3
89.3 ± 15.1
80.0 ± 3.3
Franka - Towel
0.0 ± 0.0
78.7 ± 6.8
70.7 ± 6.8
Franka - Rigid Cube
24.0 ± 4.0
90.7 ± 5.0
50.7 ± 6.8
Table I : Policy Performance on Source and Generated Datasets. Success rates of visuomotor policies trained on source demonstrations and on SoftMimicGen -generated datasets. Success rates reported as the maximum over three training seeds.
Task
50 Demos
250 Demos
500 Demos
750 Demos
Humanoid - Teddy
24.0 ± 0.0
26.0 ± 10.0
44.0 ± 0.0
44.0 ± 0.0
Humanoid - Towel
61.3 ± 6.8
70.7 ± 6.8
64.0 ± 5.7
64.0 ± 9.8
Franka - Rope
82.7 ± 5.0
100.0 ± 0.0
98.7 ± 1.9
93.3 ± 6.8
Franka - Jenga
53.3 ± 8.2
77.3 ± 6.8
93.3 ± 5.0
84.0 ± 15.0
Franka - Towel
81.3 ± 5.0
76.0 ± 3.3
74.7 ± 5.0
84.0 ± 3.3
Franka - Rigid Cube
42.0 ± 2.0
68.0 ± 6.5
82.7 ± 3.8
84.0 ± 5.7
Table II : Dataset Size Comparison. Success rates of visuomotor policies trained on subsamples of each task’s generated dataset. Success rates reported as the maximum over three training seeds.
Task
Real 30 Demos
Zero-shot Sim 1,000 Demos
Sim-Real Co-Train 1,000 + 30 Demos
Franka - Towel
76.6
70.0
76.6
Franka - Rope
46.7
33.3
76.6
YAM - Bag Loading
33.3
63.3
93.3
Table III : Real-World Deployment Results. Performance comparison across three training settings: real-only (30 demonstrations), zero-shot sim-to-real (1,000 simulation demonstrations), and sim–real co-training (1,000 simulation + 30 real demonstrations). Results show that large-scale simulation enables effective real-world transfer, while SoftMimicGen -generated data improves performance under limited real-world supervision.
Imitation learning is a promising approach for training humanoid robots to both walk and manipulate, but it requires a large number of demonstrations, which are time-intensive and difficult to collect via teleoperation. Existing data-generation algorithms can automatically synthesize demonstrations for manipulators, but they are ineffective on humanoids because their high-dimensional composite action spaces involve arms, legs, and torsos. We present HumanoidMimicGen, a method for generating humanoid legged loco-manipulation data. Our method adapts contact-rich whole-body skills from a handful of source demonstrations to new states, generalizing across changes in object pose. By interleaving these single- and dual-arm skills with whole-body locomotion and manipulation planning, the method generates stable, collision-free data across diverse scenes and layouts. To evaluate our approach, we introduce a new simulated loco-manipulation benchmark containing nine diverse tasks that test humanoid loco-manipulation capabilities. There, we demonstrate that HumanoidMimicGen automatically generates large datasets for imitation learning and enables a systematic study of how data generation and policy learning decisions impact model performance. We show that whole-body visuomotor policies co-trained with data generated by HumanoidMimicGen outperform those trained only on real-world data by 20%.
Learning in simulation provides a useful foundation for scaling robotic manipulation capabilities. However, this paradigm often suffers from a lack of data-generation-ready digital assets, in both scale and diversity. In this work, we present ManiTwin, an automated and efficient pipeline for generating data-generation-ready digital object twins. Our pipeline transforms a single image into simulation-ready and semantically annotated 3D asset, enabling large-scale robotic manipulation data generation. Using this pipeline, we construct ManiTwin-100K, a dataset containing 100K high-quality annotated 3D assets. Each asset is equipped with physical properties, language descriptions, functional annotations, and verified manipulation proposals. Experiments demonstrate that ManiTwin provides an efficient asset synthesis and annotation workflow, and that ManiTwin-100K offers high-quality and diverse assets for manipulation data generation, random scene synthesis, and VQA data generation, establishing a strong foundation for scalable simulation data synthesis and policy learning. Our webpage is available at https://manitwin.github.io/.
Kaixuan Wang, Tianxing Chen, Jiawei Liu +13
The University of Hong Kong · Xspark AI · Shenzhen University +7
Scaling data volume and diversity is critical for generalizing embodied intelligence. While synthetic data generation offers a scalable alternative to expensive physical data acquisition, transferring robotic manipulation policies from simulation to the real world (sim-to-real) remains a formidable challenge due to the domain gap. This paper presents HyperSim, a holistic framework spanning from synthetic data generation to policy training and seamless real-world deployment. To systematically bridge the sim-to-real gap, HyperSim is realized through three core pillars: high-fidelity environment synthesis, adversarial trajectory generation, and sim-and-real co-training. Collectively, these modules address domain discrepancies by enhancing visual fidelity, expanding data coverage, and enforcing domain-invariant representations. We rigorously validate HyperSim through a large-scale empirical study involving 400 real-world task executions across two representative manipulation models. Assessed across three fine-grained metrics, our complete pipeline achieves remarkable sim-to-real success rates of 80% and 95% with ACT and π_{0}, respectively. Furthermore, policies trained on our adversarial trajectories exhibit significantly enhanced robustness against dynamic uncertainties, achieving a 35% higher completion rate under physical perturbations.
Junyi Dong, Haotian Luo, Ziwei Xu +11
CloudRobo Lab, Huawei Cloud Computing Technologies Co.,Ltd. · Shanghai Jiao Tong University · The University of Hong Kong