Robotic manipulation with deformable objects represents a data-intensive regime in embodied learning, where shape, contact, and topology co-evolve in ways that far exceed the variability of rigids. Although simulation promises relief from the cost of real-world data acquisition, prevailing sim-to-real pipelines remain rooted in rigid-body abstractions, producing mismatched geometry, fragile soft dynamics, and motion primitives poorly suited for cloth interaction. We posit that simulation fails not for being synthetic, but for being ungrounded. To address this, we introduce SIM1, a physics-aligned real-to-sim-to-real data engine that grounds simulation in the physical world. Given limited demonstrations, the system digitizes scenes into metric-consistent twins, calibrates deformable dynamics through elastic modeling, and expands behaviors via diffusion-based trajectory generation with quality filtering. This pipeline transforms sparse observations into scaled synthetic supervision with near-demonstration fidelity. Experiments show that policies trained on purely synthetic data achieve parity with real-data baselines at a 1:15 equivalence ratio, while delivering 90% zero-shot success and 50% generalization gains in real-world deployment. These results validate physics-aligned simulation as scalable supervision for deformable manipulation and a practical pathway for data-efficient policy learning.
Figures & tables
Figure 1: SIM1 pioneers real-to-sim-to-real data generation for deformable manipulation. It constructs sim ulation data whose deployment behavior is the same one as reality, enabling zero-shot transfer and scalable performance on physical robots.
Figure 2: Framework of SIM1 . (1) Real-world objects are reconstructed into metric-accurate, textured simulation assets; (2) They are then executed within a deformation-stable simulation framework calibrated through real-to-sim behavior matching. (3) Upon physical alignment, diverse manipulation trajectories are synthesized via structured subtask decomposition and diffusion-based motion generation, and rendered with appearance randomization to produce real-equivalent synthetic training data.
Figure 3: Paradigm of deformation-stable physics simulation. (a) After naive VBD ( Chen et al., a ) updates under external forces, edge deformation is monitored and virtual elastic constraints are activated when stretch exceeds a threshold, injecting strain forces that accelerate convergence toward physically plausible cloth configurations. (b) A bidirectionally synchronized simulation infrastructure replaces identical dual-arm executions in simulation and aligns deformation behaviors through visual calibration.
Figure 4: Illustration of data collection and evaluation. (a) Real-world and simulated data collection via kinesthetic teaching and isomorphic teleoperation on Arx ACONE and Arx X5. (b) Domain settings for in-domain and out-of-domain evaluation in real-world experiments. Representative long-horizon T-shirt folding task (over 20 seconds) illustrating complex sequential manipulation capabilities.
Figure 5: Illustration of assets used in data generation. Scanned deformable assets and open-source environmental assets used in simulation (top-left). Diverse garment textures for appearance variation (top-right). Room-scale environments with randomized layouts and lighting for scene-level randomization (bottom).
Figure 6: Visualization of generated data across garments and tasks. The pink box highlights the representative T-shirt folding task used as the real-world benchmark in this work. The gray box shows additional garments and manipulation tasks generated in simulation, including folding, flipping, and flattening, illustrating the broader task coverage enabled by our framework beyond the single benchmark task.
Figure 7: In-domain and out-of-domain evaluation. Policies are trained with real data, collected simulation data, or simulator-generated data. Groups: π0.5 trained from scratch, π0.5 post-trained, and π0 post-trained. Simulated data matches real episodes under equal budgets and surpasses them when scaled, especially under domain shifts.
Figure 8: Curves of performance versus data scale. Synthetic data scaling improves performance and can surpass real-data-only training. Dashed lines denote the equivalence points where M synthetic samples match one real sample at saturation.
Figure 9: Qualitative results of zero-shot sim-to-real transfer. Top: Real-robot deployment of our representative T-shirt folding task using policies trained on synthetic data, illustrating successful folding from grasping to completion. Bottom: Deployment on highly dissimilar garments with material, shape, and size configurations absent from training and simulation, demonstrating generalization across extreme domain shifts.
Figure 10: Scene digitization and solver stability. Our pipeline produces detailed geometry suitable for simulation, whereas marker-based methods ( e.g. , AR Code) yield coarser reconstructions. Conventional solvers exhibit artifacts in rigid–soft interaction (dashed regions); our solver remains stable and closely matches real-world behavior.
Method
Pass Rate
Success Rate (%)
Average
(%)
In-domain
Spatial
Texture
Lighting
Viewpoint
(%)
Baseline
0
-
-
-
-
-
-
+ Traj. decomposition
65
0
0
0
0
0
0
+ Diff.-based generation
38
47
33
60
20
7
33
+ Deform.-stable solver
40
67 (+20)
93 (+60)
90 (+30)
82 (+62)
47 (+40)
76 (+43)
Table 1: Ablation on designs in SIM1 . Pass rate denotes the proportion of generated samples accepted by the discriminator. All designs contribute to the final performance.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Parameter
Value
Particle
Radius
0.008 m
Density
2.0 kg/m 2
StVK Elasticity
Shear modulus μ
1.0×102
Area modulus λ
1.0×102
Damping
1.5×10−6
Bending
Stiffness
8.0×10−4
Appendix
Table 2: Calibrated simulation parameters for cloth physics in SIM1 .
Task
Towel Flipping
Shorts Folding
Polo-shirt Folding †
Success rate (%)
80
93
93
Appendix
Table 3: Generalization performance on additional tasks. † : Out-of-domain experiments where policies are evaluated on entirely unseen real-world tasks.
Figure 11: Real-world deployment experiments. Representative results include T-shirt folding under both in-domain and out-of-domain settings. We further demonstrate generalization to additional garments, including polo-shirts, shorts, and towels.
Figure 12: Simulated T-shirt folding scenarios. Each row shows a distinct configuration generated by our pipeline.
Figure 13: Synthetic generated data of T-shirt folding. Each row shows a temporally sampled trajectory with randomized textures, illustrating the visual diversity generated by our pipeline.
Figure 14: Synthetic generated data of towels and shorts. Each row shows a temporally sampled trajectory with randomized textures, demonstrating the framework’s applicability to multiple garment types.
Method
Throughput (traj./day) ↑
Unit Cost (/traj.)\downarrow$
Relative Cost ↓
Real-world Collection
104
2.71
1.0×
Ours (Simulation)
710
0.10
0.037×
Appendix
Table 4: Comparison of data acquisition efficiency. All costs are estimated based on the daily expense of employing two data operators ($282).
RGB sim-to-real for deformable manipulation has remained largely unsolved without real-world fine-tuning. We present SimWeaver, which trains zero-shot RGB VLA policies on 200 simulated demonstrations per task, reaching above 80% per-task and 91% average real-world success across 5 diverse deformable tasks including plastic-bag manipulation, without teleoperation or per-task calibration. SimWeaver combines a reliable measurement-backed simulator (SimWeaver-Sim) with an extensible asset framework supporting single-image generation(SimWeaver-Asset), a deterministic topology-aware trajectory synthesizer (SimWeaver-Syn), and a sim-to-real protocol with ISP-aware photometric augmentation (SimWeaver-Real). On silk grasping, the sim-trained policy reaches 100% under visual distribution shifts where real-data baselines drop to 9-70%, at two orders of magnitude lower per-trajectory cost. We will release SimWeaver and a representative asset subset. Project page: https://simweaver.github.io/
Wenkang Hu, Haoran Wang, Yitong Li +10
1Shanghai Jiao Tong University · 2Horizon Robotics · 3Style3D Research
Scaling data volume and diversity is critical for generalizing embodied intelligence. While synthetic data generation offers a scalable alternative to expensive physical data acquisition, transferring robotic manipulation policies from simulation to the real world (sim-to-real) remains a formidable challenge due to the domain gap. This paper presents HyperSim, a holistic framework spanning from synthetic data generation to policy training and seamless real-world deployment. To systematically bridge the sim-to-real gap, HyperSim is realized through three core pillars: high-fidelity environment synthesis, adversarial trajectory generation, and sim-and-real co-training. Collectively, these modules address domain discrepancies by enhancing visual fidelity, expanding data coverage, and enforcing domain-invariant representations. We rigorously validate HyperSim through a large-scale empirical study involving 400 real-world task executions across two representative manipulation models. Assessed across three fine-grained metrics, our complete pipeline achieves remarkable sim-to-real success rates of 80% and 95% with ACT and π_{0}, respectively. Furthermore, policies trained on our adversarial trajectories exhibit significantly enhanced robustness against dynamic uncertainties, achieving a 35% higher completion rate under physical perturbations.
Junyi Dong, Haotian Luo, Ziwei Xu +11
CloudRobo Lab, Huawei Cloud Computing Technologies Co.,Ltd. · Shanghai Jiao Tong University · The University of Hong Kong
Simulation frameworks such as Isaac Sim have enabled scalable robot learning for locomotion and rigid-body manipulation; however, contact-rich simulation remains a major bottleneck for deformable object manipulation. The continuously changing geometry of soft materials, together with large numbers of vertices and contact constraints, makes it difficult to achieve high accuracy, speed, and stability required for large-scale interactive learning. We present FLASH, a GPU-native simulation framework for contact-rich deformable manipulation, built on an accurate NCP-based solver that enforces strict contact and deformation constraints while being explicitly designed for fine-grained GPU parallelism. Rather than porting conventional single-instruction-multiple-data (SIMD) solvers to GPUs, FLASH redesigns the physics engine from the ground up to leverage modern GPU architectures, including optimized collision handling and memory layouts. As a result, FLASH scales to over 3 million degrees of freedom at 30 FPS on a single RTX 5090, while accurately simulating physical interactions. Policies trained solely on FLASH-generated synthetic data in minutes achieve robust zero-shot sim-to-real transfer, which we validate on physical robots performing challenging deformable manipulation tasks such as towel folding and garment folding, without any real-world demonstration, providing a practical alternative to labor-intensive real-world data collection.
Siyuan Luo, Bingyang Zhou, Chong Zhang +9
NUS Human-Centered Robotic Lab · ETH · ShanghaiTech University +2