RFW-S: Everyday Deformable Objects with Independent Simulation Verification
Authors: Jiarui Guo, Saining Zhang, Jiaying Wan, Jiawei Weng, Junhao Chen, Rui Zhang, Yuanbo Lyu, Zhiheng Lee, Peishuo Li, Jingrun Du, Weiqing Xiao, Weiyi Lin, Zenan Meng, Hu Wei, Lin Qu, Zongzheng Zhang, Hao Zhao, Chaojian Li
Organizations: Alibaba Group · Tsinghua University · Nanyang Technological University · Nanjing University · The Hong Kong University of Science and Technology
Large-scale robotic simulation requires diverse deformable objects with executable physical properties and realistic interaction behaviors. However, existing 3D asset collections primarily emphasize visual diversity, while scalable physical construction and verification remain challenging. We introduce RFW-S, a large-scale dataset of 83,365 simulation-verified assets spanning 485 everyday categories across five structural families: soft solids, cloth sheets, shells, garments, and bags. We develop an automated pipeline integrating reference-guided 3D generation, structure-aware physical modeling, and LLM-assisted parameter estimation and calibration. To ensure physical reliability, we introduce an independent simulation verification framework that evaluates object behaviors under interactions distinct from those used for parameter calibration. Extensive experiments demonstrate the effectiveness of independent verification and characterize deformable responses under different physical configurations. We further evaluate robotic grasping in terms of both task success and object integrity, and demonstrate the physical realism of our simulated objects through sim-to-real replay of fragile-object manipulation. RFW-S provides a scalable foundation for physically grounded simulation and embodied robotic learning.
Figures & tables
A. Scale, semantic organization, and physical scope
Resource
Reported scale / unit
Category organization
Object scope
Objaverse-XL
10M+ 3D objects
Diverse source collections
Broad visual geometry
CLOTH3D
11.3K garments; 2.1M time samples
Six garment groups
Simulated clothing
SoftGym
10 environments
Task-defined objects
Rope, cloth, fluid
ManiTwin-100K
100K 3D objects
512 categories
Rigid graspable objects
PhysXVerse
8.7K+ 3D assets
2.9K+ general categories
Rigid, deformable, articulated
Table 1: Resources and their physical evidence. Reported units and category definitions differ across resources. Panel B describes the published construction and evaluation protocol. RFW-S tests a frozen configuration beyond its calibration interaction. Citations are given in the accompanying text.
Figure 2: Reference appearances and recorded simulation. Fifteen reference appearances sample the unfiltered asset pool across its five structural families. Two recorded simulation sequences show compression and release at matched physical times within each sequence. Reference images convey intended appearance, while simulation frames show the executed representation.
Family
Types
Pool
Physical
RC
Both
Soft solids
168
34,641
21,112
34,502
21,103
Cloth sheets
89
18,347
18,308
18,097
18,064
Shells
63
12,990
11,415
12,990
11,415
Garments
94
19,386
19,055
19,386
19,055
Bags
71
14,636
13,728
14,611
13,728
Total
485
100,000
83,618
99,586
83,365
Table 2: Full-pool census. Physical verification and the runtime and behavioral contract check (RC) use the same 100,000 assets. “Both” is their configuration-matched intersection; all categories remain represented.
Figure 3: One asset from appearance to evidence. The reference, visual mesh, and physical-proxy render belong to the same sunshade-fabric asset. The last panel reports that asset’s calibration and independent compression outcomes.
Figure 4: Coverage retained after verification. (a) Counts across all 485 categories. (b) Family pass proportions. (c) Exact overlap: 83,365 pass both checks, 253 only physical verification, 16,221 only RC, and 161 neither. RC requires execution, finiteness, and behavioral contract acceptance. Counts are a full census.
Figure 5: Resource construction and final verification. Coverage-aware generation supplies geometry and reference appearances. Structural adaptation and effective parameters make assets executable. Calibration supports corrections; independent testing and the RC probe provide evidence for the resulting configuration.
Figure 6: Calibration feedback and independent testing. Diagnostics guide bounded updates through supported solver controls. The selected geometry, route, and parameters are frozen before a distinct probe. Expert judgments inform behavioral criteria; numerical validity remains a separate requirement.
Component
Model / update
Input to proposer
Temp.
Physical prior
qwen3.8-max
Reference images, semantics, parameter catalog
0.1
Production correction
qwen3.8-max
Prior, current candidate, calibration metrics and diagnosis
0.2
Local text-only controls
qwen3.8-max
Structured parameter JSON; no images or video
0.2
Numeric agent
Mechanical rules
Numerical calibration residuals
—
Table 3: Model roles in physical construction and local controls. Temperatures specify inference sampling. The local controls’ backend is specified by their transport configuration.
Study
Distinct units and design
Outcomes
Full-pool verification
100,000 assets, 485 categories
100,000 per check
Supplementary stability
2,000 bags, stability drop
2,000
Garment configurations
4 garments × 3 settings × 2 probes
24
Response matching
30 asset/target pairs × 7 methods × 4 budgets
840
Expanded verification
300 bags × 7 methods × 4 budgets
8,400
Grasping
1,000 instances from 7 base geometries; three single-trial controllers and a five-trial reference
4,000
Table 4: Evaluation populations and counting units. Counts describe reported outcomes. Each full-pool check uses the same assets; the grasping reference selects one outcome from five closure trials per instance.
Drop
Compression
Configuration
Height ratio
Speed (m/s)
Height ratio
Speed (m/s)
Shape-preserving
0.499
0.00091
0.876
0.15938
Softer material
0.470
0.03125
0.707
0.00571
Flexible cloth
0.312
0.14820
0.303
0.15665
Table 5: Garment configuration comparison. Each cell averages four garments. Height is final/initial vertical extent; speed is the final maximum particle speed, averaged across runs.
Figure 7: Deformation and residual motion under drop. (a) Matched physical times for one garment. (b,c) All four paired height and speed measurements; speed uses a log scale. Configurations match Table 5 ; Appendix C details the settings.
Population
Both pass
Cal. only
Indep. only
Neither
Missing
Total
30 targets
682
49
3
83
23
840
300 bags
8,075
325
0
0
0
8,400
Table 6: Calibration and independent-test outcomes. “Cal. only” passes calibration but fails the independent test; “Indep. only” has the reverse outcome. Counts are candidate evaluations across methods and budgets, with missing measurements kept separate.
Figure 8: A recorded correction and its independent test. Observed drop diagnostics lead to a density update. The revised candidate is frozen before compression. Values are recorded response summaries.
Controller
Cal. task
Held-out task
Intact
Damage
Fixed command
595/660 (90.2%)
245/340 (72.1%)
0/340
0.654
Slip feedback
634/660 (96.1%)
245/340 (72.1%)
0/340
0.784
Force prior (local)
600/660 (90.9%)
200/340 (58.8%)
0/340
0.478
Privileged reference
360/660 (54.5%)
200/340 (58.8%)
0/340
0.493
Table 7: Final grasping outcomes by instance split. Task success is shown for calibration and held-out instances. Intact success and median damage refer to the 340 held-out instances per method. Intact success requires task success and damage below 0.25. The last row is a privileged reference with five closure trials.
Figure 9: Task success and damage on held-out instances. Each controller evaluates 340 instances. Points give task success and median damage; horizontal segments show damage interquartile ranges. All instances exceed the damage threshold, so intact success is zero.
Figure 10: Sim-to-real replay of tofu grasping. (a) A crushing grasp breaks the tofu in MPM simulation, and replaying the same gripper closure on the physical Piper arm breaks the real tofu as well. (b) A non-damaging grasp transfers the tofu intact in both domains. In each block, simulation frames sit above the real-robot replay at grasp, lift-and-transfer, and outcome stages; receiving plates appear on the left in both domains. The matching outcomes show that simulated fragile-object behavior transfers to the physical robot.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11: Asset diversity, executable configurations, and interaction. The five structural families connect semantic identity to geometry, physical parameters, and verification evidence. Reference appearances illustrate coverage, and the simulation schematics illustrate interaction and integrity criteria.
Selected configuration, distinct probe, outcome and measurements
Appendix
Table 8: Information associated with an evaluated asset configuration.
Construction outcome
Assets
Accepted at first candidate
83,292
Accepted after correction
326
Model-limit rejection
16,317
Numerical rejection
2
Unresolved judgment
63
Appendix
Table 9: Construction outcomes across the complete asset pool. First-candidate acceptance and acceptance after correction are distinct events.
Family
First pass
Corrected
Model limit
Numerical
Unresolved
Soft solids
21,071
41
13,493
0
36
Cloth sheets
18,099
209
36
0
3
Shells
11,349
66
1,553
1
21
Garments
19,055
0
330
1
0
Bags
13,718
10
905
0
3
Appendix
Table 10: Construction outcomes by structural family. Corrected acceptances are events with a recorded initial failure and update; model-limit rejection is the recorded diagnosis.
Executed solver
Assets
Physically verified
PBD cloth
193
189
Elastic MPM
7
6
Elastoplastic MPM
100
74
Appendix
Table 11: Actual executed solver in the 300-asset routing sample.
Figure 12: Supplementary verification evidence. (a) Production outcomes over 100,000 assets. (b) The 2,000 supplementary bag runs execute stability-drop checks, with 1,998 passes. (c) Executed solver modes for 300 assets, 60 per structural family. (d) Ratings of 150 core clips, with one expert rating per clip; recovery has 75 applicable ratings. Correction counts describe observed construction events.
Quantity
Definition
Acceptance
Finite fraction
Fraction of recorded states with finite positions and velocities
1.0
Contact fraction
Fraction of recorded observations with detected contact
>0
Drop deformation
1−hmin,contact/h0
MPM [0.005,0.98] ; PBD [0,0.98]
Compression deformation
1−emin/e0 , using compression-axis extent
MPM [0.066,0.95] at commanded ratio 0.22 ; PBD [0,0.95]
Recovery
Final tail mean vertical extent / pre-contact mean vertical extent
MPM [0.45,1.16] ; PBD [0,1.16]
Penetration
Any recorded particle below the negative penetration margin
Absent
Appendix
Table 12: Production behavioral criteria. MPM/PBD bands follow the executed representation. Calibration uses drop; independent verification uses compression–release.
Garment
Probe
Shape-preserving
Softer material
Flexible cloth
000193
Drop
0.399 / 0.002
0.379 / 0.122
0.289 / 0.181
000193
Compression
0.966 / 0.620
0.323 / 0.005
0.338 / 0.160
000202
Drop
0.451 / 0.001
0.406 / 0.001
0.279 / 0.170
000202
Compression
0.988 / 0.002
0.985 / 0.002
0.256 / 0.161
002117
Drop
0.594 / 0.001
0.571 / 0.001
0.324 / 0.110
002117
Compression
0.549 / 0.016
0.524 / 0.014
0.259 / 0.087
Appendix
Table 13: All garment measurements. Each cell gives height ratio / final maximum speed (m/s).
Figure 13: All 24 executed garment comparisons. Four garments are evaluated under three configurations and two actions. Lines pair the same garment across configurations. Height ratio and final maximum particle speed are separate response measures. The configurations jointly change compliance, bending iterations, and rest-curvature support.
Figure 14: Local solver and observation diagnostics. Points show recorded tail mean particle speed. Colors and symbols indicate the complete behavioral decision, which also uses other measurements. Variants change material fields, damping, substeps, or observation duration; each point represents one case at its recorded settings, supplied with the data.
Dimension
Rated clips
Score 4 or 5
Proportion
Settling
150
148
98.7%
Deformation
150
146
97.3%
Recovery
75
72
96.0%
Contact
150
143
95.3%
Overall
150
141
94.0%
Appendix
Table 14: Core expert judgments. Each clip has one expert rating of perceived behavioral plausibility. Proportions use core clips only, with dimension-specific denominators.
Figure 15: Expert plausibility judgments and behavioral criteria. Ten experts supplied one rating per clip for 150 core clips and 40 anchors. Judgments inform representation-dependent response bands; settling remains a diagnostic. Numerical validity is checked separately; the illustrated clips explain the assessment workflow.
Method
Recovery pass
Error
Valid n
Expanded pass
Error
Fixed profile
100/120
0.0521
115
1152/1200
0.0127
One-shot prior
99/120
0.0704
119
1148/1200
0.0128
Rule-based update
98/120
0.0626
117
1151/1200
0.0125
Sobol search
97/120
0.0670
117
1156/1200
0.0121
Vid2Sid-style (text)
98/120
0.0622
115
1151/1200
0.0128
Swim2Real-style (text)
96/120
0.0734
118
1159/1200
0.0118
Appendix
Table 15: Seven-method independent evaluation. Recovery uses 30 targets and 120 runs per method; the expanded set uses 300 bags and 1,200 runs per method. Errors are means over recorded values (lower is better). All expanded errors are observed; recovery valid counts expose missingness. The two language-model controls are local text-only adaptations.
Method
Budget 1
Budget 2
Budget 4
Budget 8
Fixed profile
0.0533 (28)
0.0527 (29)
0.0446 (29)
0.0579 (29)
One-shot prior
0.0698 (30)
0.0696 (30)
0.0836 (30)
0.0582 (29)
Rule-based update
0.0659 (29)
0.0727 (29)
0.0585 (29)
0.0537 (30)
Sobol search
0.0676 (29)
0.0583 (29)
0.0663 (29)
0.0757 (30)
Vid2Sid-style (text)
0.0756 (29)
0.0709 (29)
0.0438 (28)
0.0578 (29)
Swim2Real-style (text)
0.0633 (29)
0.0884 (30)
0.0658 (29)
0.0755 (30)
Appendix
Table 16: Recovery error by allowed candidate budget. Parentheses give the number of recorded errors out of 30 scheduled targets.
Method
Budget 1
Budget 2
Budget 4
Budget 8
Fixed profile
0.0126 (300)
0.0136 (300)
0.0104 (300)
0.0141 (300)
One-shot prior
0.0138 (300)
0.0126 (300)
0.0127 (300)
0.0119 (300)
Rule-based update
0.0135 (300)
0.0122 (300)
0.0129 (300)
0.0114 (300)
Sobol search
0.0128 (300)
0.0110 (300)
0.0120 (300)
0.0125 (300)
Vid2Sid-style (text)
0.0168 (300)
0.0109 (300)
0.0116 (300)
0.0119 (300)
Swim2Real-style (text)
0.0131 (300)
0.0115 (300)
0.0123 (300)
0.0104 (300)
Appendix
Table 17: Expanded bag verification error by allowed budget. Every cell contains 300 recorded values; all runs evaluate one candidate.
Experiment
Method
Both
Cal. only
Indep. only
Neither
Missing
Recovery
Fixed profile
99
4
1
11
5
Recovery
One-shot prior
98
5
1
15
1
Recovery
Rule-based update
97
6
1
13
3
Recovery
Sobol search
97
9
0
11
3
Recovery
Vid2Sid-style (text)
98
10
0
7
5
Recovery
Swim2Real-style (text)
96
10
0
12
2
Appendix
Table 18: Cross-tabulation by method. “Both” passes both response criteria; the two “only” columns pass exactly one. Missing executions remain outside the four observed outcome cells.
Experiment
Budget
Pairs
Difference
95% interval
Recovery
1
27
-0.0102
[-0.0355, 0.0113]
Recovery
2
28
0.0227
[-0.0028, 0.0593]
Recovery
4
29
0.0196
[-0.0015, 0.0544]
Recovery
8
28
0.0052
[0.0000, 0.0117]
Filter
1
300
-0.0003
[-0.0043, 0.0037]
Filter
2
300
-0.0015
[-0.0048, 0.0018]
Appendix
Table 19: Exploratory paired differences: full numerical agent minus fixed profile. Negative values favor the numerical agent on recorded error. Intervals are unadjusted for multiple comparisons.
Figure 16: Recovery evaluation across allowed budgets. Columns show independent-pass fraction, observed behavior-summary error, and candidates evaluated; rows separate method groups. Budgets are separate runs. The 840 runs include 685 successes and 23 infrastructure failures with missing errors. Failures remain in success denominators. Whiskers are 95% percentile intervals from 5,000 asset/target bootstrap resamples with shared draws across methods and budgets. Asterisks mark local text-only language-model adaptations.
Figure 17: Expanded bag verification across allowed budgets. The 8,400 runs cover 300 bags, seven methods, and four allowed budgets; 8,075 pass independent verification. All errors are recorded and every run evaluates one candidate. Columns show pass fraction and response error; rows separate method groups. Bootstrap intervals follow Figure 16 . Each point summarizes candidate outcomes at the indicated allowed budget.
Split
Controller
N
Task
Intact
Flag
Damage
Integrity
Calibration
Fixed command
660
595
0
660
0.658
0.342
Held-out
Fixed command
340
245
0
340
0.654
0.346
All
Fixed command
1000
840
0
1000
0.654
0.346
Calibration
Slip feedback
660
634
0
660
0.950
0.050
Held-out
Slip feedback
340
245
0
340
0.784
0.216
All
Slip feedback
1000
879
0
1000
0.784
0.216
Appendix
Table 20: All application splits. Task, intact success, and damage flag are counts; damage and integrity are medians. The flag overlaps task success. “All” pools the two instance splits; each controller has the same underlying instances.
Figure 18: Drop tests: passing and non-settling cases. Rows (a, b) show passing peach and dish-cloth runs; rows (c, d) show memory-foam-cushion and rug runs that fail the recorded settling criterion. Frames are sampled at steps 1, 84, 168, 252, 336, and 420. Physical times use the effective solver step: 0.001 s in (a, c) and 0.003 s in (b, d). The first frame follows the first simulation step. The camera and crop remain fixed within each row; object scales differ across rows. The orange appearance comes from the original simulation videos.
Figure 19: Compression–release tests: passing and non-settling cases. Rows (a, b) show passing pleated-skirt and ankle-support-cushion runs; rows (c, d) show workwear and potato-chip-bag runs that fail the recorded settling criterion. All four trajectories remain finite with no recorded ground penetration; FAIL denotes residual motion after release. Compression, holding, and retraction last 196, 28, and 28 steps, respectively. Frames show steps 1, 98, 196, 224, 252, and 560, using a 0.001 s solver step in (b) and 0.003 s in (a, c, d). Each row retains its fixed camera and full original frame; object scales differ across rows.
Creating deformable assets for robot manipulation requires jointly specifying their geometry, appearance, and physical properties. This is especially challenging for deformable objects, since text and images provide limited evidence about how they deform and respond to contact, yet these responses directly affect their suitability for interaction. Automated generation therefore needs to resolve coupled physical requirements and use interaction evidence to guide construction and refinement. We present DeformSmith, a framework that enables automated generation of interactive, physically credible deformable assets from text or a single image. Through hierarchical agentic construction and a shared physics-grounded harness, it progressively builds, tests, and refines geometry, physical models, material behavior, and robot interaction until the resulting asset is ready for simulation and manipulation. Robot interaction closes the generation loop through manipulation feedback and replayable interaction data. Results show that DeformSmith generates assets with better visual quality and physical plausibility than state-of-the-art baselines, including PhysGen3D, PhysGM, and PhysX-Omni, while supporting the synthesis of data for robotic manipulation of deformable objects. Project page: https://can-lee.github.io/deformsmith-web/
Embodied AI, including vision-language-action and world-action models, must operate reliably in the physical world. Yet methods that perform well in simulation can degrade substantially on real robots, especially in long-horizon deformable-object manipulation, where policies must track changing states and execute reliable multi-stage bimanual interactions. Existing real-robot benchmarks mainly focus on short-horizon rigid-object tasks and offer limited coverage of long-horizon deformable manipulation. We introduce RoboFolDeX, a physical-world benchmark built entirely from real-robot data, with garment folding as its primary task. Since real-robot data collection is costly, RoboFolDeX studies how heterogeneous physical experience can be reused efficiently. The benchmark is organized around four research axes: leveraging human intervention and recovery data collected during deployment; transferring data across tasks, including across garment categories and from rigid to deformable-object manipulation; reusing data across scenes with changes in lighting, background, and layout; and transferring data across robotic embodiments. RoboFolDeX provides 2,000+ hours of real-robot data spanning 20+ tasks and 10+ embodiments. We also establish a fair real-robot evaluation platform for externally submitted policies, with standardized tasks, held-out physical objects, controlled initializations, and a unified execution protocol. The platform is publicly accessible at https://ai.midea.com/#/fold-challenge. We hope RoboFolDeX will serve as a unified testbed for heterogeneous real-robot data reuse and reliable long-horizon deformable manipulation.
Chenhuan Liu, Yi Xu, Feng Wu +7
Fudan University · AI Research Center, Midea Group (Shanghai) Co., Ltd. · Carnegie Mellon University
While simulation-ready deformable assets are essential for in-silico robotic manipulation tasks, existing generation frameworks typically assess physical plausibility after generation, leaving an object's simulated response unused as feedback for repairing upstream errors. We present DiagGen, an agentic framework that turns a single in-the-wild image into a simulation-ready deformable asset through a generate--simulate--diagnose--refine loop. DiagGen constructs part-aware geometry and material parameters, then uses a VLM (vision-language model)-based agent to select semantically informative regions, probe them in a physics simulator, observe material responses, and route evidence-backed repair cues to the responsible generation stage. Experiments on 40 assets show that diagnostics provides useful repair cues and can moderately improve the quality of generated deformable assets. Finally, we show that unlike assets generated from visual foundation models which may not be simulatable, DiagGen-generated deformables can be directly dropped into a high-fidelity physical simulator for the planning and simulation of contact-rich pick-and-place tasks. The project's website is https://diaggen.github.io/.
Guanxiong Chen, Yiduo Qu, Qianjun Xia +12
University of British Columbia, Vancouver, Canada. · National University of Singapore, Singapore. · University of Cambridge, Cambridge, United Kingdom. +1