SoftPAT3D: Physics-Augmented Text-to-3D Scene Generation with Deformables
Organizations: Stanford University
Abstract
We present SoftPAT3D, a physics-augmented framework for text-to-3D scene generation in which deformable objects, such as cloth, stuffed toys and ropes, are generated as physically valid, simulation-ready bodies and placed in intersection-free contact with the furniture that supports them, with one another and with themselves. Existing physics-augmented pipelines simulate rigid bodies only and cannot accommodate a generated deformable object, which text-to-3D generation returns as a geometrically invalid mesh, sometimes self-intersecting, that has no rest shape, no material model and no admissible initial state. These are the inputs a deformable simulation requires, and we construct each from information already in the pipeline. The rest shape comes from how the object is manufactured rather than from the generated physically inconsistent deformed configuration, and a material library assigns each deformable object its initial elastic parameters, which can then be optimized. The occluded support is regenerated and converted into a watertight collision mesh, each object is released from an optimizable placement above the supporting part that the prompt specifies, and the complete scene is driven to static equilibrium under one unified contact model, with completion verified by geometric checks and a vision-language model. When a scene fails the prompt, a grid search over placements corrects it, and the adjoint of a differentiable simulation of the same contact energy reaches the targets the grid does not, places several bodies jointly and recovers and optimizes material parameters. Multi-body scenes equilibrate in a single simulation, and the pipeline also runs from the text prompt alone. On all sixty-five prompts, SoftPAT3D produces no undesired interpenetration of any kind, whereas every other system does, and its scenes directly support scene editing and robotic manipulation.
Figures & tables
| prompt | rigid | soft sim. | soft equil. | soft grip | soft grad. | |
|---|---|---|---|---|---|---|
| Holodeck [ 15 ] | ✓ | – | – | – | – | – |
| PAT3D [ 1 ] | ✓ | ✓ | – | – | – | – |
| SimuScene [ 20 ] ‡ | – | ✓ | – | – | – | – |
| SceneSmith [ 21 ] | ✓ | ✓ | – | – | – | – |
| CoDimRecon [ 22 ] †‡ | – | ✓ | ✓ | ✓ | ✓ | – |
| SoftPAT3D (ours) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Holodeck | Astra | rigid | Blender | Blender ∗ | ours | |
| scenes | 32 | 29 | 34 | 34 | 32 | 34 |
| deformable present | 8/32 | 29/32 | 34/34 | 34/34 | 32/32 | 34/34 |
| soft bodies | 39 | 74 | 48 | 48 | 40 | 43 |
| crossings (per 1k edges) | 71.2 | 40.2 | 5.1 | 196.2 | 66.2 | 0.0 |
| penetration (mm) | 0.0 | 1.7 | 14.8 | 1.9 | 2.5 | 0.0 |
| in contact (%) | 10.3 | 38.5 | 1.8 | 22.8 | 25.2 | 39.9 |
| scenes | plausibility | CLIP | VQAScore | |
|---|---|---|---|---|
| rigid pipeline | 13 | 43 | 23.9 | 0.50 |
| ours, the same scenes | 13 | 90 | 28.2 | 0.74 |
| GPT-6 Astra | 11 | 81 | 24.9 | 0.47 |
| ours, the same scenes | 11 | 87 | 27.7 | 0.65 |
| condition | S | G | J | C |
|---|---|---|---|---|
| full stage, asset as rest shape | 22 | 19 | 14 | 14 (9) |
| centred placement, hull-based support check | 32 | 28 | 17 | 17 |
| + watertight collision mesh | 33 | 31 | 18 | 18 (13) |
| + placement on the part the language model names | 33 | 30 | 20 | 20 (15) |
| + its size estimate, full stage | 33 | 30 | 22 | 21 (16) |
Appendix figures & tables37 assets
Supplementary material from the paper’s appendix.
Appendix
| scene | bodies | crossings | contact (%) | blocked (%) |
|---|---|---|---|---|
| apron | 1 | 0 | 23 | 17 |
| armchair | 1 | 0 | 38 | 0 |
| bathtub2 | 1 | 0 | 55 | 7 |
| bathtub | 1 | 0 | 4 | 11 |
| bedsheet | 1 | 0 | 56 | 0 |
| bench2b | 1 | 0 | 61 | 0 |
| bodies | cross. (per 1k) | pen. (mm) | contact (%) | floating | unsup. | zero-cross. scenes | |
|---|---|---|---|---|---|---|---|
| Blender ∗ | 27 | 24.4 | 0.1 | 17.5 | 9/27 | 0/27 | 23/25 |
| ours | 27 | 0.0 | 0.0 | 52.5 | 0/27 | 0/27 | 25/25 |
| configuration | S | G | J | C |
|---|---|---|---|---|
| physics stage | 24 | 23 | 14 | 14 |
| + placement grid of Eq. 9 (the pipeline) | 24 | 23 | 19 | 19 |
| scene | prompt | outcome |
|---|---|---|
| bedfoot | A folded wool blanket lying across the foot of a neatly made bed. | complete |
| bookshelf | A knitted blanket draped over the top of a low wooden bookshelf. | complete |
| counter | A kitchen towel draped over the edge of a small kitchen island counter. | incomplete |
| drawers | A pleated skirt draped over the top of a chest of drawers. | incomplete |
| footboard | A patchwork quilt draped over the wooden footboard of a bed. | incomplete |
| ironing2 | A white dress shirt lying flat on an ironing board. | complete |
| Placement (Sec. 3.4 of the main paper) | |
| part | the planner names a surface, a part (seat, backrest, arm, rim) and a drop height; objects seen in the image keep their layout place |
| straddle | a sheet that fits inside the seat is centred on the seat’s front edge rather than on the seat |
| flush | a sheet deeper than a backed seat is pushed against the back, so the excess hangs over the front |
| throw | a throw on seating goes over the backrest unless the planner names another part |
| order | coverings first, smaller items on top; a sheet in a container is shrunk below the opening; line-mounted cloth starts vertical, top row pinned |
| lift | every release is tested for intersection and lifted until it is free |
| base | + collision mesh | + placement | full | ||||||||||||||||
| scene | S | G | J | x | S | G | J | x | S | G | J | x | S | G | J | x | |||
| apron | y | y | n | 0 | y | y | n | 0 | y | y | y | 0 | y | y | y | 0 | |||
| armchair | y | y | n | 0 | y | y | n | 0 | y | y | y | 0 | y | y | y | 0 | |||
| bathtub2 | y | y | n | 0 | y | y | n | 0 | y | y | y | 0 | y | y | y | 0 | |||
| bathtub | y | y | y | 0 | y | y | y | 0 | y | y | y | 0 | y | y | y | 0 | |||
| bedsheet | y | y | n | 0 | y | y | n | 0 | y | y | n | 0 | y | y | n | 0 | |||
| component what preceded it | effect without it | scenes |
|---|---|---|
| sewing-pattern rest shape generated asset | layers closer than , world invalid at step 0, mm displacement; the render still shows a drape | 6 of 6 |
| intersection-free closure watertightness test | nine supports fall back to convex hulls; cloth equilibrates on a block, a wedge, a tub with no interior | 9 of 32 |
| convex-cell collider single hull | Hausdorff error to the asset vs mm; the towel hangs through the chair’s open back | 1 of 1 |
| placement rules planner’s point | the sheet rests as a flat rectangle centred on the seat and passes every check | 6 of 32 |
| support de-occlusion masked crop | the generator fills the mask with cloth fused into the support; the sheet drapes over it | 32 of 33 |
| gradient release correction straight drop | obstructed targets within cm: 1 straight, 4 random search, 6 gradient | 14 targets |
| criterion | scenes (of 60) |
|---|---|
| same kind of object | 48 |
| same colour and pattern | 34 |
| A | C | B | D | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| scene | S | G | J | x | S | G | J | x | S | G | J | x | S | G | J | x | |||
| apron | y | y | n | 0 | y | y | n | 0 | y | y | y | 0 | y | y | y | 0 | |||
| armchair | y | y | n | 0 | y | y | y | 0 | y | y | n | 0 | y | y | y | 0 | |||
| bathtub2 | y | y | n | 0 | y | y | n | 0 | y | y | y | 0 | y | y | y | 0 | |||
| bathtub | y | y | y | 0 | y | y | n | 0 | y | y | y | 0 | y | y | n | 0 | |||
| bedsheet | y | y | n | 0 | y | y | n | 0 | y | y | n | 0 | y | y | n | 0 | |||
| collider | mean/max (mm) | closed |
|---|---|---|
| single convex hull | / | ✓ |
| convex cells ( ) | / | ✓ |
| raw generated mesh | – | |
| vs. finite diff. | ||
| Hessian-vector vs. finite diff. | ||
| Input: the prompt and the generated scene, that is, the reference image, masks, lifted assets and language model layout. Output: an equilibrated simulation state, or a rejection with its cause. |
|---|
| 1. Route every object: rigid support, shell, volumetric body or rod, by the keyword material library; take its material from the material library (Sec. 3.2). |
| 2. Supports: de-occlude the support’s crop and regenerate the asset, align it to the layout, close it (open-edge test, convex cells), and write its certificate (Sec. 3.4). |
| 3. Rest shapes: cut a sheet to the outline its class names, or sew the library garment pattern, at the size the language model estimates or the class prior; stuff the bodies that are stuffed (Sec. 3.2). |
| 4. Release: resolve the part the prompt names on the collider, place the rest shape above it, lifted clear of the bodies already placed; check that step zero is intersection-free (Sec. 3.4). |
| 5. Equilibrate: minimize the incremental potential with the per-object energies for up to steps of s; stop once the largest displacement over steps is under mm (Sec. 3.3). |
| 6. Checks: crossings, support, floor and asset tests on the equilibrated geometry; then the prompt check on the render (Sec. 3.5). |
| step (s) | settled | cross. | pen. (mm) | contact (%) | |
|---|---|---|---|---|---|
| 128 | 6/6 | 0 | 0.0 | 5–100 | |
| 128 | 6/6 | 0 | 0.0 | 9–100 | |
| 128 | 6/6 | 0 | 0.0 | 8–100 | |
| 128 | 6/6 | 0 | 0.0 | 4–100 | |
| 64 | 6/6 | 0 | 0.0 | 2–100 | |
| 192 | 4/4 | 0 | 0.0 | 2–90 |
| scene | deformable objects | shells | volumetric bodies | wall | result |
|---|---|---|---|---|---|
| messy bed | quilt, pillows, plush bear | 2 | 2 | 8.9 | equilibrated, passes |
| made bed | sheet and three pillows | 1 | 3 | 8.4 | equilibrated, passes |
| wardrobe | shirt and trousers on a bed | 2 | 0 | 8.0 | equilibrated, passes † |
| dinner | tablecloth and napkin on a chair | 2 | 0 | 8.6 | equilibrated, passes † |
| messy bed 2 | quilt, two pillows, five toys | 1 | 7 | 11.5 | equilibrated, passes |
| reading nook | blanket, two pillows, plush rabbit, charging cable | 1 | 3 | 6.9 | equilibrated, passes (one rod) |
| material | (Pa) | (mm) | (kg/m 3 ) | ||
|---|---|---|---|---|---|
| rope | 12 | 900 | 0.7 | ||
| cable | 2.5 | 1200 | 0.5 | ||
| cord | 3 | 1100 | 0.5 | ||
| string | 1.5 | 900 | 0.5 | ||
| hose | 8 | 1000 | 0.6 |
| single body, target past an obstruction | runs | grid | adjoint |
|---|---|---|---|
| four supports, two seeds (Table S24 ) | 8 | 2 | 4 |
| eight supports, four seeds (Table S24 ) | 32 | 20 | 24 |
| scene | episode | grip | err (mm) | max (mm) | reached | equilibrated | held | released | finite |
|---|---|---|---|---|---|---|---|---|---|
| armchair | pull | 54 | 8.3 | 44.4 | 0.99 | 18.5 | 0.0 | 0.0 | ✓ |
| bedsheet | pull | 21 | 19.3 | 75.6 | 0.99 | 10.6 | 4.4 | 0.0 | ✓ |
| bench2b | pull | 144 | 6.1 | 42.6 | 1.00 | 48.1 | 0.0 | 0.0 | ✓ |
| daybed | pull | 34 | 12.9 | 73.8 | 0.99 | 8.0 | 0.0 | 0.0 | ✓ |
| dining2 | pull | 28 | 14.1 | 92.5 | 0.99 | 16.0 | 11.1 | 4.3 | ✓ |
| dining | pull | 42 | 9.9 | 76.2 | 1.00 | 21.9 | 8.4 | 0.1 | ✓ |
| scene | carry (cm) | retained (%) | contact, settled (%) | contact, end (%) |
|---|---|---|---|---|
| bedsheet | 200 | 104 | 30 | 15 |
| bench | 59 | 100 | 69 | 37 |
| blanket | 141 | 99 | 10 | 4 |
| daybed | 147 | 100 | 6 | 4 |
| dining | 96 | 5 | 22 | 20 |
| picnic | 184 | 101 | 11 | 4 |
| intermediate | rigid | Blender, asset | Blender, ours | ours |
|---|---|---|---|---|
| image, segmentation, assets | ||||
| layout, support graph | ||||
| cloth rest shape | asset | asset | sewn sheet | sewn sheet |
| support geometry | painted | painted | de-occluded, closed | de-occluded, closed |
| release pose | layout | layout | rules (Sec. 3.7) | rules (Sec. 3.7) |
| solver | rigid IPC | Blender cloth | Blender cloth | IPC, routed |
| system | scenes | present | finished | complete | |||
|---|---|---|---|---|---|---|---|
| Holodeck | 32 | 8 | 8 | 5 | 2 | 2 | 2 |
| Astra | 29 | 28 | 28 | 14 | 15 | 19 | 7 |
| rigid | 34 | 34 | 34 | 14 | 15 | 11 | 0 |
| Blender | 34 | 34 | 22 | 11 | 11 | 0 | 0 |
| Blender ∗ | 34 | 32 | 32 | 23 | 19 | 8 | 8 |
| Blender ∗ , 40 substeps | 34 | 30 | 30 | 25 | 14 | 7 | 7 |
| Holodeck | Astra | rigid | Blender | Blender ∗ | ours | |
| scenes | 32 | 29 | 34 | 34 | 32 | 34 |
| deformable present | 8/32 | 29/32 | 34/34 | 34/34 | 32/32 | 34/34 |
| soft bodies | 39 | 74 | 48 | 48 | 40 | 43 |
| crossings (per 1k edges) | 71.2 | 40.2 | 5.1 | 196.2 | 66.2 | 0.0 |
| penetration (mm) | 0.0 | 1.7 | 14.8 | 1.9 | 2.5 | 0.0 |
| in contact (%) | 10.3 | 38.5 | 1.8 | 22.8 | 25.2 | 39.9 |
| cross/1k | pen. (mm) | contact (%) | floating | |||||
|---|---|---|---|---|---|---|---|---|
| scene | prev. | ours | prev. | ours | prev. | ours | prev. | ours |
| apron | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 23.1 | 1 | 0 |
| armchair | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 37.5 | 1 | 0 |
| bathtub2 | 25.8 | 0.0 | 2.0 | 0.0 | 3.2 | 54.8 | 0 | 0 |
| bathtub | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 4.0 | 1 | 0 |
| bedsheet | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 55.7 | 1 | 0 |
| scene | requested | generated | soft bodies solved | soft bodies scored |
|---|---|---|---|---|
| apron | 2 | 2 | 1 | 1 |
| armchair | 2 | 2 | 1 | 1 |
| bathtub2 | 2 | 2 | 1 | 1 |
| bathtub | 2 | 2 | 1 | 1 |
| bedsheet | 2 | 2 | 1 | 1 |
| bench2b | 2 | 2 | 1 | 1 |
| med. | (%) | med. dist. (cm) | mm (%) | |
|---|---|---|---|---|
| naive | 1.00 | 0 | 0.25 | 23 |
| search | 0.11 | 0 | 0.02 | 69 |
| ours | 0.02 | 100 | 0.00 | 73 |
| bodies landed (%) | mean error (cm) | ||||||||
| targets | runs | naive | search | ours | naive | search | ours | ||
| obstructed, lr 0.04 | 1 | 4 | 50 | 75 | 100 | 2.2 | 9.1 | 12.6 | |
| 2 | 4 | 50 | 62 | 100 | 2.0 | 2.0 | 13.5 | ||
| 4 | 4 | 50 | 69 | 100 | 2.3 | 2.3 | 13.8 | ||
| obstructed, lr 0.01 | 4 | 4 | 50 | 69 | 50 | 2.3 | 8.5 | 1.2 | |
| obstructed, two-phase | 2 | 4 | 50 | 75 | 100 | 2.0 | 9.7 | 12.9 | |
| bodies landed (%) | median error (cm) | ||||||||||
| targets | runs | naive | search | grid | ours | naive | search | grid | ours | ||
| obstructed | 1 | 8 | 25 | 38 | 25 | 50 | 2.2 | 0.6 | 0.9 | 12.6 | |
| 2 | 8 | 25 | 31 | 25 | 50 | 2.0 | 2.1 | 1.0 | 13.0 | ||
| 4 | 8 | 25 | 41 | 25 | 56 | 2.5 | 2.8 | 1.4 | 21.5 | ||
| expanded | 1 | 32 | 62 | 69 | 62 | 75 | 0.1 | 0.1 | 0.0 | 0.1 | |
| 2 | 32 | 62 | 66 | 62 | 75 | 0.3 | 0.5 | 0.2 | 0.4 | ||
| prompt | Holodeck’s outcome |
|---|---|
| freestanding bathtub / towel | named in the plan, no mesh (native asset) |
| bathtub / towel | named in the plan, no mesh (native asset) |
| bed / blanket | absent from the scene |
| wooden bench / towel | rigid retrieved mesh, on nothing it named |
| sofa / blanket | absent from the scene |
| leather couch / pillow | rigid retrieved mesh, on the requested support |
Functional 3D Scene Generation
Functional 3D scene generation creates editable three-dimensional environments arranged to support specified activities, interactions, and physical constraints.
See all 37 papers