We present SoftPAT3D, a physics-augmented framework for text-to-3D scene generation in which deformable objects, such as cloth, stuffed toys and ropes, are generated as physically valid, simulation-ready bodies and placed in intersection-free contact with the furniture that supports them, with one another and with themselves. Existing physics-augmented pipelines simulate rigid bodies only and cannot accommodate a generated deformable object, which text-to-3D generation returns as a geometrically invalid mesh, sometimes self-intersecting, that has no rest shape, no material model and no admissible initial state. These are the inputs a deformable simulation requires, and we construct each from information already in the pipeline. The rest shape comes from how the object is manufactured rather than from the generated physically inconsistent deformed configuration, and a material library assigns each deformable object its initial elastic parameters, which can then be optimized. The occluded support is regenerated and converted into a watertight collision mesh, each object is released from an optimizable placement above the supporting part that the prompt specifies, and the complete scene is driven to static equilibrium under one unified contact model, with completion verified by geometric checks and a vision-language model. When a scene fails the prompt, a grid search over placements corrects it, and the adjoint of a differentiable simulation of the same contact energy reaches the targets the grid does not, places several bodies jointly and recovers and optimizes material parameters. Multi-body scenes equilibrate in a single simulation, and the pipeline also runs from the text prompt alone. On all sixty-five prompts, SoftPAT3D produces no undesired interpenetration of any kind, whereas every other system does, and its scenes directly support scene editing and robotic manipulation.
Figures & tables
prompt
rigid
soft sim.
soft equil.
soft grip
soft grad.
Holodeck [ 15 ]
✓
–
–
–
–
–
PAT3D [ 1 ]
✓
✓
–
–
–
–
SimuScene [ 20 ] ‡
–
✓
–
–
–
–
SceneSmith [ 21 ]
✓
✓
–
–
–
–
CoDimRecon [ 22 ] †‡
–
✓
✓
✓
✓
–
SoftPAT3D (ours)
✓
✓
✓
✓
✓
✓
Table 1: Scope. Input (a text prompt; ‡ SimuScene takes a single image and CoDimRecon a multi-view capture), a rigid-body physics stage, and, for deformable objects, simulation, equilibrium on their supports, manipulation by a gripper and a gradient through contact. PAT3D’s gradient and manipulation act on rigid bodies. † Concurrent work.
Figure 2: Overview of SoftPAT3D. Each card shows the state of one prompt at that point of the pipeline, and its label names the step that produced it. (a) The prompt is expanded into a reference image, segmented and lifted as in PAT3D, and the support hidden under each deformable object is inpainted, regenerated and converted into a collision mesh (Sec. 3.4 ). (b) Each deformable object receives a rest shape from its manufacturing process under a size prior (Sec. 3.2 ), a material from the material library and a placement above the part the prompt names (Sec. 3.4 ). (c) One simulation brings the scene to equilibrium under a single contact barrier (Sec. 3.3 ), and the result passes the geometric checks and the prompt check (Sec. 3.5 ); a scene that fails enters placement optimization (Sec. 3.6 ).
Figure 3: Generated scenes. Top two rows: single-object scenes from the main run, development and held-out, each a generated support with a sewn cloth at equilibrium on the part the prompt specifies. Bottom: multi-body scenes with two to eight deformable objects, quilts, pillows, garments and stuffed toys, brought to equilibrium in one simulation. Every scene is rendered on the collision mesh of the solver.
Figure 4: Comparison with the rigid pipeline. Top: the rigid pipeline places the generated deformable object as a rigid body that intersects or floats above the furniture. Bottom: the same scenes from the SoftPAT3D main run, as the solver returned them on their collision meshes.
Holodeck
Astra
rigid
Blender
Blender ∗
ours
scenes
32
29
34
34
32
34
deformable present
8/32
29/32
34/34
34/34
32/32
34/34
soft bodies
39
74
48
48
40
43
crossings (per 1k edges)
71.2
40.2
5.1
196.2
66.2
0.0
penetration (mm)
0.0
1.7
14.8
1.9
2.5
0.0
in contact (%)
10.3
38.5
1.8
22.8
25.2
39.9
Table 2: Comparison with other systems. SoftPAT3D has zero crossings and zero penetration, and its deformable objects contact their supports broadly and rest on them. Blender ∗ has two scenes timed out. Deformable objects follow the labels of each system, and the per-body rows pool the bodies each system placed. The last two rows apply our completion criterion to every system.
scenes
plausibility
CLIP
VQAScore
rigid pipeline
13
43
23.9
0.50
ours, the same scenes
13
90
28.2
0.74
GPT-6 Astra
11
81
24.9
0.47
ours, the same scenes
11
87
27.7
0.65
Table 3: Render metrics under the protocol of PAT3D. GPT-5.4 physical plausibility (0 to 100), CLIP score and VQAScore on one render per scene, on the scenes each baseline rendered.
Figure 5: Vertical sections, cloth in red, furniture in grey, cloth inside the furniture in dark red, and a dashed line where the cloth floats above its support. Top: GPT-6 Astra; the blanket and the towel hang above their seats, the quilt runs through the mattress, the pillow sits inside the backrest, and the skirt ripples in mid-air. Bottom: ours; the cloth rests on the support, wraps its edges and hangs from them.
Figure 6: Scene editing and manipulation on equilibrated scenes. Top: the five stuffed toys of the messy bed are set on the floor in turn while the quilt and pillows keep settling. Middle: a cloth pulled off a stool by one corner. Bottom: the throw is lifted off the armchair, the rabbit carried to the floor and a pillow placed on the seat, every body simulated; the arm is posed for visualization.
condition
S
G
J
C
full stage, asset as rest shape
22
19
14
14 (9)
centred placement, hull-based support check
32
28
17
17
+ watertight collision mesh
33
31
18
18 (13)
+ placement on the part the language model names
33
30
20
20 (15)
+ its size estimate, full stage
33
30
22
21 (16)
Table 4: Same-solver controls on the 33 development scenes, same assets, solver and budget. First row: the generated asset’s closed surface as the rest shape; below it the components of the physics stage are added one at a time; bold: the main run. In parentheses: the 26 cloth scenes alone. S: static equilibrium; G: geometric checks; J: prompt check; C: complete. Crossings are zero in every cell.
Figure 7: Placement ablation. An apron and a towel placed centred on their support and on the part the language model names, both at the size it estimates.
Figure 8: Failure cases. Left: the generated bed is a solid block with no footboard, so the quilt covers the whole bed. Right: pillows placed over the seat settle in a cluster, not in the row the prompt asks for.
Figure S1 : Equilibrated scenes , rendered from the solver’s own output with the cloth texture extracted from each scene’s reference image and the furniture’s regenerated texture, under one studio environment map (HDRI). Top: a blanket over a rocking chair, a knit blanket over an armchair, a tablecloth on a square dining table, a gingham cloth on a picnic table. Bottom: a quilt on a daybed pushed flush to its back and hanging over the front rail, a patterned throw over the back of a leather sofa, a sheet folded over the foot of a bed, and an apron hung over the back of a kitchen chair. Eight of the thirty-two scenes are shown. Further scenes are shown in Fig. S2 , and all thirty-two are reported with their numbers in Table S1 , including the ones that a geometric check rejected. Because the renderer draws the solver’s collider with the asset’s texture projected onto it, a render cannot show contact that the simulation did not produce.
Appendix figures & tables37 assets
Supplementary material from the paper’s appendix.
Appendix
scene
bodies
crossings
contact (%)
blocked (%)
apron
1
0
23
17
armchair
1
0
38
0
bathtub2
1
0
55
7
bathtub
1
0
4
11
bedsheet
1
0
56
0
bench2b
1
0
61
0
Appendix
Table S1 : The full scene set. Every scene that any version of the pipeline was asked to equilibrate is included, including the ones that earlier versions rejected. Crossings are cloth edges passing through a support, counted against the collider the solver used. “Blocked” is the upward-ray measure of Sec. H , the fraction of cloth vertices with support surface above them. It is large where a body has come to rest on something other than the top. Scenes below the rule were rejected and are labelled with what the containment or support check found.
Figure S2 : Scenes of the main set that pass the geometric checks , in the development simulations, under the same camera, lighting and texture pipeline as the gallery. Furniture is drawn as the solver’s collider with the asset’s texture projected onto it, and the cloth as the solver left it.
bodies
cross. (per 1k)
pen. (mm)
contact (%)
floating
unsup.
zero-cross. scenes
Blender ∗
27
24.4
0.1
17.5
9/27
0/27
23/25
ours
27
0.0
0.0
52.5
0/27
0/27
25/25
Appendix
Table S2 : Blender’s cloth solver and ours on the held-out batch , same sewn sheets, placements and colliders, over the 25 held-out scenes both produced; the metrics of the main paper’s Table 3.
configuration
S
G
J
C
physics stage
24
23
14
14
+ placement grid of Eq. 9 (the pipeline)
24
23
19
19
Appendix
Table S3 : Held-out completion at each stage of placement optimization. Settled (S), passing the geometric checks (G), passing the prompt check (J) and complete (C) out of 32, Eq. 9 is the placement grid of the main paper, which is the single-body correction of the pipeline, run at six re-simulations per scene.
scene
prompt
outcome
bedfoot
A folded wool blanket lying across the foot of a neatly made bed.
complete
bookshelf
A knitted blanket draped over the top of a low wooden bookshelf.
complete
counter
A kitchen towel draped over the edge of a small kitchen island counter.
incomplete
drawers
A pleated skirt draped over the top of a chest of drawers.
incomplete
footboard
A patchwork quilt draped over the wooden footboard of a bed.
incomplete
ironing2
A white dress shirt lying flat on an ironing board.
complete
Appendix
Table S4 : The held-out batch. The 32 prompts written after the pipeline was frozen, each with the outcome of the final code’s run on the same assets, the full-stage cell of Sec. C.1 , which is the number reported in the main paper. “Complete” is the full criterion of the main set. Every prompt ends with a setting clause, “in a simple room” or similar, omitted here. 14 of 32 are complete.
Placement (Sec. 3.4 of the main paper)
part
the planner names a surface, a part (seat, backrest, arm, rim) and a drop height; objects seen in the image keep their layout place
straddle
a sheet that fits inside the seat is centred on the seat’s front edge rather than on the seat
flush
a sheet deeper than a backed seat is pushed against the back, so the excess hangs over the front
throw
a throw on seating goes over the backrest unless the planner names another part
order
coverings first, smaller items on top; a sheet in a container is shrunk below the opening; line-mounted cloth starts vertical, top row pinned
lift
every release is tested for intersection and lifted until it is free
Appendix
Table S5 : Placement rules and completion tests. The rules of Sec. 3.4 and the tests of Sec. 3.5 of the main paper convert the language model’s answer into an intersection-free release, and the tests decide whether an equilibrated scene is complete.
base
+ collision mesh
+ placement
full
scene
S
G
J
x
S
G
J
x
S
G
J
x
S
G
J
x
apron
y
y
n
0
y
y
n
0
y
y
y
0
y
y
y
0
armchair
y
y
n
0
y
y
n
0
y
y
y
0
y
y
y
0
bathtub2
y
y
n
0
y
y
n
0
y
y
y
0
y
y
y
0
bathtub
y
y
y
0
y
y
y
0
y
y
y
0
y
y
y
0
bedsheet
y
y
n
0
y
y
n
0
y
y
n
0
y
y
n
0
Appendix
Table S6 : Per-scene results of the same-solver controls on the development set. S: reached static equilibrium; G: containment and support checks; J: prompt check; x: crossings per thousand edges. A dash marks a run without an equilibrium state to score.
component → what preceded it
effect without it
scenes
sewing-pattern rest shape → generated asset
layers closer than d^ , world invalid at step 0, 0.0 mm displacement; the render still shows a drape
6 of 6
intersection-free closure → watertightness test
nine supports fall back to convex hulls; cloth equilibrates on a block, a wedge, a tub with no interior
9 of 32
convex-cell collider → single hull
Hausdorff error to the asset 82.8 vs 8.4 mm; the towel hangs through the chair’s open back
1 of 1
placement rules → planner’s point
the sheet rests as a flat rectangle centred on the seat and passes every check
6 of 32
support de-occlusion → masked crop
the generator fills the mask with cloth fused into the support; the sheet drapes over it
32 of 33
gradient release correction → straight drop
obstructed targets within 3 cm: 1 straight, 4 random search, 6 gradient
14 targets
Appendix
Table S7 : Origin of each placement rule. Each component replaced by what preceded it, on the scenes where it was introduced.
criterion
scenes (of 60)
same kind of object
48
same colour and pattern
34
Appendix
Table S8 : Properties of the generated object preserved by the sewn template. GPT-5.4 is shown the reference image and the record render of each of the 60 record-run scenes and asked, of the deformable object alone, whether its kind and its colour and pattern survive.
A
C
B
D
scene
S
G
J
x
S
G
J
x
S
G
J
x
S
G
J
x
apron
y
y
n
0
y
y
n
0
y
y
y
0
y
y
y
0
armchair
y
y
n
0
y
y
y
0
y
y
n
0
y
y
y
0
bathtub2
y
y
n
0
y
y
n
0
y
y
y
0
y
y
y
0
bathtub
y
y
y
0
y
y
n
0
y
y
y
0
y
y
n
0
bedsheet
y
y
n
0
y
y
n
0
y
y
n
0
y
y
n
0
Appendix
Table S9 : Per-scene results of the size and placement conditions under one code version, on the development set. A: centred placement at the class size; C: centred placement at the size the language model estimates; B: placement on the part the language model names, class size; D: that part and size, which is the full stage. Columns as in Table S6 ; x is the crossing count.
collider
mean/max (mm)
closed
single convex hull
82.8 / 330.5
✓
convex cells ( K=35 )
8.4 / 29.4
✓
raw generated mesh
–
×
∇ϕ~ vs. finite diff.
1.6×10−7
Hessian-vector vs. finite diff.
1.9×10−6
Appendix
Table S10 : Collider validation on the towel-scene chair. Convex cells reduce the fabricated geometry by an order of magnitude compared with the single convex hull, whose 330 mm maximum spans the chair’s open back, where a draped towel hangs through the frame. Both are closed by construction, which gives the inside/outside test that the raw mesh cannot, and the analytic contact field matches finite differences to the order that the release-pose gradient requires.
Input: the prompt and the generated scene, that is, the reference image, masks, lifted assets and language model layout. Output: an equilibrated simulation state, or a rejection with its cause.
1. Route every object: rigid support, shell, volumetric body or rod, by the keyword material library; take its material from the material library (Sec. 3.2).
2. Supports: de-occlude the support’s crop and regenerate the asset, align it to the layout, close it (open-edge test, convex cells), and write its certificate (Sec. 3.4).
3. Rest shapes: cut a sheet to the outline its class names, or sew the library garment pattern, at the size the language model estimates or the class prior; stuff the bodies that are stuffed (Sec. 3.2).
4. Release: resolve the part the prompt names on the collider, place the rest shape above it, lifted clear of the bodies already placed; check that step zero is intersection-free (Sec. 3.4).
5. Equilibrate: minimize the incremental potential with the per-object energies for up to 300 steps of 0.02 s; stop once the largest displacement over 40 steps is under 0.5 mm (Sec. 3.3).
6. Checks: crossings, support, floor and asset tests on the equilibrated geometry; then the prompt check on the render (Sec. 3.5).
Appendix
Figure S3: The equilibration stage. One pass over a generated scene.
step (s)
n
settled
cross.
pen. (mm)
contact (%)
0.005
128
6/6
0
0.0
5–100
0.01
128
6/6
0
0.0
9–100
0.02
128
6/6
0
0.0
8–100
0.04
128
6/6
0
0.0
4–100
0.02
64
6/6
0
0.0
2–100
0.02
192
4/4
0
0.0
2–90
Appendix
Table S11 : Numerical sensitivity. Six development scenes re-solved at other time steps (simulated time fixed at 6 s) and sheet resolutions. Settled: scenes reaching static equilibrium; crossings and penetration are summed and maximised over the scenes; contact: range of the contact fraction.
scene
deformable objects
shells
volumetric bodies
wall
result
messy bed
quilt, pillows, plush bear
2
2
8.9
equilibrated, passes
made bed
sheet and three pillows
1
3
8.4
equilibrated, passes
wardrobe
shirt and trousers on a bed
2
0
8.0
equilibrated, passes †
dinner
tablecloth and napkin on a chair
2
0
8.6
equilibrated, passes †
messy bed 2
quilt, two pillows, five toys
1
7
11.5
equilibrated, passes
reading nook
blanket, two pillows, plush rabbit, charging cable
1
3
6.9
equilibrated, passes (one rod)
Appendix
Table S12 : Multi-body scenes. Shells and volumetric bodies are deformable objects, and every scene also has one rigid support. Wall time is in minutes for the whole job on one A5000 or A6000. † The smaller shell of the pair is a membrane-only shell. ‡ Solved on a 96 GB card with the library garments decimated to about 5,200 faces each, four of the six as membrane-only shells. It is not shown, because its wardrobe reached the solver as three loose shelves, which is an asset defect.
Figure S4 : Multi-body scenes , each the last simulation recorded in Table S12 : two to eight deformable objects, quilts, pillows, garments, stuffed toys and a cord, equilibrated in one simulation.
Figure S5 : Rods. Three rods laid on supports of the scene set, a cable, a rope and a telephone cord on an equilibrated tablecloth, and the two scenes whose prompt named the rod, equilibrated in the same simulation as the shells.
Figure S6 : Six garments on a sofa. The prompt asks for several clothes thrown messily over a sofa. Left: the rigid pipeline lifts each garment to a frozen shell, and its pile intersects the backrest and hangs in the air. Middle and right: our equilibration stage brings the six garments to rest on the seat and back in one simulation, with none of their 12,089 cloth edges crossing the sofa, seen from the generation viewpoint and from two novel views.
material
E (Pa)
kbend
r (mm)
ρ (kg/m 3 )
μ
rope
2×107
5×105
12
900
0.7
cable
1×108
5×106
2.5
1200
0.5
cord
1×108
2×106
3
1100
0.5
string
5×107
2×105
1.5
900
0.5
hose
5×107
3×106
8
1000
0.6
Appendix
Table S13 : Rods. Left: the material table, chosen from the rod’s name. kbend is the stiffness passed to the Kirchhoff term, α=kπr4/4 . Right: the five rod scenes of Fig. S5 , with the curve each rod was released as and the wall time in minutes. The last two are scenes whose prompt named the rod as the main object. The telephone cord lies on the dining table’s tablecloth, a shell equilibrated in the same simulation.
single body, target past an obstruction
runs
grid
adjoint
four supports, two seeds (Table S24 )
8
2
4
eight supports, four seeds (Table S24 )
32
20
24
Appendix
Table S14 : Single-body placement past an obstruction. Runs in which the body lands on its target, for the grid at twice the adjoint’s forward budget and for the adjoint. No point on the grid’s line reaches a target behind an obstruction, whereas the gradient carries the body across it.
Figure S7 : Scene editing. A move and an addition on the equilibrated second messy bed in one simulation, with every body simulated throughout (Sec. 4.4 of the main paper): the plush toy on the pillows is gripped, carried to the foot of the bed and set down on the quilt, and a foam pillow introduced by the prompt is held while the scene equilibrates at a release that the differentiable surrogate corrected toward the head of the bed, and then released. It comes to rest 1.2 cm from the target, compared with 67 cm from the language model’s placement.
Figure S8 : Pull. The tablecloth of the dining scene is gripped at its farthest corner after equilibration and drawn off the table, through the same soft position constraint as every other episode.
Figure S9 : Press. A stuffed pillow on the leather couch is gripped at its crown and driven down by half its height. The couch’s seat, a rigid collider in this scene, does not deform, so the indentation is all in the pillow.
Figure S10 : Lift. The towel on the chair is gripped at its top centre after equilibration and raised 0.5 m. It hangs from the grip, and on release it falls back onto the seat.
scene
episode
grip
err (mm)
max (mm)
reached
equilibrated
held
released
finite
armchair
pull
54
8.3
44.4
0.99
18.5
0.0
0.0
✓
bedsheet
pull
21
19.3
75.6
0.99
10.6
4.4
0.0
✓
bench2b
pull
144
6.1
42.6
1.00
48.1
0.0
0.0
✓
daybed
pull
34
12.9
73.8
0.99
8.0
0.0
0.0
✓
dining2
pull
28
14.1
92.5
0.99
16.0
11.1
4.3
✓
dining
pull
42
9.9
76.2
1.00
21.9
8.4
0.1
✓
Appendix
Table S15 : All manipulation episodes. “Grip” is the number of constrained vertices. Contact columns are the fraction of body vertices within 5 mm of the support at static equilibrium, at the end of the hold, and 1.8 s after release.
scene
carry (cm)
retained (%)
contact, settled (%)
contact, end (%)
bedsheet
200
104
30
15
bench
59
100
69
37
blanket
141
99
10
4
daybed
147
100
6
4
dining
96
5
22
20
picnic
184
101
11
4
Appendix
Table S16 : Fold episodes. Carry: horizontal travel of the gripped corner at the hold. Retained: the fraction of that travel surviving the release. Contact: fraction of sheet vertices within 5 mm of the support, at the settled state and at the end of the episode.
intermediate
rigid
Blender, asset
Blender, ours
ours
image, segmentation, assets
∙
∙
∙
∙
layout, support graph
∙
∙
∙
∙
cloth rest shape
asset
asset
sewn sheet
sewn sheet
support geometry
painted
painted
de-occluded, closed
de-occluded, closed
release pose
layout
layout
rules (Sec. 3.7)
rules (Sec. 3.7)
solver
rigid IPC
Blender cloth
Blender cloth
IPC, routed
Appendix
Table S17 : Pipeline components shared by each condition of Table 2. A dot marks an intermediate taken unchanged from the pipeline.
Figure S11 : The rigid pipeline and ours on the same prompts and assets. Top row of each pair: previous work, every generated object kept rigid. Bottom row: the same scene after our stage. The cloth in the top row keeps the pose the generator gave it, which is why it hovers, tilts, and passes through the furniture it is meant to rest on.
system
scenes
present
finished
X=0
G
J
complete
Holodeck
32
8
8
5
2
2
2
Astra
29
28
28
14
15
19
7
rigid
34
34
34
14
15
11
0
Blender
34
34
22
11
11
0
0
Blender ∗
34
32
32
23
19
8
8
Blender ∗ , 40 substeps
34
30
30
25
14
7
7
Appendix
Table S18 : Completion criterion applied to every system. Scenes of the comparison set in which the deformable object is present, the run finished, no edge crosses another body, both geometric checks pass with the object resting on its support, the prompt check passes, and all of these hold, Eq. 6 of the main paper.
Holodeck
Astra
rigid
Blender
Blender ∗
ours
scenes
32
29
34
34
32
34
deformable present
8/32
29/32
34/34
34/34
32/32
34/34
soft bodies
39
74
48
48
40
43
crossings (per 1k edges)
71.2
40.2
5.1
196.2
66.2
0.0
penetration (mm)
0.0
1.7
14.8
1.9
2.5
0.0
in contact (%)
10.3
38.5
1.8
22.8
25.2
39.9
Appendix
Table S19 : Aggregate comparison over the comparison set. The rigid pipeline, both Blender runs and ours share prompts and generated assets. Holodeck and Astra build their own scenes from the prompts.
cross/1k
pen. (mm)
contact (%)
floating
scene
prev.
ours
prev.
ours
prev.
ours
prev.
ours
apron
0.0
0.0
0.0
0.0
0.0
23.1
1
0
armchair
0.0
0.0
0.0
0.0
0.0
37.5
1
0
bathtub2
25.8
0.0
2.0
0.0
3.2
54.8
0
0
bathtub
0.0
0.0
0.0
0.0
0.0
4.0
1
0
bedsheet
0.0
0.0
0.0
0.0
0.0
55.7
1
0
Appendix
Table S20 : Per-scene comparison. “prev.” is the rigid pipeline, the generated asset frozen in its layout transform. “Ours” replaces it by the sewn sheet and solves. Deformable objects are taken from the pipeline’s material library rather than filtered by name. Table S21 counts them per scene.
scene
requested
generated
soft bodies solved
soft bodies scored
apron
2
2
1
1
armchair
2
2
1
1
bathtub2
2
2
1
1
bathtub
2
2
1
1
bedsheet
2
2
1
1
bench2b
2
2
1
1
Appendix
Table S21 : Objects per scene. What each scene of the comparison set asked for, received and scored: objects in the parsed request, objects the generator produced, deformable objects the solver simulated, and deformable objects the evaluator scored. The two columns on the right agree in every scene. The clothes sofa’s six garments come from one request entry, and the steamer’s two are its dumplings.
med. ∣log4(w/w⋆)∣
≤10% (%)
med. dist. (cm)
≤1 mm (%)
naive
1.00
0
0.25
23
search
0.11
0
0.02
69
ours
0.02
100
0.00
73
Appendix
Table S22 : Material recovery over 26 cases: the weight-to-stiffness ratio, perturbed by a factor of four, recovered from the equilibrated drape.
bodies landed (%)
mean error (cm)
targets
N
runs
naive
search
ours
naive
search
ours
obstructed, lr 0.04
1
4
50
75
100
2.2
9.1
12.6
2
4
50
62
100
2.0
2.0
13.5
4
4
50
69
100
2.3
2.3
13.8
obstructed, lr 0.01
4
4
50
69
50
2.3
8.5
1.2
obstructed, two-phase
2
4
50
75
100
2.0
9.7
12.9
Appendix
Table S23 : Joint placement optimization of N coupled sheets on the obstructed layout, at the optimiser steps discussed in the text. “Landed” is the solver’s own flag, contact with the support. Error is the contact patch’s distance from the requested point over the sheets that method landed, so it is comparable across columns only where landing rates agree. Targets no method ever reached are excluded. “Runs” counts the runs that enter the landing rate.
Figure S12 : GPT-6 Astra’s scenes, rendered. Ten of the thirty-two scenes the model wrote as Blender programs, rendered from its exported meshes under our studio camera and lighting with the diffuse colours its scripts assign. The fabric is convincing in a rendered view, with a throw with fringe, a tablecloth with a hem, and a striped beach towel. Each is a surface posed by the script rather than equilibrated by a solver. Table S25 and Fig. S13 measure where those surfaces pass through the furniture.
bodies landed (%)
median error (cm)
targets
N
runs
naive
search
grid
ours
naive
search
grid
ours
obstructed
1
8
25
38
25
50
2.2
0.6
0.9
12.6
2
8
25
31
25
50
2.0
2.1
1.0
13.0
4
8
25
41
25
56
2.5
2.8
1.4
21.5
expanded
1
32
62
69
62
75
0.1
0.1
0.0
0.1
2
32
62
66
62
75
0.3
0.5
0.2
0.4
Appendix
Table S24 : Coordinate grid search at the budget of the random search. The four obstructed scenes of Table S23 at learning rate 0.04 , re-run with a per-body record, two seeds each, every requested body counted and no target excluded. Lower block, expanded: the same four supports and four further ones, an armchair, a bench, a bed corner and a dining table, four seeds each. Landed over all bodies; median centroid error over the bodies each method landed, which is why a method that lands more bodies can show a larger error.
prompt
Holodeck’s outcome
freestanding bathtub / towel
named in the plan, no mesh (native asset)
bathtub / towel
named in the plan, no mesh (native asset)
bed / blanket
absent from the scene
wooden bench / towel
rigid retrieved mesh, on nothing it named
sofa / blanket
absent from the scene
leather couch / pillow
rigid retrieved mesh, on the requested support
Appendix
Table S25 : Holodeck and GPT-6 Astra, per prompt. The two systems that build their own scenes. Left, Holodeck on all thirty-two prompts: where it placed a mesh, the surface named is its own label for the object it chose. Right, GPT-6 Astra per prompt, scored by the same rule as every other system in Table S19 . Astra tags its own upholstery as soft, so the cushions, arms and welts it authors for a sofa are treated here as furniture and scored as colliders rather than as failed cloth. What remains is the cloth that the prompt asked for. Its count is still not comparable across systems, because Astra writes a garment as a garment plus its separate hems and fringe, which is why the rates beside it pool vertices rather than averaging over bodies.
Figure S13 : Vertical sections through the scene. Cloth in red, the furniture it should rest on filled in grey, cloth inside the furniture in dark red, and a dashed line where the cloth hangs above its support without touching it. A render cannot show penetration, because the surface that is inside another body is hidden by the body it is inside, whereas a section shows it directly. Top two rows: twelve scenes of GPT-6 Astra. The blanket, the towel, the throw and the clothes hang stiffly above their seats; the quilts and the pillow run through the mattress, the daybed and the backrest; the skirt ripples above the chest, the cloth on the table is a tangle of folds, the dumplings sit on a rippled liner, and the tissues hover inside the bin. Bottom two rows: twelve of our scenes, where the cloth meets the support and stops, wraps its edges and hangs from them under gravity.
Functional 3D scene generation creates editable three-dimensional environments arranged to support specified activities, interactions, and physical constraints.
We propose Tetris3D, a generative framework for single-image 3D scene reconstruction that recovers objects which are physically and geometrically coherent as a scene. Existing methods often generate objects independently or couple them implicitly, providing limited guidance for ensuring fine-grained spatial compatibility between neighboring objects that interact with one another. To address this, we explicitly condition the generation of each object on the geometry of surrounding objects and their physical relationships, guiding its shape and pose to remain geometrically and physically plausible within the scene. Moreover, we introduce ComOb, a physics simulation-based dataset of 1.2M scenes featuring physical interactions across diverse object categories, with per-object meshes and pairwise physical relation annotations. Comprehensive experiments on synthetic and realworld scenes show that Tetris3D recovers coherent object shapes and poses even when interacting regions are occluded, and achieves state-of-the-art performance in both generation quality and physical stability.
Generating physically consistent 3D tabletop scenes is a fundamental yet underexplored problem for interactive and generalist robotic learning. The challenge stems from dense object hierarchies and irregular affordances. Here, an interactive scene denotes a physically valid, collision-free environment directly loadable into physics simulators. Existing methods, ranging from decoupled symbolic solvers to end-to-end regression models, often suffer from error propagation or overfitting to noisy supervision containing widespread physical violations. To address these limitations, we introduce PhyScene3D, a framework that reformulates generation as a Human-Mimetic Constructive Process. The proposed Cognitive Topological Reasoning Chain (CTRC) factorizes scene synthesis into a sequential, anchor-conditioned process. It employs a 3D AABB-based placement scheme that imposes a strong structural inductive bias. To address imperfect supervision and physical infeasibility, we introduce Physics-Aware Denoising Alignment (PADA). It integrates a differentiable Signed Distance Field (SDF) with Test-Time Optimization (TTO) to project generated scenes onto a physics-feasible manifold while preserving semantic intent. Experiments demonstrate that PhyScene3D outperforms state-of-the-art approaches in both semantic accuracy and physical validity, achieving a 40% reduction in scene-wise collision rate relative to the human-annotated training data.
Weixing Chen, Zhuoqian Feng, Yang Liu +6
Sun Yat-sen University, China · Guangdong Key Laboratory of Big Data Analysis and Processing · Huawei +1
Modeling and generating 3D human-object interactions from text is crucial for applications in AR, XR, and gaming. Existing approaches often rely on score distillation from text-to-image models, but their results suffer from the Janus problem and do not follow text prompts faithfully due to the scarcity of high-quality interaction data. We introduce Hoi3DGen, a framework that generates high-quality textured meshes of human-object interaction that follow the input interaction descriptions precisely. We first curate realistic and high-quality interaction data leveraging multimodal large language models, and then create a full text-to-3D pipeline, which achieves orders-of-magnitude improvements in interaction fidelity. Our method surpasses baselines by 4-15x in text consistency and 3-7x in 3D model quality, exhibiting strong generalization to diverse categories and interaction types, while maintaining high-quality 3D generation.
Agniv Sharma, Xianghui Xie, Tom Fischer +2
University of Tübingen · Technische Universit4t N4rnberg · Tübingen AI Center +1