Beyond Placement and Articulation: Usage-Driven Code Scenes for Embodied Interaction
Organizations: Shanghai Jiao Tong University · Meituan
Abstract
Indoor scene synthesis provides essential environments for embodied AI, robotic manipulation, and simulation-based policy learning. Recent code-based scene generation methods produce editable and extensible environments, yet they remain focused on visual construction and object-level articulation, leaving the functional usage of scenes largely unmodeled. To address this problem, we present RoomWright, an agentic usage-driven framework for generating 3D scenes represented entirely as code for embodied interaction. RoomWright performs usage-driven object reasoning, which treats each anchor as a task centre and admits task-required objects and their affordances. A code agent further enables multi-part interaction by compiling each interaction into a trigger, condition, effect rule that updates structured object states, capturing causal dependencies across objects. Moreover, since manipuland orientation is ambiguous and hard to recover from pixels, RoomWright alleviates this via annotation-informed usage-guided orientation. Extensive experiments demonstrate the effectiveness of our method. The resulting scenes are executable, editable, and simulation-ready, providing interactive environments for embodied AI and policy learning.
Figures & tables
| Method | Collision-free | Navig. | In-bound | Opening-clear | Support | Access. | ObjCount | ObjAttr | OO-Rel | OA-Rel |
|---|---|---|---|---|---|---|---|---|---|---|
| SceneCode | 91.2 | 99.0 | 81.2 | 96.4 | 26.9 | 26.1 | 38.1 | 30.7 | 16.1 | 55.6 |
| Code-as-Room | 45.5 | 89.5 | 73.3 | 47.2 | 51.9 | 63.1 | 37.0 | 56.7 | 20.5 | 22.2 |
| SceneSmith | 95.0 | 99.6 | 99.7 | 98.0 | 58.4 | 39.8 | 48.8 | 42.5 | 27.0 | 42.2 |
| Ours | 95.9 | 97.3 | 99.7 | 95.7 | 53.6 | 65.6 | 57.0 | 64.6 | 22.4 | 40.6 |
| Method | IO | AO | CHI | MPC |
|---|---|---|---|---|
| Code-as-Room | – | – | – | – |
| SceneSmith | 2.4 | 2.4 | – | – |
| SceneCode | 4.4 | 4.4 | – | – |
| Ours | 16.4 | 13.2 | 6.4 | 3.6 |
| ID | Room type | Prompt (verbatim) |
|---|---|---|
| laundry_room | laundry room | A laundry room with a washing machine and a dryer. |
| bathroom1 | bathroom | A bathroom with a vanity, a bathtub, and a toilet. |
| office1 | home office | A home office with a desk, an office chair, and a bookshelf. |
| gym1 | home gym | A home gym with a treadmill and an exercise bike, and a wall mirror. |
| library | home library | A home library with a bookshelf, a reading armchair, and a floor lamp. |
| art_studio | art studio | An art studio with an easel, a stool, and a worktable. |
| ID | Room type | Prompt (verbatim) |
|---|---|---|
| kitchen | kitchen | A kitchen with a counter along the main wall holding a microwave and a knife block, a refrigerator at one end of the counter, and a built-in oven with a stovetop, with an island in the middle with two bar stools tucked underneath. |
| bedroom1 | bedroom | A bedroom with a queen bed against the main wall, a nightstand with a table lamp on each side of the bed, a tall wardrobe with sliding doors, a low dresser with a wide mirror above it, and a cushioned bench at the foot of the bed. |
| living_room | living room | A living room with a three-seat sofa facing a wall-mounted TV, a coffee table with a stack of magazines on a wide area rug, a media console beneath the TV flanked by two tall floor speakers, and a bookshelf along the side wall. |
| dining_room1 | dining room | A dining room with a long rectangular table surrounded by six chairs and set with tableware, a sideboard against the wall holding serving dishes, and a pendant chandelier above the table. |
| garage_workshop | garage workshop | A garage workshop with a heavy workbench along the main wall fitted with a bench vise and a toolbox, a tall pegboard above the bench holding hanging tools, a rolling tool chest with drawers, and a metal shelving unit against the side wall. |
| playroom | children’s playroom | A children’s playroom with a low activity table with four small chairs, a tall toy shelf filled with bins of toys, a cozy reading corner with a bean bag chair and a floor lamp, and a play tent near the window. |
| Family | Metrics | What it measures |
|---|---|---|
| Physical plausibility | COL, NAV, OOB, OPC | realistic, walkable, in-room layout |
| VLM-judged physical | SUP, ACC | objects properly supported & reachable |
| Prompt fidelity | CNT, ATR, OO-Rel, OA-Rel | scene matches the text prompt |
| Layout parameters | Resulting asset |
|---|---|
| rows=2, cols=1 | nightstand |
| rows=2, cols=3 | wide dresser |
| double doors top + rows=2 bottom | cupboard with drawers |
| rows=1 top + door bottom | nightstand with cabinet |
| door column + rows=3 column | wardrobe with drawers |
| rows=3 column + door-over-drawer column | combo dresser |
| Cost item | Tokens (M) |
|---|---|
| Total token count | 1,204.5 |
| Average token count | 43.0 |
| Maximum token count | 72.0 |
| Minimum token count | 27.2 |
| Statistic | Wall-clock time | Seconds |
|---|---|---|
| Average time cost | 9:10:38 | 33,038.0 s |
| Maximum time cost | 12:21:11 | 44,471.0 s |
| Minimum time cost | 6:34:49 | 23,689.0 s |
3D Scene Generation
3D scene generation creates structured, explorable environments from text, images, maps, or other inputs.
See all 189 papers