RoboLayout: Differentiable 3D Scene Generation for Embodied Agents

Ali Shamsaddinlou

cs.AIFeb 18, 2026

RoboLayout: Differentiable 3D Scene Generation for Embodied Agents

Authors: Ali Shamsaddinlou

Abstract

Recent advances in vision language models (VLMs) have shown strong potential for spatial reasoning and 3D scene layout generation from open-ended language instructions. However, generating layouts that are not only semantically coherent but also feasible for interaction by embodied agents remains challenging, particularly in physically constrained indoor environments. In this paper, RoboLayout is introduced as an extension of LayoutVLM that augments the original framework with agent-aware reasoning and improved optimization stability. RoboLayout integrates explicit reachability constraints into a differentiable layout optimization process, enabling the generation of layouts that are navigable and actionable by embodied agents. Importantly, the agent abstraction is not limited to a specific robot platform and can represent diverse entities with distinct physical capabilities, such as service robots, warehouse robots, humans of different age groups, or animals, allowing environment design to be tailored to the intended agent. In addition, a local refinement stage is proposed that selectively reoptimizes problematic object placements while keeping the remainder of the scene fixed, improving convergence efficiency without increasing global optimization iterations. Overall, RoboLayout preserves the strong semantic alignment and physical plausibility of LayoutVLM while enhancing applicability to agent-centric indoor scene generation, as demonstrated by experimental results across diverse scene configurations.

Figures & tables

Fig. 1 : Overall architecture of RoboLayout

Type	Name	Rule
Hard	Boundary (inside room)	Every asset’s 2D footprint must lie inside the room polygon. Any corner outside is projected onto the boundary.
Hard	On-top (vertical + horizontal)	For each on_top_of(A,B): (1) $A$ ’s $z$ is set so $A$ sits on top of $B$ ( $z=B.z+B.h/2+A.h/2$ ). (2) $A$ ’s $xy$ is projected onto $B$ ’s 2D polygon so $A$ is centered over $B$ .
Hard	Rotation unit norm	When using solver_type of minimize, rotation is parameterized as $(\cos\theta,\sin\theta)$ with $\cos^{2}\theta+\sin^{2}\theta=1$ via NonlinearConstraint.
Soft	Overlap (no intersection)	2D footprints of non–on-top pairs must not overlap. Loss: IoU / bbox overlap (e.g. scaled by 1000); minimized so overlap goes to zero.
Soft	Reachability	Pairs of objects should leave enough clearance for a disc of radius robot_radius. Loss: penalize if center distance $<(r_{a}+r_{b}+2\times\text{robot\_radius})$ .
Soft	Against_wall	Asset should be (1) close to a wall segment and (2) front parallel to the wall. Loss: distance of footprint to wall + angle between asset front and wall direction.

TABLE 1: List of hard and soft constraints.

Panel of Figure 3 — Fig. 2 : Example spaces generated by RoboLayout

Explore similar work

PhyScensis: Physics-Augmented LLM Agents for Complex Physical Scene Arrangement
Feb 16, 2026Yian Wang, Han Yang, Minghao Guo +53D Scene Layout Generation Functional 3D Scene Generation
UMass Amherst · Genesis AI · MIT +1
HOG-Layout: Hierarchical 3D Scene Generation, Optimization and Editing via Vision-Language Models
Apr 12, 2026Haiyan Jiang, Deyu Zhang, Dongdong Weng +23D Scene Generation 3D Scene Layout Generation
The Hong Kong Polytechnic University · Beijing Institute of Technology
ThinkBLOX: 3D Indoor Scene Generation with Progressive Reasoning
Jul 15, 2026Yuan Xiao, Can Wang, Xiangyu Kong +13D Scene Editing 3D Scene Generation
Department of Computer Science, City University of Hong Kong, Hong Kong · School of Computing and Data Science, University of Hong Kong · School of Computer Science, Beijing Information Science and Technology University

Feb 16, 2026J/K move · Enter open · S save

PhyScensis: Physics-Augmented LLM Agents for Complex Physical Scene Arrangement

Yian Wang, Han Yang, Minghao Guo, Xiaowen Qiu+4

UMass Amherst · Genesis AI · MIT+1

3D Scene Layout Generation Functional 3D Scene Generation Synthetic Data Generation for Robotics LLM Agents

Open paper AskPDF arXiv

Automatically generating interactive 3D environments is crucial for scaling up robotic data collection in simulation. While prior work has primarily focused on 3D asset placement, it often overlooks the physical relationships between objects (e.g., contact, support, balance, and containment), which are essential for creating complex and realistic manipulation scenarios such as tabletop arrangements, shelf organization, or box packing. Compared to classical 3D layout generation, producing complex physical scenes introduces additional challenges: (a) higher object density and complexity (e.g., a small shelf may hold dozens of books), (b) richer supporting relationships and compact spatial layouts, and (c) the need to accurately model both spatial placement and physical properties. To address these challenges, we propose PhyScensis, an LLM agent-based framework powered by a physics engine, to produce physically plausible scene configurations with high complexity. Specifically, our framework consists of three main components: an LLM agent iteratively proposes assets with spatial and physical predicates; a solver, equipped with a physics engine, realizes these predicates into a 3D scene; and feedback from the solver informs the agent to refine and enrich the configuration. Moreover, our framework preserves strong controllability over fine-grained textual descriptions and numerical parameters (e.g., relative positions, scene stability), enabled through probabilistic programming for stability and a complementary heuristic that jointly regulates stability and spatial relations. Experimental results show that our method outperforms prior approaches in scene complexity, visual quality, and physical accuracy, offering a unified pipeline for generating complex physical scene layouts for robotic manipulation.

Abstract

Explore similar work

PhyScensis: Physics-Augmented LLM Agents for Complex Physical Scene Arrangement

HOG-Layout: Hierarchical 3D Scene Generation, Optimization and Editing via Vision-Language Models

ThinkBLOX: 3D Indoor Scene Generation with Progressive Reasoning