Automatically generating interactive 3D environments is crucial for scaling up robotic data collection in simulation. While prior work has primarily focused on 3D asset placement, it often overlooks the physical relationships between objects (e.g., contact, support, balance, and containment), which are essential for creating complex and realistic manipulation scenarios such as tabletop arrangements, shelf organization, or box packing. Compared to classical 3D layout generation, producing complex physical scenes introduces additional challenges: (a) higher object density and complexity (e.g., a small shelf may hold dozens of books), (b) richer supporting relationships and compact spatial layouts, and (c) the need to accurately model both spatial placement and physical properties. To address these challenges, we propose PhyScensis, an LLM agent-based framework powered by a physics engine, to produce physically plausible scene configurations with high complexity. Specifically, our framework consists of three main components: an LLM agent iteratively proposes assets with spatial and physical predicates; a solver, equipped with a physics engine, realizes these predicates into a 3D scene; and feedback from the solver informs the agent to refine and enrich the configuration. Moreover, our framework preserves strong controllability over fine-grained textual descriptions and numerical parameters (e.g., relative positions, scene stability), enabled through probabilistic programming for stability and a complementary heuristic that jointly regulates stability and spatial relations. Experimental results show that our method outperforms prior approaches in scene complexity, visual quality, and physical accuracy, offering a unified pipeline for generating complex physical scene layouts for robotic manipulation.
Figures & tables
Figure 1: We present PhyScensis , an agentic framework that incorporates a physics engine for physical scene arrangement. PhyScensis is: (a) capable of generating complex scenes with high object density and intricate physical interactions; (b) highly controllable, with with strong text-following abilities; and (c) adaptable to diverse, open-vocabulary scenarios.
Figure 2: Our framework consists of three components: (a) an LLM agent that takes a user prompt and generates spatial and physical predicates, along with object descriptions for retrieval; (b) a solver that computes the final scene using a physics engine for physical predicates and a sample-based constraint solver for spatial predicates; and (c) a feedback system that reports success or diagnoses failure, allowing the LLM agent to iteratively refine and regenerate predicates.
Figure 3: Examples of placements generated by physical solvers.
Figure 4: The stacking generation pipeline uses an occupancy-grid-based heuristic to efficiently compute candidate placement locations via grid search, which are then ranked by user requirements. A physics simulator verifies physical validity (e.g., whether an object will fall), and probabilistic programming further assesses stability, enabling control over the robustness of valid states.
Metrics
Method
VQA Score ↑
GPT Ranking ↓
Settle Distance ↓
Reaching ↑
Placing ↑
Architect
0.493±0.392
2.607±0.673
0.405±0.471
3/10
0/10
3D-Generalist
0.578±0.399
1.946±0.731
0.033±0.048
4/10
1/10
Ours
0.704±0.425
1.429±0.562
0.003±0.008
9/10
3/10
Table 1: Quantitative comparison of PhyScensis with baselines.
Figure 5: Qualitative comparison of PhyScensis with baselines for different generating scenarios.
Table 7
Baseline
Match with Text ↑
Naturalness & Physics ↑
Complexity ↑
Architect
2.68
2.65
2.69
3D-Generalist
2.54
2.72
3.04
Ours
4.04
3.98
3.82
Table 4: User Study Results. We report the mean scores from user evaluations (scale 1-5).
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Qualitative comparison with Random and LLM-Only baselines.
Figure 7: Examples of cases when the retrieved objects are not reasonable.
Figure 8: Examples of cases when spatial solver cannot find a solution.
Figure 9: Examples of unstable stack.
Figure 10: Examples of results with low VQA Scores.
Figure 11: An example of policy rollout.
Figure 12: Visualized samples from generated scene for the robot manipulation task. The result shows that our method, while still being diverse, resembles more to the human intuitions.
Metrics
Method
VQA Score ↑
GPT Ranking ↓
Settle Distance ↓
Architect
0.493±0.392
3.250±0.947
0.405±0.471
3D-Generalist
0.578±0.399
2.393±0.899
0.033±0.048
LayoutVLM
0.648±0.446
2.643±1.109
0.115±0.238
Ours
0.704±0.425
1.714±0.920
0.003±0.008
Appendix
Table 5: Additional quantitative comparison of PhyScensis with baselines, including LayoutVLM.
Figure 13: Qualitative results for LayoutVLM, using the same prompt as in Fig 5 .
Figure 14: Qualitative comparison between our method and ClutterGen ( Jia & Chen, 2024 ) using the same asset library. We pick the best 3 placements of ClutterGen to render.
Figure 15: Generated placements for different stability level.
Generating physically consistent 3D tabletop scenes is a fundamental yet underexplored problem for interactive and generalist robotic learning. The challenge stems from dense object hierarchies and irregular affordances. Here, an interactive scene denotes a physically valid, collision-free environment directly loadable into physics simulators. Existing methods, ranging from decoupled symbolic solvers to end-to-end regression models, often suffer from error propagation or overfitting to noisy supervision containing widespread physical violations. To address these limitations, we introduce PhyScene3D, a framework that reformulates generation as a Human-Mimetic Constructive Process. The proposed Cognitive Topological Reasoning Chain (CTRC) factorizes scene synthesis into a sequential, anchor-conditioned process. It employs a 3D AABB-based placement scheme that imposes a strong structural inductive bias. To address imperfect supervision and physical infeasibility, we introduce Physics-Aware Denoising Alignment (PADA). It integrates a differentiable Signed Distance Field (SDF) with Test-Time Optimization (TTO) to project generated scenes onto a physics-feasible manifold while preserving semantic intent. Experiments demonstrate that PhyScene3D outperforms state-of-the-art approaches in both semantic accuracy and physical validity, achieving a 40% reduction in scene-wise collision rate relative to the human-annotated training data.
Weixing Chen, Zhuoqian Feng, Yang Liu +6
Sun Yat-sen University, China · Guangdong Key Laboratory of Big Data Analysis and Processing · Huawei +1
Recent advances in vision language models (VLMs) have shown strong potential for spatial reasoning and 3D scene layout generation from open-ended language instructions. However, generating layouts that are not only semantically coherent but also feasible for interaction by embodied agents remains challenging, particularly in physically constrained indoor environments. In this paper, RoboLayout is introduced as an extension of LayoutVLM that augments the original framework with agent-aware reasoning and improved optimization stability. RoboLayout integrates explicit reachability constraints into a differentiable layout optimization process, enabling the generation of layouts that are navigable and actionable by embodied agents. Importantly, the agent abstraction is not limited to a specific robot platform and can represent diverse entities with distinct physical capabilities, such as service robots, warehouse robots, humans of different age groups, or animals, allowing environment design to be tailored to the intended agent. In addition, a local refinement stage is proposed that selectively reoptimizes problematic object placements while keeping the remainder of the scene fixed, improving convergence efficiency without increasing global optimization iterations. Overall, RoboLayout preserves the strong semantic alignment and physical plausibility of LayoutVLM while enhancing applicability to agent-centric indoor scene generation, as demonstrated by experimental results across diverse scene configurations.
Generating simulation-ready tabletop scenes from task instructions is an intriguing and promising research direction in the field of Embodied AI. However, existing task-to-scene generation methods rely exclusively on large language models (LLMs) to predict scene layouts, inevitably yielding object collisions or floating due to LLMs' inherent limitations in 3D spatial reasoning. In this paper, we present STABLE, a semantics-physics dual-system tailored for simulation-ready tabletop scene generation. STABLE consists of two complementary modules: (i) a Semantic Reasoner, a fine-tuned LLM trained on a structured tabletop scene dataset to generate coarse layouts from input task instructions, and (ii) a Physics Corrector, a physics-aware flow-based denoising model that outputs pose updates to refine layouts, which ensures the physical plausibility of scenes while preserves semantic alignment with task instructions. STABLE adopts a progressive generation paradigm: by alternating between the Semantic Reasoner and Physics Corrector, it incrementally expands the scene from task-critical objects to background objects. Experiments demonstrate that STABLE successfully generates simulation-ready tabletop scenes that strictly conform to task instructions and significantly enhances the physical validity of scenes over prior art.
Zhen Luo, Yixuan Yang, Xudong Xu +5
1SII 2SUSTech · 2SUSTech · 3Shanghai AI Laboratory +4