SceneAssistant: A Visual Feedback Agent for Open-Vocabulary 3D Scene Generation

Jun Luo; Jiaxiang Tang; Ruijie Lu; Gang Zeng

cs.CVMar 12, 2026

SceneAssistant: A Visual Feedback Agent for Open-Vocabulary 3D Scene Generation

Authors: Jun Luo, Jiaxiang Tang, Ruijie Lu, Gang Zeng

Organizations: Peking University · NVIDIA

Abstract

Text-to-3D scene generation from natural language is highly desirable for digital content creation. However, existing methods are largely domain-restricted or reliant on predefined spatial relationships, limiting their capacity for unconstrained, open-vocabulary 3D scene synthesis. In this paper, we introduce SceneAssistant, a visual-feedback-driven agent designed for open-vocabulary 3D scene generation. Our framework leverages modern 3D object generation model along with the spatial reasoning and planning capabilities of Vision-Language Models (VLMs). To enable open-vocabulary scene composition, we provide the VLMs with a comprehensive set of atomic operations (e.g., Scale, Rotate, FocusOn). At each interaction step, the VLM receives rendered visual feedback and takes actions accordingly, iteratively refining the scene to achieve more coherent spatial arrangements and better alignment with the input text. Experimental results demonstrate that our method can generate diverse, open-vocabulary, and high-quality 3D scenes. Both qualitative analysis and quantitative human evaluations demonstrate the superiority of our approach over existing methods. Furthermore, our method allows users to instruct the agent to edit existing scenes based on natural language commands. Our code is available at https://github.com/ROUJINN/SceneAssistant

Figures & tables

Figure 1: Open-vocabulary text-to-3D scene generation via SceneAssistant. Each scene exhibits high fidelity to long-tail objects and intricate spatial constraints described in the text.

Figure 2: The SceneAssistant framework and its iterative scene generation process. Our approach utilizes a vision-feedback-driven closed loop (bottom left) where a VLM agent processes multimodal context—including rendered views, scene metadata, and system messages—to generate reasoning and actions following the ReAct [ 42 ] paradigm. This framework enables an iterative refinement process (top), progressively building complex scenes from step $1$ to $T$ . As highlighted in steps $8$ and $9$ (bottom right), the agent can dynamically respond to system-generated collision warnings, performing corrective manipulations to resolve spatial inconsistencies and achieve a high-quality final 3D layout.

Action Name	Arguments	Description
Create	name , description	Generate new object
Duplicate	name , count	Duplicate specified objects
Delete	name	Delete objects
Translate	name , axis , distance	Relative translation
Place	name , position	Absolute positioning
Rotate	name , axis , angle_degrees	Set rotation angle (absolute)

Table 1: Comprehensive suite of Action APIs. We define a functionally complete set of primitives for asset management, 6-DoF spatial manipulation, and visual perception control. These APIs abstract low-level engine operations into semantically intuitive commands, enabling the VLM agent to autonomously orchestrate 3D scene synthesis.

Figure 3: Human-agent collaboration for interactive scene editing. SceneAssistant effectively interprets user-provided message and produces the corresponding scene.

Figure 4: Qualitative comparison of indoor scene generation. We compare SceneAssistant with Holodeck [ 41 ] and SceneWeaver [ 40 ] across various indoor categories. SceneAssistant performs precise spatial arrangements and faithfully reconstructs nuanced objects often simplified or omitted by existing baselines.

Figure 5: Qualitative results of open-vocabulary scene generation. SceneAssistant consistently produces superior spatial layouts and semantic consistency.

Method	Layout Correctness $\uparrow$	Object Quality $\uparrow$	Preference $\uparrow$
Holodeck [ 41 ]	4.475	4.763	6.25%
SceneWeaver [ 40 ]	5.800	6.150	36.25%
SceneAssistant	6.888	6.950	61.25%
NoActionAPI	7.005	6.591	35.91%
NoVisFeedback	6.255	5.673	26.82%
SceneAssistant	7.600	7.277	65.00%

Table 2: Quantitative comparison across different scene categories . The upper section reports the average scores for 8 indoor scenes, comparing SceneAssistant against state-of-the-art baselines. The lower section presents results for 22 open-vocabulary scenes under various configurations. Layout and Object Quality are rated on a 1–10 scale.

Figure 6: Ablation study. The left side compares SceneAssistant with NoVisualPrompt variant, where the absence of visual grounding leads to disorganized layouts. The right side illustrates the NoCollisionCheck variant, with red circles highlighting physical interpenetrations that the agent fails to resolve without explicit feedback.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1 : Gallery of rendered images for scenes generated by SceneAssistant in indoor scenarios.

Figure S2 : Gallery of rendered images for scenes generated by SceneAssistant in open-vocabulary scenarios.

Figure S3 : Detailed reasoning and action trajectory of SceneAssistant terminating at the maximum step limit ( $T=20$ ).

Figure S4 : Detailed reasoning and action trajectory of SceneAssistant terminating via the Finish action.

3D Scene Generation

3D scene generation creates structured, explorable environments from text, images, maps, or other inputs.

See all 190 papers

ThinkBLOX: 3D Indoor Scene Generation with Progressive Reasoning
Jul 15, 2026Yuan Xiao, Can Wang, Xiangyu Kong +13D Scene Editing 3D Scene Generation
Department of Computer Science, City University of Hong Kong, Hong Kong · School of Computing and Data Science, University of Hong Kong · School of Computer Science, Beijing Information Science and Technology University
ReSpace: Text-Driven Autoregressive 3D Indoor Scene Synthesis and Editing
Jun 3, 2025Martin JJ. Bucher, Iro Armeni3D Scene Editing 3D Scene Representation
Stanford University
HOG-Layout: Hierarchical 3D Scene Generation, Optimization and Editing via Vision-Language Models
Apr 12, 2026Haiyan Jiang, Deyu Zhang, Dongdong Weng +23D Scene Generation 3D Scene Layout Generation
The Hong Kong Polytechnic University · Beijing Institute of Technology

Jul 15, 2026J/K move · Enter open · S save

ThinkBLOX: 3D Indoor Scene Generation with Progressive Reasoning

Yuan Xiao, Can Wang, Xiangyu Kong, Jing Liao

Department of Computer Science, City University of Hong Kong, Hong Kong · School of Computing and Data Science, University of Hong Kong · School of Computer Science, Beijing Information Science and Technology University

3D Scene Editing 3D Scene Generation 3D Scene Layout Generation Indoor Scene Generation

Open paper AskPDF arXiv

While traditional graphics methods often synthesize 3D indoor scenes autoregressively or hierarchically, recent vision-language model (VLM)-based generators predominantly adopt a one-shot paradigm where the full layout is planned at once. This one-shot approach often requires global re-optimization or complete reconstruction during interactive editing (e.g., inserting or moving objects) and can lead to physically or semantically poorly organized arrangements. To address these challenges, we propose ThinkBLOX, a VLM-based progressive reasoning framework that iteratively designs and refines 3D scenes. ThinkBLOX treats layout generation as a state-conditioned, step-by-step reasoningand-action process. To power this, we construct the ThinkBLOX-Data-200K dataset, containing 224,757 procedural placement pairs annotated with multi-view scene context, explicit Chain-of-Thought (CoT) rationales, and structured JSON layouts. Through supervised fine-tuning (SFT) on this dataset, the VLM learns to bridge the reasoning-action gap under incremental updates. Furthermore, recognizing that scene synthesis is inherently a multisolution task where SFT suffers from reward conflict, we introduce Tier-Decoupled GDPO. This reinforcement learning scheme organizes heterogeneous rewards into distinct tiers, stabilizing policy optimization across physical validity, semantic plausibility, and reasoning-action consistency. Extensive experiments show that ThinkBLOX significantly outperforms recent one-shot and iterative baselines in physical plausibility, semantic alignment, and interactive editability. Additionally, we show that it supports diverse applications, including both global and local generation and rearrangement of 3D scenes.

Abstract

3D Scene Generation

Explore similar work

ThinkBLOX: 3D Indoor Scene Generation with Progressive Reasoning

ReSpace: Text-Driven Autoregressive 3D Indoor Scene Synthesis and Editing

HOG-Layout: Hierarchical 3D Scene Generation, Optimization and Editing via Vision-Language Models