DesignAgent3D: Interactive 3D Scene Editing via Designer-like Multimodal Reasoning

Xiujin Liu; Tianyu Yang; Yilun Zhao; Xiangliang Zhang

cs.CVAug 18, 2026

DesignAgent3D: Interactive 3D Scene Editing via Designer-like Multimodal Reasoning

Authors: Xiujin Liu, Tianyu Yang, Yilun Zhao, Xiangliang Zhang

Organizations: University of Notre Dame South Bend, Indiana, USA · Yale University New Haven, Connecticut, USA

Abstract

Text guided 3D scene editing provides an intuitive interface for modifying reconstructed environments, but remains difficult because natural language design requests are often semantically underspecified and must be grounded in cluttered 3D scenes. Existing methods typically formulate the task as one-shot conditional generation from a single prompt, failing to resolve ambiguous user intents or achieve precise spatial grounding. Consequently, they suffer from severe object localization drift, tracking failure under occlusions, and the notorious multi-view "sticker effect." To overcome these limitations, we present DesignAgent3D, an interactive multimodal agentic framework that reformulates 3D scene editing as a designer-like Plan-Perceive-Act paradigm. The agent first plans by interacting with the user to clarify underspecified design goals, then perceives by grounding the intended edit to specific objects or regions in the 3D scene, and finally acts by applying controlled visual modifications while preserving scene consistency. The edits are further integrated into the underlying 3D representation, supporting persistent and multi-view consistent novel-view rendering. Extensive experiments across both NeRF and 3D Gaussian Splatting backbones demonstrate that DesignAgent3D significantly outperforms state-of-the-art baselines, delivering superior semantic intent alignment, impeccable spatial localization accuracy, and high-fidelity multi-view consistency.

Figures & tables

Figure 1. Conceptual comparison between traditional one-shot 3D editing methods and DesignAgent3D. Traditional approaches rely on direct and one-shot prompt-to-scene optimization, frequently suffering from object localization drift or background corruption due to semantic underspecification. In contrast, DesignAgent3D adopts a designer-like Plan–Perceive–Act paradigm that clarifies ambiguous intents via human-in-the-loop collaboration and leverages geometric spatial reasoning, ensuring multi-view consistent and precisely intent-aligned 3D editing results.

Figure 2. Overview of the DesignAgent3D Pipeline. The top row illustrates our agentic Plan–Perceive–Act workflow, where the agent clarifies the user request, generates a structured design protocol, grounds the target object and texture, and updates the 3D scene through geometry-constrained repainting. The bottom row details the three core modules: Geometric Anchored Segmentation for view-consistent object masks, Texture Reference Generation for a user-verified material prior, and Geometry-Constrained Repainting for multi-view consistent edited images.

Figure 3. Comparison of 3D Scene Editing Methods. While existing methods often suffer from global color shifts (e.g., Instruct-NeRF2NeRF ( 6 ) , GaussianEditor ( 4 ) ) or fail to edit any object (e.g., Instruct-GS2GS ( 22 ) , EditSplat ( 10 ) ), our agentic framework precisely grounds abstract prompts into specific material updates. Moreover, our method maintains sharp object boundaries and superior cross-view consistency, whereas baselines introduce significant "floaters" or blurred textures.

Figure 4. Comparison of 3D Scene Editing Methods (Continue). Our agentic framework precisely grounds abstract prompts into specific material updates.

Figure 5. Case Studies. While baselines "guess" vague intents or fail to distinguish identical objects, our framework uses conversational dialogue and a geometrically anchored segmentation module to clarify goals and confirm target instances via visual candidates.

Scene Edit	Nerfstudio - Poster		Nerfstudio - Kitchen		Nerfstudio - Stump		Mip-Nerf 360 - Room		Mip-Nerf 360 - Counter		Replica - Office0		MEAN
	CLIP ${}_{\text{dir}}$	CLIP ${}_{\text{sim}}$	CLIP ${}_{\text{dir}}$	CLIP ${}_{\text{sim}}$	CLIP ${}_{\text{dir}}$	CLIP ${}_{\text{sim}}$	CLIP ${}_{\text{dir}}$	CLIP ${}_{\text{sim}}$	CLIP ${}_{\text{dir}}$	CLIP ${}_{\text{sim}}$	CLIP ${}_{\text{dir}}$	CLIP ${}_{\text{sim}}$	CLIP ${}_{\text{dir}}$	CLIP ${}_{\text{sim}}$
Instruct-NeRF2NeRF ( 6 )	-0.023	0.229	0.020	0.266	-0.060	0.272	-0.064	0.318	-0.066	0.236	0.061	0.246	-0.022	0.261
Instruct-GS2GS ( 22 )	0.000	0.290	0.000	0.294	0.000	0.263	0.000	0.312	0.000	0.289	0.036	0.238	0.006	0.281
EditSplat ( 10 )	-0.076	0.242	-0.003	0.248	0.008	0.233	0.005	0.313	0.004	0.315	0.055	0.235	-0.001	0.264
GaussianEditor ( 4 )	0.000	0.147	-0.007	0.243	0.038	0.252	0.005	0.301	-0.003	0.276	0.039	0.180	0.012	0.233
Ours (Nerf)	0.115	0.239	0.076	0.256	0.017	0.216	0.027	0.314	0.067	0.296	0.126	0.306	0.071	0.271

Table 1. Quantitative Comparison. CLIP

{}_{\text{dir}}

: CLIP directional similarity; CLIP

{}_{\text{sim}}

: CLIP Text-Image Similarity

Figure 6. Spatial and geometric integrity evaluation. We visualize the bounding boxes of target objects before and after editing. Our method maintains a near-perfect IoU, confirming that the editing process does not alter object’s original position or spatial footprint within the 3D scene. The high fidelity of the depth maps post-edit proves that our framework effectively preserves the underlying 3D geometry while performing complex texture and material modifications.

Figure 7. User Study. The chart compares our framework (Ours) against three baseline models across five key metrics. Each bar represents the average score (0–5). Our method consistently outperforms all baselines, demonstrating particular strength in specialized reasoning tasks while maintaining high competitiveness in rendering indicators.

Figure 8. Ablation Studies. Evaluate the necessity of our core components: (a) Plan: Without interactive loop, the model fails to ground abstract concepts, resulting in irrelevant style shifts. (b) Geometry Anchored Segmentation: Replacing our 3D-aware module with 2D-only methods leads to inaccurate masks (e.g., over-exclusion) and floaters. (c) Texture Reference: Our dual-stream strategy ensures superior cross-view consistency compared to single-prompt baselines, which suffer from flickering and appearance drifting.

3D Scene Editing

3D scene editing studies how to make controlled changes to objects, appearance, layout, and motion while keeping a scene coherent across views.

See all 88 papers

Chat-Edit-3D++: Interactive 3D and 4D Scene Editing via Large Language Models
Aug 29, 2026Shuangkang Fang, Yufeng Wang, Yi-Hsuan Tsai +43D Scene Editing Text-Guided 3D Scene Editing
School of Electrical and Information Engineering, Beihang University, Beijing 100191, China. · Institute of Unmanned System, Beihang University, Beijing 100191, China. · Atmanity Inc., USA. +2
Vinedresser3D: Agentic Text-guided 3D Editing
Feb 23, 2026Yankuan Chi, Xiang Li, Zixuan Huang +1Text-Guided 3D Scene Editing 3D Asset Editing
The Hong Kong University of Science and Technology · University of Illinois Urbana-Champaign
Edit-As-Act: Goal-Regressive Planning for Open-Vocabulary 3D Indoor Scene Editing
Mar 18, 2026Seongrae Noh, SeungWon Seo, Gyeong-Moon Park +13D Scene Editing Text-Guided 3D Scene Editing
Korea University

Aug 29, 2026J/K move · Enter open · S save

Chat-Edit-3D++: Interactive 3D and 4D Scene Editing via Large Language Models

Shuangkang Fang, Yufeng Wang, Yi-Hsuan Tsai, Wenrui Ding+3

School of Electrical and Information Engineering, Beihang University, Beijing 100191, China. · Institute of Unmanned System, Beihang University, Beijing 100191, China. · Atmanity Inc., USA.+2

3D Scene Editing Text-Guided 3D Scene Editing LLM Tool Use 3D Scene Representation

Open paper AskPDF arXiv

Recent work on image content manipulation based on vision-language pre-training models has been effectively extended to text-driven 3D scene editing. However, existing schemes for 3D scene editing still have certain shortcomings, hindering their further development as interactive design tools. Such schemes typically adhere to fixed input patterns, limiting flexibility in text input. Furthermore, their editing capabilities are constrained by a single or a few 2D visual models and require intricate pipeline design to integrate these models into 3D reconstruction processes. To address the aforementioned issues, we propose the Hash-Atlas network, which reformulates 3D scene editing as operations on 2D atlas images, thereby achieving a workflow decoupling of the 2D editing and 3D reconstruction processes. Building on this foundation, we introduce a dialogue-based 3D scene editing approach, termed CE3D++, which is centered on a large language model (LLM) that allows arbitrary textual input from users and interprets their intentions, subsequently facilitating the autonomous invocation of the corresponding visual models. Additionally, we extend CE3D++ to monocular 4D scenes by imposing motion constraints on moving objects and further fine-tuning the LLM by creating a trajectory dataset related to editing tasks, which enables the smaller LLM to schedule up to 30 different visual tools accurately. Experimental results demonstrate that CE3D++ effectively integrates multiple visual models to achieve diverse visual editing effects, possessing strong scene comprehension and multi-round dialog capabilities. The source codes and trained models are available at https://github.com/Fangkang515/CE3D.

Abstract

3D Scene Editing

Explore similar work

Chat-Edit-3D++: Interactive 3D and 4D Scene Editing via Large Language Models

Vinedresser3D: Agentic Text-guided 3D Editing

Edit-As-Act: Goal-Regressive Planning for Open-Vocabulary 3D Indoor Scene Editing