Chat-Edit-3D++: Interactive 3D and 4D Scene Editing via Large Language Models
Authors: Shuangkang Fang, Yufeng Wang, Yi-Hsuan Tsai, Wenrui Ding, Yi Yang, Shuchang Zhou, Ming-Hsuan Yang
Organizations: School of Electrical and Information Engineering, Beihang University, Beijing 100191, China. · Institute of Unmanned System, Beihang University, Beijing 100191, China. · Atmanity Inc., USA. · Megvii Research, Beijing 100190, China. · University of California at Merced, Merced.
Recent work on image content manipulation based on vision-language pre-training models has been effectively extended to text-driven 3D scene editing. However, existing schemes for 3D scene editing still have certain shortcomings, hindering their further development as interactive design tools. Such schemes typically adhere to fixed input patterns, limiting flexibility in text input. Furthermore, their editing capabilities are constrained by a single or a few 2D visual models and require intricate pipeline design to integrate these models into 3D reconstruction processes. To address the aforementioned issues, we propose the Hash-Atlas network, which reformulates 3D scene editing as operations on 2D atlas images, thereby achieving a workflow decoupling of the 2D editing and 3D reconstruction processes. Building on this foundation, we introduce a dialogue-based 3D scene editing approach, termed CE3D++, which is centered on a large language model (LLM) that allows arbitrary textual input from users and interprets their intentions, subsequently facilitating the autonomous invocation of the corresponding visual models. Additionally, we extend CE3D++ to monocular 4D scenes by imposing motion constraints on moving objects and further fine-tuning the LLM by creating a trajectory dataset related to editing tasks, which enables the smaller LLM to schedule up to 30 different visual tools accurately. Experimental results demonstrate that CE3D++ effectively integrates multiple visual models to achieve diverse visual editing effects, possessing strong scene comprehension and multi-round dialog capabilities. The source codes and trained models are available at https://github.com/Fangkang515/CE3D.
Figures & tables
Fig. 1 : Examples of chatting with CE3D ++ . We propose CE3D ++ , a novel paradigm for 3D and 4D scene editing, which is compatible with a variety of extant visual models. By managing these visual tools through the LLMs, we achieve challenging dialogue-based scene editing tasks that are difficult to accomplish with previous methods.
Fig. 2 : Differences between CE3D ++ and other typical text-driven editing frameworks . Existing methods (a)-(c) require complex integration of specific 2D and 3D models. While CE3D ++ (d) decouples the 2D editing process from the 3D representation, enhancing its compatibility with various visual models.
Figure 3
Fig. 5 : Illustration of the customized LLM agent . We show example prompts for configuring an LLM as an agent, including the linguistic management of multiple non-linguistic file formats within a scene, the specification of thought processes, and the integration of external visual tool invocation knowledge. Additionally, we construct a trajectory-tuning dataset for tool invocation, which is utilized to fine-tune the LLM, thereby enhancing its comprehension of complex instructions and improving its reliability. We simplify some text to fit into the page. Please refer to our code for the complete prompts.
Figure 5
Fig. 7 : Effectiveness of pivot-motion loss . For comparison, we enlarge the object in the foreground atlas.
Fig. 8 : Multi-round dialogue editing results . CE3D ++ performs editing of 3D scenes on the 2D atlases, thus facilitating broad compatibility with a wide range of existing 2D models and supporting diverse editing capabilities . Moreover, by utilizing LLMs for the management of various visual models and parsing user text inputs, CE3D ++ enables users to engage in more flexible text inputs and easily achieve multi-round dialogue.
Fig. 9 : Comparison with other 3D and 4D scene editing methods . Existing state-of-the-art methods exhibit limited editing capability. In contrast, the proposed CE3D ++ provides broad compatibility with a wide range of 2D visual models, enabling the integration of diverse editing functionalities .
Fig. 10 : Comparison with IN2N [ 13 ] for text-query diversity and multi-round dialogue .
Fig. 11 : Visualization of multi-layer atlas . We extend Hash-Atlas to a three-layer setting. Compared with LNA, our method produces atlas representations with fewer visual artifacts.
3D scene
LLFF
CE3D-collect
NeRF-Art
4D scene
DyCheck
DynamicNeRF
CLIP Similarity ↑
CLIP Similarity ↑
EditSplat [ 114 ]
0.188
0.212
0.201
CTRL-D [ 81 ]
0.186
0.194
RoMaP [ 115 ]
0.224
0.240
0.229
Instruct-4DGS [ 116 ]
0.223
0.251
DN2N [ 14 ]
0.193
0.233
0.218
Dynamic-eDiTor [ 117 ]
0.236
0.267
CE3D ++
0.304
0.352
0.321
CE3D ++
0.313
0.320
CLIP Directional Score ↑
CLIP Directional Score ↑
TABLE II : Quantitative comparisons of editing results.
Fig. 12 : Success rate of editing tasks for different large language models . We evaluate the success rate of LLMs in accurately invoking external visual tools across 100 diverse complex user instructions. By fine-tuning Qwen-14B and LLaMA-8B with our curated trajectory data (shown in green ), we observe a significant improvement in success rates, demonstrating the effectiveness of this approach in enhancing the robustness of LLM-based agents.
Fig. 13 : Comparison of LLM performance. ‘SR’ denotes the success rate under multiple editing instructions. Qwen-14B-finetuned successfully invokes external tools to complete complex multi-step editing tasks.
Fig. 14 : Qualitative ablation studies of CE3D ++ . The red box marks failed results.
w/o Linit
w/o Lrecpro
w/o Lmotion
w/o Excutor
w/o SSN
Full
LLFF dataset
CLIP ↑
0.242
0.267
−
0.190
×
0.304
CDS ↑
0.163
0.179
−
0.154
×
0.192
CE3D-collect dataset
CLIP ↑
0.286
0.263
−
0.249
×
0.352
CDS ↑
0.214
0.197
−
0.201
×
0.248
TABLE III : Quantitative ablation studies of CE3D ++ . Here “ × ” indicates that the model variant fails to provide edited results, and “ − ” denotes that the evaluation is not applicable. The CLIP-S calculates the similarity between edited results and text. CDS means CLIP Directional Score [ 13 ] , which measures how much the change in text captions agrees with the change in images.
Category
Sub-Category
Original
Few-shot
Trajectory-tuned
Type
Understanding
2%
1%
0%
Generation
5%
4%
1%
Enhancement
7%
5%
3%
Conditional Editing
17%
14%
7%
I/O
I2T
2%
1%
0%
T2I
5%
4%
1%
TABLE IV : Fine-grained tool-calling failure-rate for Qwen-14B model under different settings.
Atlas resolution
PSNR ↑
Video length
PSNR ↑
Pivot-view robustness
PSNR ↑
250 × 250
25.17
10
33.64
first frame
28.02
500 × 500
27.59
30
32.35
median frame
27.93
1000 × 1000
28.21
60
29.82
last frame
27.86
1500 × 1500
28.35
120
27.46
Pivot frame (ours)
28.21
Moving object count
PSNR ↑
Tracking quality
PSNR ↑
Complexity
PSNR ↑
1
28.85
Low
25.32
Level 1
29.32
TABLE V : Impact of different factors on 4D atlas quality .
Fig. 15 : Failure cases . (a) In 360-degree scenes, atlas distortions may reduce compatibility with 2D visual models, potentially leading to failed editing results. (b) Under rapid non-rigid deformations or severe occlusions, the tracking model may fail to maintain consistent trajectories, resulting in artifacts in the generated atlas.
Text guided 3D scene editing provides an intuitive interface for modifying reconstructed environments, but remains difficult because natural language design requests are often semantically underspecified and must be grounded in cluttered 3D scenes. Existing methods typically formulate the task as one-shot conditional generation from a single prompt, failing to resolve ambiguous user intents or achieve precise spatial grounding. Consequently, they suffer from severe object localization drift, tracking failure under occlusions, and the notorious multi-view "sticker effect." To overcome these limitations, we present DesignAgent3D, an interactive multimodal agentic framework that reformulates 3D scene editing as a designer-like Plan-Perceive-Act paradigm. The agent first plans by interacting with the user to clarify underspecified design goals, then perceives by grounding the intended edit to specific objects or regions in the 3D scene, and finally acts by applying controlled visual modifications while preserving scene consistency. The edits are further integrated into the underlying 3D representation, supporting persistent and multi-view consistent novel-view rendering. Extensive experiments across both NeRF and 3D Gaussian Splatting backbones demonstrate that DesignAgent3D significantly outperforms state-of-the-art baselines, delivering superior semantic intent alignment, impeccable spatial localization accuracy, and high-fidelity multi-view consistency.
Xiujin Liu, Tianyu Yang, Yilun Zhao +1
University of Notre Dame South Bend, Indiana, USA · Yale University New Haven, Connecticut, USA
Text-guided 3D editing aims to modify existing 3D assets using natural-language instructions. Current methods struggle to jointly understand complex prompts, automatically localize edits in 3D, and preserve unedited content. We introduce Vinedresser3D, an agentic framework for high-quality text-guided 3D editing that operates directly in the latent space of a native 3D generative model. Given a 3D asset and an editing prompt, Vinedresser3D uses a multimodal large language model to infer rich descriptions of the original asset, identify the edit region and edit type (addition, modification, deletion), and generate decomposed structural and appearance-level text guidance. The agent then selects an informative view and applies an image editing model to obtain visual guidance. Finally, an inversion-based rectified-flow inpainting pipeline with an interleaved sampling module performs editing in the 3D latent space, enforcing prompt alignment while maintaining 3D coherence and unedited regions. Experiments on diverse 3D edits demonstrate that Vinedresser3D outperforms prior baselines in both automatic metrics and human preference studies, while enabling precise, coherent, and mask-free 3D editing.
Yankuan Chi, Xiang Li, Zixuan Huang +1
The Hong Kong University of Science and Technology · University of Illinois Urbana-Champaign
High-quality 3D scene reconstruction has recently advanced toward generalizable feed-forward architectures, enabling the generation of complex environments in a single forward pass. However, despite their strong performance in static scene perception, these models remain limited in responding to dynamic human instructions, which restricts their use in interactive applications. Existing editing methods typically rely on a 2D-lifting strategy, where individual views are edited independently and then lifted back into 3D space. This indirect pipeline often leads to blurry textures and inconsistent geometry, as 2D editors lack the spatial awareness required to preserve structure across viewpoints. To address these limitations, we propose VGGT-Edit, a feed-forward framework for text-conditioned native 3D scene editing. VGGT-Edit introduces depth-synchronized text injection to align semantic guidance with the backbone's spatial poses, ensuring stable instruction grounding. This semantic signal is then processed by a residual transformation head, which directly predicts 3D geometric displacements to deform the scene while preserving background stability. To ensure high-fidelity results, we supervise the framework with a multi-term objective function that enforces geometric accuracy and cross-view consistency. We also construct the DeltaScene Dataset, a large-scale dataset generated through an automated pipeline with 3D agreement filtering to ensure ground-truth quality. Experiments show that VGGT-Edit substantially outperforms 2D-lifting baselines, producing sharper object details, stronger multi-view consistency, and near-instant inference speed. The project page is https://chriszkxxx.github.io/VGGT-Edit/.
Kaixin Zhu, Yiwen Tang, Yifan Yang +9
1Peking University · 2Tencent · 3The Chinese University of Hong Kong +4