Chat-Edit-3D++: Interactive 3D and 4D Scene Editing via Large Language Models
Authors: Shuangkang Fang, Yufeng Wang, Yi-Hsuan Tsai, Wenrui Ding, Yi Yang, Shuchang Zhou, Ming-Hsuan Yang
Organizations: School of Electrical and Information Engineering, Beihang University, Beijing 100191, China. · Institute of Unmanned System, Beihang University, Beijing 100191, China. · Atmanity Inc., USA. · Megvii Research, Beijing 100190, China. · University of California at Merced, Merced.
Recent work on image content manipulation based on vision-language pre-training models has been effectively extended to text-driven 3D scene editing. However, existing schemes for 3D scene editing still have certain shortcomings, hindering their further development as interactive design tools. Such schemes typically adhere to fixed input patterns, limiting flexibility in text input. Furthermore, their editing capabilities are constrained by a single or a few 2D visual models and require intricate pipeline design to integrate these models into 3D reconstruction processes. To address the aforementioned issues, we propose the Hash-Atlas network, which reformulates 3D scene editing as operations on 2D atlas images, thereby achieving a workflow decoupling of the 2D editing and 3D reconstruction processes. Building on this foundation, we introduce a dialogue-based 3D scene editing approach, termed CE3D++, which is centered on a large language model (LLM) that allows arbitrary textual input from users and interprets their intentions, subsequently facilitating the autonomous invocation of the corresponding visual models. Additionally, we extend CE3D++ to monocular 4D scenes by imposing motion constraints on moving objects and further fine-tuning the LLM by creating a trajectory dataset related to editing tasks, which enables the smaller LLM to schedule up to 30 different visual tools accurately. Experimental results demonstrate that CE3D++ effectively integrates multiple visual models to achieve diverse visual editing effects, possessing strong scene comprehension and multi-round dialog capabilities. The source codes and trained models are available at https://github.com/Fangkang515/CE3D.
Figures & tables
Fig. 1 : Examples of chatting with CE3D ++ . We propose CE3D ++ , a novel paradigm for 3D and 4D scene editing, which is compatible with a variety of extant visual models. By managing these visual tools through the LLMs, we achieve challenging dialogue-based scene editing tasks that are difficult to accomplish with previous methods.
Fig. 2 : Differences between CE3D ++ and other typical text-driven editing frameworks . Existing methods (a)-(c) require complex integration of specific 2D and 3D models. While CE3D ++ (d) decouples the 2D editing process from the 3D representation, enhancing its compatibility with various visual models.
Figure 3
Fig. 5 : Illustration of the customized LLM agent . We show example prompts for configuring an LLM as an agent, including the linguistic management of multiple non-linguistic file formats within a scene, the specification of thought processes, and the integration of external visual tool invocation knowledge. Additionally, we construct a trajectory-tuning dataset for tool invocation, which is utilized to fine-tune the LLM, thereby enhancing its comprehension of complex instructions and improving its reliability. We simplify some text to fit into the page. Please refer to our code for the complete prompts.
Figure 5
Fig. 7 : Effectiveness of pivot-motion loss . For comparison, we enlarge the object in the foreground atlas.
Fig. 8 : Multi-round dialogue editing results . CE3D ++ performs editing of 3D scenes on the 2D atlases, thus facilitating broad compatibility with a wide range of existing 2D models and supporting diverse editing capabilities . Moreover, by utilizing LLMs for the management of various visual models and parsing user text inputs, CE3D ++ enables users to engage in more flexible text inputs and easily achieve multi-round dialogue.
Fig. 9 : Comparison with other 3D and 4D scene editing methods . Existing state-of-the-art methods exhibit limited editing capability. In contrast, the proposed CE3D ++ provides broad compatibility with a wide range of 2D visual models, enabling the integration of diverse editing functionalities .
Fig. 10 : Comparison with IN2N [ 13 ] for text-query diversity and multi-round dialogue .
Fig. 11 : Visualization of multi-layer atlas . We extend Hash-Atlas to a three-layer setting. Compared with LNA, our method produces atlas representations with fewer visual artifacts.
3D scene
LLFF
CE3D-collect
NeRF-Art
4D scene
DyCheck
DynamicNeRF
CLIP Similarity ↑
CLIP Similarity ↑
EditSplat [ 114 ]
0.188
0.212
0.201
CTRL-D [ 81 ]
0.186
0.194
RoMaP [ 115 ]
0.224
0.240
0.229
Instruct-4DGS [ 116 ]
0.223
0.251
DN2N [ 14 ]
0.193
0.233
0.218
Dynamic-eDiTor [ 117 ]
0.236
0.267
CE3D ++
0.304
0.352
0.321
CE3D ++
0.313
0.320
CLIP Directional Score ↑
CLIP Directional Score ↑
TABLE II : Quantitative comparisons of editing results.
Fig. 12 : Success rate of editing tasks for different large language models . We evaluate the success rate of LLMs in accurately invoking external visual tools across 100 diverse complex user instructions. By fine-tuning Qwen-14B and LLaMA-8B with our curated trajectory data (shown in green ), we observe a significant improvement in success rates, demonstrating the effectiveness of this approach in enhancing the robustness of LLM-based agents.
Fig. 13 : Comparison of LLM performance. ‘SR’ denotes the success rate under multiple editing instructions. Qwen-14B-finetuned successfully invokes external tools to complete complex multi-step editing tasks.
Fig. 14 : Qualitative ablation studies of CE3D ++ . The red box marks failed results.
w/o Linit
w/o Lrecpro
w/o Lmotion
w/o Excutor
w/o SSN
Full
LLFF dataset
CLIP ↑
0.242
0.267
−
0.190
×
0.304
CDS ↑
0.163
0.179
−
0.154
×
0.192
CE3D-collect dataset
CLIP ↑
0.286
0.263
−
0.249
×
0.352
CDS ↑
0.214
0.197
−
0.201
×
0.248
TABLE III : Quantitative ablation studies of CE3D ++ . Here “ × ” indicates that the model variant fails to provide edited results, and “ − ” denotes that the evaluation is not applicable. The CLIP-S calculates the similarity between edited results and text. CDS means CLIP Directional Score [ 13 ] , which measures how much the change in text captions agrees with the change in images.
Category
Sub-Category
Original
Few-shot
Trajectory-tuned
Type
Understanding
2%
1%
0%
Generation
5%
4%
1%
Enhancement
7%
5%
3%
Conditional Editing
17%
14%
7%
I/O
I2T
2%
1%
0%
T2I
5%
4%
1%
TABLE IV : Fine-grained tool-calling failure-rate for Qwen-14B model under different settings.
Atlas resolution
PSNR ↑
Video length
PSNR ↑
Pivot-view robustness
PSNR ↑
250 × 250
25.17
10
33.64
first frame
28.02
500 × 500
27.59
30
32.35
median frame
27.93
1000 × 1000
28.21
60
29.82
last frame
27.86
1500 × 1500
28.35
120
27.46
Pivot frame (ours)
28.21
Moving object count
PSNR ↑
Tracking quality
PSNR ↑
Complexity
PSNR ↑
1
28.85
Low
25.32
Level 1
29.32
TABLE V : Impact of different factors on 4D atlas quality .
Fig. 15 : Failure cases . (a) In 360-degree scenes, atlas distortions may reduce compatibility with 2D visual models, potentially leading to failed editing results. (b) Under rapid non-rigid deformations or severe occlusions, the tracking model may fail to maintain consistent trajectories, resulting in artifacts in the generated atlas.