Most instruction-driven 3D editing methods rely on 2D models to guide the explicit and iterative optimization of 3D representations. This paradigm, however, suffers from two primary drawbacks. First, it lacks a universal design of different 3D editing tasks because the explicit manipulation of 3D geometry necessitates task-dependent rules, e.g., 3D appearance editing demands inherent source 3D geometry, while 3D removal alters source geometry. Second, the iterative optimization process is highly time-consuming, often requiring thousands of invocations of 2D/3D updating. We present Omni-3DEdit, a unified, learning-based model that generalizes various 3D editing tasks implicitly. One key challenge to achieve our goal is the scarcity of paired source-edited multi-view assets for training. To address this issue, we construct a data pipeline, synthesizing a relatively rich number of high-quality paired multi-view editing samples. Subsequently, we adapt the pre-trained generative model SEVA as our backbone by concatenating source view latents along with conditional tokens in sequence space. A dual-stream LoRA module is proposed to disentangle different view cues, largely enhancing our model's representational learning capability. As a learning-based model, our model is free of the time-consuming online optimization, and it can complete various 3D editing tasks in one forward pass, reducing the inference time from tens of minutes to approximately two minutes. Extensive experiments demonstrate the effectiveness and efficiency of Omni-3DEdit.
Figures & tables
Figure 1: Motivation of Omni-3DEdit. (a) 3D editing via iterative 2D-3D-2D optimization with explicit 3D representation lacks generality and is time-consuming. (b) Performing 3D editing in latent space is hard to handle scene-level assets with arbitrary viewpoints. (c) Our Omni-3DEdit aims to solve these issues in multi-view space to perform fast, general, and consistent editing.
Figure 2: Overview of Omni-3DEdit. Given the instruction and multi-view images as inputs, we first employ Qwen-Image to obtain an edited reference image as condition view. Then an OmniNet is trained to map the editing cues from condition view to other views. The outputs of OmniNet are edited multi-view images, which can be used to obtain the edited 3D asset optionally.
Figure 3: Data Construction Pipeline. The original multi-view images are passed through a four-stage pipeline to obtain their paired multi-view counterparts after editing. The pipeline covers tasks of 3D removal, addition, and appearance editing.
Editing Type
CO3Dv2 [ 53 ]
DL3DV [ 41 ]
WildRGB-D [ 74 ]
3D Addition
15K
7K
6K
3D Removal
15K
6K
6K
3D Appearance Editing
13K
5K
5K
Table 1: Statistics of paired edited multi-views in our curated training set, categorized by dataset and edit type.
Methods
Carton
Cone
Cookie
Newcone
Plant
Skateboard
Sunflower
Average
SPIn-NeRF [ 48 ]
16.659 / 0.539
15.438 / 0.389
11.879 / 0.521
17.131 / 0.519
16.850 / 0.401
15.645 / 0.675
23.538 / 0.206
16.734 / 0.464
2DGS + LaMa [ 29 , 57 ]
16.433 / 0.499
15.591 / 0.351
11.711 / 0.538
16.598 / 0.670
14.491 / 0.564
15.520 / 0.639
23.024 / 0.194
16.195 / 0.494
2DGS + LeftRefill [ 29 , 5 ]
15.157 / 0.567
16.143 / 0.372
12.458 / 0.526
16.717 / 0.677
12.856 / 0.666
16.429 / 0.634
24.216 / 0.181
16.282 / 0.518
LeftRefill [ 5 ]
14.667 / 0.560
14.933 / 0.380
11.148 / 0.519
16.264 / 0.448
16.183 / 0.463
14.912 / 0.572
18.851 / 0.331
15.280 / 0.468
Gaussian Grouping [ 81 ]
16.695 / 0.502
14.549 / 0.366
11.564 / 0.731
16.745 / 0.533
16.175 / 0.440
16.002 / 0.577
20.787 / 0.209
16.074 / 0.480
GScream [ 66 ]
14.687 / 0.587
14.655 / 0.476
12.733 / 0.429
13.662 / 0.605
16.238 / 0.437
12.941 / 0.626
18.470 / 0.436
14.758 / 0.514
Table 2: Quantitative comparison (PSNR ↑ / LPIPS ↓ ) of 360∘ 3D removal methods on the 360-USID dataset.
Figure 4: Qualitative comparisons to 3D removal methods. Our Omni-3DEdit not only removes the specific object completely but also presents rich details in the removed regions compared to other methods. We center crop views to adapt OmniNet resolution (white box).
Method
PSNR ↑
LPIPS ↓
CLIP-T ↑
ZeroNVS [ 56 ]
14.56
0.716
0.196
MVInpainter [ 6 ]
19.20
0.344
0.271
Omni-3DEdit (Ours)
20.67
0.278
0.277
Table 3: Quantitative comparison on CO3Dv2 val set.
Figure 5: Comparison of 3D appearance editing. Red box represents the reference view. White boxes are object masks as additional inputs for MVInpainter.
Method
CLIP-T/I
CLIP-Dir.
Gemini score
Time
DGE [ 11 ]
0.246
0.132
1.7
5min
GaussianEditor [ 13 ]
0.253
0.146
2.0
17min
ViCANeRF [ 19 ]
0.257
0.141
2.2
28min
Nano-banana [ 17 ]
0.281
0.165
3.8
-
Omni-3DEdit (Ours)
0.286
0.170
4.0
2min
Table 4: Comparison of methods on complex 3D editing.
Figure 6: Results of competing 3D editing methods for a complex task “Removing the book.” then “Adding an apple to desk.”
Figure 7: Ablation study of different architectures on performing multiview editing with edited reference view. Both plain feature-space concatenation and sequence-space concatenation fail to achieve desired editing results, while dual-stream LoRA improves the editing quality significantly. We provide edited multi-view here, and the edited 3D assets can be found in the Supplementary File .
Settings
SSIM ↑
PSNR ↑
LPIPS ↓
Omni-3DEdit
0.925
17.72
0.395
SEVA zeroshot
0.911
13.99
0.575
Omni-3DEdit w/o indicator
0.917
15.20
0.545
Omni-3DEdit w/o pose
0.903
14.54
0.565
Table 5: Ablation study on 360-USID benchmark.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1 : System prompts for instruction generation.
Figure S2 : Comparison of per-view editing and consistent refinement. Consistent refinement eliminates the minor inconsistencies that occur on per-view appearance editing.
Figure S3 : System prompts for quality filter. Firstly, evaluation is conducted on per source-edited views to ensure that the instruction has been successfully executed. Then, a subjectively multi-view consistent evaluation is conducted to filter cases with obvious inconsistency.
Figure S4 : Visualization of successful cases and failed cases. We present several training data on the left and some failed cases on the right, which are caused by the inconsistent editing and incomplete views.
config
value
optimizer
AdamW
base learning rate
1e-4
weight decay
0.01
eps
1e-8
optimizer momentum
β1,β2=0.9,0.999
batch size
32
Appendix
Table S1 : Training settings and hyperparameters.
Figure S5 : Attention visualization of OmniNet Block during different denoising stages. Omni-3DEdit can capture corresponding regions from source and conditional views based on the given camera poses. Pentagram is placed on the query regions.
Scene
Edit Instruction
Source Prompt
Target Prompt
Bear
“Turn it into a black bear.”
“A bear in the park.”
“A black bear in the park.”
Bear
“Remove the bear.”
“A bear in the park.”
“A plain stone in the park.”
Book
“Remove the book + add an apple.”
“A table with a book.”
“A table with an apple.”
Spinnerf_1
“Remove the trash + add a poster.”
“A trash on the ground.”
“A poster.”
Spinnerf_2
“Remove the box under tree.”
“A box under the tree.”
“A tree.”
CO3D_Keyboard
“Make the keyboard green + add a bottle.”
“A keyboard on the table.”
“A green keyboard and a bottle on the table.”
Appendix
Table S2 : The scene and instruction prompts in the datasets.
Beihang University, China · Beihang University, China and Beijing Key Laboratory of Intelligent Creative Content Generation and Immersive Experience, China · Beihang University, China and VAST, China