Omni-3DEdit: Generalized Versatile 3D Editing in One-Pass
Organizations: The Hong Kong Polytechnic University
Abstract
Most instruction-driven 3D editing methods rely on 2D models to guide the explicit and iterative optimization of 3D representations. This paradigm, however, suffers from two primary drawbacks. First, it lacks a universal design of different 3D editing tasks because the explicit manipulation of 3D geometry necessitates task-dependent rules, e.g., 3D appearance editing demands inherent source 3D geometry, while 3D removal alters source geometry. Second, the iterative optimization process is highly time-consuming, often requiring thousands of invocations of 2D/3D updating. We present Omni-3DEdit, a unified, learning-based model that generalizes various 3D editing tasks implicitly. One key challenge to achieve our goal is the scarcity of paired source-edited multi-view assets for training. To address this issue, we construct a data pipeline, synthesizing a relatively rich number of high-quality paired multi-view editing samples. Subsequently, we adapt the pre-trained generative model SEVA as our backbone by concatenating source view latents along with conditional tokens in sequence space. A dual-stream LoRA module is proposed to disentangle different view cues, largely enhancing our model's representational learning capability. As a learning-based model, our model is free of the time-consuming online optimization, and it can complete various 3D editing tasks in one forward pass, reducing the inference time from tens of minutes to approximately two minutes. Extensive experiments demonstrate the effectiveness and efficiency of Omni-3DEdit.
Figures & tables
| Editing Type | CO3Dv2 [ 53 ] | DL3DV [ 41 ] | WildRGB-D [ 74 ] |
|---|---|---|---|
| 3D Addition | 15K | 7K | 6K |
| 3D Removal | 15K | 6K | 6K |
| 3D Appearance Editing | 13K | 5K | 5K |
| Methods | Carton | Cone | Cookie | Newcone | Plant | Skateboard | Sunflower | Average |
|---|---|---|---|---|---|---|---|---|
| SPIn-NeRF [ 48 ] | 16.659 / 0.539 | 15.438 / 0.389 | 11.879 / 0.521 | 17.131 / 0.519 | 16.850 / 0.401 | 15.645 / 0.675 | 23.538 / 0.206 | 16.734 / 0.464 |
| 2DGS + LaMa [ 29 , 57 ] | 16.433 / 0.499 | 15.591 / 0.351 | 11.711 / 0.538 | 16.598 / 0.670 | 14.491 / 0.564 | 15.520 / 0.639 | 23.024 / 0.194 | 16.195 / 0.494 |
| 2DGS + LeftRefill [ 29 , 5 ] | 15.157 / 0.567 | 16.143 / 0.372 | 12.458 / 0.526 | 16.717 / 0.677 | 12.856 / 0.666 | 16.429 / 0.634 | 24.216 / 0.181 | 16.282 / 0.518 |
| LeftRefill [ 5 ] | 14.667 / 0.560 | 14.933 / 0.380 | 11.148 / 0.519 | 16.264 / 0.448 | 16.183 / 0.463 | 14.912 / 0.572 | 18.851 / 0.331 | 15.280 / 0.468 |
| Gaussian Grouping [ 81 ] | 16.695 / 0.502 | 14.549 / 0.366 | 11.564 / 0.731 | 16.745 / 0.533 | 16.175 / 0.440 | 16.002 / 0.577 | 20.787 / 0.209 | 16.074 / 0.480 |
| GScream [ 66 ] | 14.687 / 0.587 | 14.655 / 0.476 | 12.733 / 0.429 | 13.662 / 0.605 | 16.238 / 0.437 | 12.941 / 0.626 | 18.470 / 0.436 | 14.758 / 0.514 |
| Method | PSNR | LPIPS | CLIP-T |
|---|---|---|---|
| ZeroNVS [ 56 ] | 14.56 | 0.716 | 0.196 |
| MVInpainter [ 6 ] | 19.20 | 0.344 | 0.271 |
| Omni-3DEdit (Ours) | 20.67 | 0.278 | 0.277 |
| Method | CLIP-T/I | CLIP-Dir. | Gemini score | Time |
|---|---|---|---|---|
| DGE [ 11 ] | 0.246 | 0.132 | 1.7 | 5min |
| GaussianEditor [ 13 ] | 0.253 | 0.146 | 2.0 | 17min |
| ViCANeRF [ 19 ] | 0.257 | 0.141 | 2.2 | 28min |
| Nano-banana [ 17 ] | 0.281 | 0.165 | 3.8 | - |
| Omni-3DEdit (Ours) | 0.286 | 0.170 | 4.0 | 2min |
| Settings | SSIM | PSNR | LPIPS |
|---|---|---|---|
| Omni-3DEdit | 0.925 | 17.72 | 0.395 |
| SEVA zeroshot | 0.911 | 13.99 | 0.575 |
| Omni-3DEdit w/o indicator | 0.917 | 15.20 | 0.545 |
| Omni-3DEdit w/o pose | 0.903 | 14.54 | 0.565 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| config | value |
|---|---|
| optimizer | AdamW |
| base learning rate | 1e-4 |
| weight decay | 0.01 |
| eps | 1e-8 |
| optimizer momentum | |
| batch size | 32 |
| Scene | Edit Instruction | Source Prompt | Target Prompt |
|---|---|---|---|
| Bear | “Turn it into a black bear.” | “A bear in the park.” | “A black bear in the park.” |
| Bear | “Remove the bear.” | “A bear in the park.” | “A plain stone in the park.” |
| Book | “Remove the book + add an apple.” | “A table with a book.” | “A table with an apple.” |
| Spinnerf_1 | “Remove the trash + add a poster.” | “A trash on the ground.” | “A poster.” |
| Spinnerf_2 | “Remove the box under tree.” | “A box under the tree.” | “A tree.” |
| CO3D_Keyboard | “Make the keyboard green + add a bottle.” | “A keyboard on the table.” | “A green keyboard and a bottle on the table.” |
3D Scene Editing
3D scene editing studies how to make controlled changes to objects, appearance, layout, and motion while keeping a scene coherent across views.
See all 88 papers