Existing 3D editing methods have made notable progress in controllability, yet they remain limited in several important ways. Most approaches rely on text-driven editing, which struggles to express fine-grained visual changes intended by the user. Moreover, many methods require manually supplied 3D masks or introduce unintended changes to regions that should remain untouched. These limitations largely arise from the absence of fine-grained semantic understanding, making it difficult for existing models to retrieve or modify specific 3D components. We introduce ES3D, a framework that embeds semantics directly into 3D space, enabling component-aware retrieval and editing of a 3D asset conditioned on multiple local reference images and optional text queries. We first construct a 3D semantic embedding by projecting multi-view semantic features into the voxelized space of the asset. We then perform 3D component retrieval by computing feature similarity between the 3D semantic embedding and the semantic embeddings of image or text queries. For editing, we employ a pretrained 3D generative model with an inpainting mechanism to modify the retrieved components guided by user-provided images while preserving the rest of the asset. Overall, ES3D is a 3D editing framework that retrieves editable regions based on semantic cues and uses multiple images as conditions. Extensive experiments demonstrate that ES3D produces geometrically consistent and semantically coherent edits, enabling robust image-based and text-assisted control for 3D editing.
Figures & tables
Figure 1: ES3D is capable of editing 3D asset conditioned on multiple local reference images.
Figure 2: Difference between Fuse3D and ES3D in 3D retrieval. (a) Fuse3D retrieves 3D regions by leveraging cross-attention module inside pretrained image-to-3D models, but this mechanism breaks when the reference image only covers a partial view of the full object. (b) ES3D constructs a voxel-level semantic embedding and performs cluster-level 3D component retrieval, enabling robust localization from either local reference images or text queries.
Figure 3: Given a 3D asset along with multiple local reference images and optional text, we first construct a 3D semantic embedding on the voxel grid. We then perform clustering on this embedding and retrieve the target component using image or text queries. Finally, a unified conditional token, produced by multi-masked-image feature extraction module, is injected into a pretrained 3D generative model together with a refined inpainting mask to obtain the final edited 3D asset.
Figure 4: Editing quality comparison with previous methods.
Figure 5: Ablation study of Semantic Feature Enhancement Module and 3D Component Cluster & Retrieval Module.
Metric
Ours
IP-Adapter
TRELLIS
Fuse3D
CLIP ↑
32.19
29.02
29.38
30.29
Reg.-Spec. CLIP ↑
77.94
73.63
72.61
76.16
ImageReward ↑
0.5853
0.0198
0.2501
0.1738
CD keep ( ×10−3 ) ↓
0.91
13.12
11.25
9.87
3D BBox IoU ↑
0.73
N/A
N/A
0.39
Table 1: Quantitative comparison of different methods.
Method
V. Q.
E. A.
P. C.
vs. Fuse3D
59
53
67
vs. IP-Adapter
98
92
91
vs. TRELLIS
62
57
53
Table 2: GPTEvals3D results (% preferred) for 3D editing across different methods. Scores above 50 indicate that our method is preferred.
Method
V. Q.
E. A.
P. C.
Ours
4.4
4.6
4.8
Fuse3D
4.1
3.9
4.1
IP-Adapter
2.8
3.1
2.1
TRELLIS
3.9
3.3
4.7
Table 3: User ratings (1–5) for different methods.
Figure 6: More results of edited 3d assets.
Figure 7: Ablation study of Refined Inpainting Mask and Stacked Image strategy.
Method
CLIP ↑
Reg.-Spec. CLIP ↑
ImageReward ↑
Ours
32.19
77.94
0.5853
w/o Semantic Feature Enhancement
30.02
72.10
0.2399
w/o 3D Component Cluster Retrieval
30.15
71.40
0.1764
w/o Refined Inpainting Mask
30.78
76.23
0.3264
w/o 3D Stacked Image
27.33
72.53
0.2291
Table 4: Quantitative comparison of ablation study.
Text-guided 3D editing aims to modify existing 3D assets using natural-language instructions. Current methods struggle to jointly understand complex prompts, automatically localize edits in 3D, and preserve unedited content. We introduce Vinedresser3D, an agentic framework for high-quality text-guided 3D editing that operates directly in the latent space of a native 3D generative model. Given a 3D asset and an editing prompt, Vinedresser3D uses a multimodal large language model to infer rich descriptions of the original asset, identify the edit region and edit type (addition, modification, deletion), and generate decomposed structural and appearance-level text guidance. The agent then selects an informative view and applies an image editing model to obtain visual guidance. Finally, an inversion-based rectified-flow inpainting pipeline with an interleaved sampling module performs editing in the 3D latent space, enforcing prompt alignment while maintaining 3D coherence and unedited regions. Experiments on diverse 3D edits demonstrate that Vinedresser3D outperforms prior baselines in both automatic metrics and human preference studies, while enabling precise, coherent, and mask-free 3D editing.
Yankuan Chi, Xiang Li, Zixuan Huang +1
The Hong Kong University of Science and Technology · University of Illinois Urbana-Champaign
High-fidelity semantic 3D scene representations are crucial for numerous applications, including robotics, autonomous driving, and simulation. Beyond this, the ability to edit such representations enables developers to adapt these applications more easily to specific target scenarios. Current approaches provide limited support for controllable editing. We introduce TASE, a method that projects pretrained 2D semantic features into a truncation-aware embedding space to enable flexible 3D scene editing. Our method explicitly optimizes a feature space in which progressively reducing feature channels yields increasingly abstract semantic representations, while retaining more channels preserves fine-grained detail. Additionally, we improve multi-view consistency of the features using a scale- and translation-equivariance loss. The resulting truncation-aware embedding space enables text-driven edits to 3D scenes, providing explicit control over how strongly edits adhere to the original scene content and allowing more substantial modifications than prior methods. Moreover, we propose a finetuning stage for the editing diffusion model to mitigate artifacts caused by geometric changes. Experimental results demonstrate competitive performance in 3D scene editing, substantially outperforming prior methods on edits involving large geometric modifications.
Tim-Felix Faasch, Jochen Kall, Lucas Nunes +2
Bosch Research, Hildesheim, Germany · Rheinisch-Westfälische Technische Hochschule Aachen, Aachen, Germany · University of Bonn, Bonn, Germany
3D editing refers to the ability to apply local or global modifications to 3D assets. Effective 3D editing requires maintaining semantic consistency by performing localized changes according to prompts, while also preserving local invariance so that unchanged regions remain consistent with the original. However, existing approaches have significant limitations: multi-view editing methods incur losses when projecting back to 3D, while voxel-based editing is constrained in both the regions that can be modified and the scale of modifications. Moreover, the lack of sufficiently large editing datasets for training and evaluation remains a challenge. To address these challenges, we propose a Beyond Voxel 3D Editing (BVE) framework with a self-constructed large-scale dataset specifically tailored for 3D editing. Building upon this dataset, our model enhances a foundational image-to-3D generative architecture with lightweight, trainable modules, enabling efficient injection of textual semantics without the need for expensive full-model retraining. Furthermore, we introduce an annotation-free 3D masking strategy to preserve local invariance, maintaining the integrity of unchanged regions during editing. Extensive experiments demonstrate that BVE achieves superior performance in generating high-quality, text-aligned 3D assets, while faithfully retaining the visual characteristics of the original input.
Yizhao Xu, Hongyuan Zhu, Caiyun Liu +6
The Peking University · Microsoft AI · HKUST(GZ) +1