Existing 3D editing methods have made notable progress in controllability, yet they remain limited in several important ways. Most approaches rely on text-driven editing, which struggles to express fine-grained visual changes intended by the user. Moreover, many methods require manually supplied 3D masks or introduce unintended changes to regions that should remain untouched. These limitations largely arise from the absence of fine-grained semantic understanding, making it difficult for existing models to retrieve or modify specific 3D components. We introduce ES3D, a framework that embeds semantics directly into 3D space, enabling component-aware retrieval and editing of a 3D asset conditioned on multiple local reference images and optional text queries. We first construct a 3D semantic embedding by projecting multi-view semantic features into the voxelized space of the asset. We then perform 3D component retrieval by computing feature similarity between the 3D semantic embedding and the semantic embeddings of image or text queries. For editing, we employ a pretrained 3D generative model with an inpainting mechanism to modify the retrieved components guided by user-provided images while preserving the rest of the asset. Overall, ES3D is a 3D editing framework that retrieves editable regions based on semantic cues and uses multiple images as conditions. Extensive experiments demonstrate that ES3D produces geometrically consistent and semantically coherent edits, enabling robust image-based and text-assisted control for 3D editing.
Figures & tables
Figure 1: ES3D is capable of editing 3D asset conditioned on multiple local reference images.
Figure 2: Difference between Fuse3D and ES3D in 3D retrieval. (a) Fuse3D retrieves 3D regions by leveraging cross-attention module inside pretrained image-to-3D models, but this mechanism breaks when the reference image only covers a partial view of the full object. (b) ES3D constructs a voxel-level semantic embedding and performs cluster-level 3D component retrieval, enabling robust localization from either local reference images or text queries.
Figure 3: Given a 3D asset along with multiple local reference images and optional text, we first construct a 3D semantic embedding on the voxel grid. We then perform clustering on this embedding and retrieve the target component using image or text queries. Finally, a unified conditional token, produced by multi-masked-image feature extraction module, is injected into a pretrained 3D generative model together with a refined inpainting mask to obtain the final edited 3D asset.
Figure 4: Editing quality comparison with previous methods.
Figure 5: Ablation study of Semantic Feature Enhancement Module and 3D Component Cluster & Retrieval Module.
Metric
Ours
IP-Adapter
TRELLIS
Fuse3D
CLIP ↑
32.19
29.02
29.38
30.29
Reg.-Spec. CLIP ↑
77.94
73.63
72.61
76.16
ImageReward ↑
0.5853
0.0198
0.2501
0.1738
CD keep ( ×10−3 ) ↓
0.91
13.12
11.25
9.87
3D BBox IoU ↑
0.73
N/A
N/A
0.39
Table 1: Quantitative comparison of different methods.
Method
V. Q.
E. A.
P. C.
vs. Fuse3D
59
53
67
vs. IP-Adapter
98
92
91
vs. TRELLIS
62
57
53
Table 2: GPTEvals3D results (% preferred) for 3D editing across different methods. Scores above 50 indicate that our method is preferred.
Method
V. Q.
E. A.
P. C.
Ours
4.4
4.6
4.8
Fuse3D
4.1
3.9
4.1
IP-Adapter
2.8
3.1
2.1
TRELLIS
3.9
3.3
4.7
Table 3: User ratings (1–5) for different methods.
Figure 6: More results of edited 3d assets.
Figure 7: Ablation study of Refined Inpainting Mask and Stacked Image strategy.
Method
CLIP ↑
Reg.-Spec. CLIP ↑
ImageReward ↑
Ours
32.19
77.94
0.5853
w/o Semantic Feature Enhancement
30.02
72.10
0.2399
w/o 3D Component Cluster Retrieval
30.15
71.40
0.1764
w/o Refined Inpainting Mask
30.78
76.23
0.3264
w/o 3D Stacked Image
27.33
72.53
0.2291
Table 4: Quantitative comparison of ablation study.