Organizations: Guangdong Provincial Key Laboratory of Visual Media and Multidimensional Intelligence, CSSE, Shenzhen University, China · NetEase, China · Nanyang Technological University, Singapore
Recognizing geometric texture patterns on 3D models is critical for applications such as heritage protection, model editing, and 3D printing. Existing 3D model segmentation methods predominantly focus on identifying high-level semantic parts, thereby failing to capture middle-level textures. While recent Vision-Language Models can perform image segmentation with appropriate text prompts, they still struggle to recognize 3D geometric textures. In this work, we present a method that segments 3D mesh models based on middle-level geometric textures. We further introduce a new geometric texture segmentation benchmark and fine-tune SAM3 accordingly. The segmented texture regions are then clustered based on DINO feature similarity and lifted from 2D to 3D. Additionally, we develop an interactive tool to enable both 3D model editing based on our segmentation results and model retrieval from the dataset based on geometric and texture similarity. Experiments demonstrate that our method significantly outperforms existing methods in segmenting geometric textures and can facilitate various applications.
Figures & tables
Figure 1: High-quality 3D digital models with intricate geometric textures obtained from scanning (left) and manual sculpting (right).
Figure 2: The generation pipeline of our DTSet dataset and GTSeg benchmark.
Figure 3: Overview of our 3D geometric texture segmentation framework.
Figure 4: The intra-mask consistency on two masks of the Merlion example. For each view, we visualize the curvature map (left) with its candidate mask marked in red. The token-to-centroid cosine similarity map is provided on the right. The value of cosine similarity is visualized from red (high) to blue (low).
Figure 5: Visualization of mask clustering in the token-distance embedding space. Nodes denote mask instances, with sizes proportional to mask areas and colors indicating final clusters. Positions are obtained by metric MDS [ BG05 ] from the pairwise distance: 1−sim(a,b) . Edges represent valid mutual- k NN connections, with thicker edges indicating higher similarity.
Method
MUCov ↑
MWCov ↑
SQ 50 ↑
RQ 50 ↑
PQ 50 ↑
w/o SAM3 Fine-tuning
35.64
49.04
81.66
34.57
29.15
w/o Clustering
39.91
54.26
77.96
41.03
32.79
w/o Filtering
54.33
71.08
79.18
59.87
48.55
Ours
61.78
78.24
83.15
70.99
59.32
Table 1: Ablation study of our instance segmentation pipeline. All variants are evaluated using class-agnostic, face-level metrics, and all values are reported as percentages.
Figure 6: Comparisons of the segmentation results with TextureSAM, SAM3, and our GTSAM on five rendered views of reconstructed real objects.
Method
mIoU ↑
Dice ↑
Precision ↑
Recall 50 ↑
AP 50 ↑
AP 75 ↑
Mean AP ↑
TextureSAM
15.61
25.87
15.67
5.73
0.70
0.17
0.43
SAM3
59.33
69.03
72.62
36.93
29.76
17.03
23.40
Ours (GTSAM)
78.18
85.74
88.90
81.42
74.50
52.72
62.06
Table 2: Quantitative comparison on segmentation of 2D images.
Method
MUCoV ↑
Recall 50 ↑
TextureSAM + Voting
18.13
13.43
ShapeDiameter
35.18
19.80
SAMesh
32.35
21.33
Ours
61.78
70.54
Table 3: Quantitative comparison on segmentation of 3D models.
Figure 7: Comparisons of the segmentation results with Shape Diameter, SAMesh, and our method on synthetic 3D models from SHREC’25 [ PTB24 ] .
Figure 8: Comparisons of the segmentation results with Shape Diameter, SAMesn, and our method on reconstructed real objects. Colors of instances are assigned randomly.
Figure 9: Comparisons of the segmentation results with Shape Diameter, SAMesh, and our method on reconstructed real objects. Colors of instances are assigned randomly.
Textured 3D meshes jointly represent geometry, topology, and appearance, yet their irregular structure poses significant challenges for deep-learning-based semantic segmentation. While a few recent methods operate directly on meshes without imposing geometric constraints, they typically overlook the rich textural information also provided by such meshes. We introduce a texture-aware transformer that learns directly from raw pixels associated with each mesh face, coupled with a new hierarchical learning scheme for multi-scale feature aggregation. A texture branch summarizes all face-level pixels into a learnable token, which is fused with geometrical descriptors and processed by a stack of Two-Stage Transformer Blocks (TSTB), which allow for both a local and a global information flow. We evaluate our model on the Semantic Urban Meshes (SUM) benchmark and a newly curated cultural-heritage dataset comprising textured roof tiles with triangle-level annotations for damage types. Our method achieves 81.9% mF1 and 94.3% OA on SUM and 49.7% mF1 and 72.8% OA on the new dataset, substantially outperforming existing approaches.
Mohammadreza Heidarianbaei, Max Mehltretter, Franz Rottensteiner
Institute of Photogrammetry and GeoInformation, Leibniz University Hannover, Germany
Open-vocabulary 3D semantic segmentation aims to segment arbitrary categories beyond the training set. Existing methods predominantly rely on distilling knowledge from 2D open-vocabulary models. However, aligning 3D features to the 2D representation space restricts intrinsic 3D geometric learning and inherits errors from 2D predictions. To address these limitations, we propose GeoGuide, a novel framework that leverages pretrained 3D models to integrate hierarchical geometry-semantic consistency for open-vocabulary 3D segmentation. Specifically, we introduce an Uncertainty-based Superpoint Distillation module to fuse geometric and semantic features for estimating per-point uncertainty, adaptively weighting 2D features within superpoints to suppress noise while preserving discriminative information to enhance local semantic consistency. Furthermore, our Instance-level Mask Reconstruction module leverages geometric priors to enforce semantic consistency within instances by reconstructing complete instance masks. Additionally, our Inter-Instance Relation Consistency module aligns geometric and semantic similarity matrices to calibrate cross-instance consistency for same-category objects, mitigating viewpoint-induced semantic drift. Extensive experiments on ScanNet v2, Matterport3D, and nuScenes demonstrate the superior performance of GeoGuide.
Xujing Tao, Chuxin Wang, Yubo Ai +8
University of Science and Technology of China · National Key Laboratory of Deep Space Exploration, Deep Space Exploration Laboratory
3D editing has emerged as a critical research area to provide users with flexible control over 3D assets. While current editing approaches predominantly focus on 3D Gaussian Splatting or multi-view images, the direct editing of 3D meshes remains underexplored. Prior attempts, such as VoxHammer, rely on voxel-based representations that suffer from limited resolution and necessitate labor-intensive 3D mask. To address these limitations, we propose VecSet-Edit, the first pipeline that leverages the high-fidelity VecSet Large Reconstruction Model (LRM) as a backbone for mesh editing. Our approach is grounded on a analysis of the spatial properties in VecSet tokens, revealing that token subsets govern distinct geometric regions. Based on this insight, we introduce Mask-guided Token Seeding and Attention-aligned Token Gating strategies to precisely localize target regions using only 2D image conditions. Also, considering the difference between VecSet diffusion process versus voxel we design a Drift-aware Token Pruning to reject geometric outliers during the denoising process. Finally, our Detail-preserving Texture Baking module ensures that we not only preserve the geometric details of original mesh but also the textural information. More details can be found in our project page: https://github.com/BlueDyee/VecSet-Edit/tree/main