Material Magic Wand: Material-Aware Grouping of 3D Parts in Untextured Meshes

Umangi Jain; Vladimir Kim; Matheus Gadelha; Igor Gilitschenski; Zhiqin Chen

cs.CVMar 18, 2026

Material Magic Wand: Material-Aware Grouping of 3D Parts in Untextured Meshes

Authors: Umangi Jain, Vladimir Kim, Matheus Gadelha, Igor Gilitschenski, Zhiqin Chen

Organizations: University of Toronto · Adobe Research

Abstract

We introduce the problem of material-aware part grouping in untextured meshes. Many real-world shapes, such as scales of pinecones or windows of buildings, contain repeated structures that share the same material but exhibit geometric variations. When assigning materials to such meshes, these repeated parts often require piece-by-piece manual identification and selection, which is tedious and time-consuming. To address this, we propose Material Magic Wand, a tool that allows artists to select part groups based on their estimated material properties -- when one part is selected, our algorithm automatically retrieves all other parts likely to share the same material. The key component of our approach is a part encoder that generates a material-aware embedding for each 3D part, accounting for both local geometry and global context. We train our model with a supervised contrastive loss that brings embeddings of material-consistent parts closer while separating those of different materials; therefore, part grouping can be achieved by retrieving embeddings that are close to the embedding of the selected part. To benchmark this task, we introduce a curated dataset of 100 shapes with 241 part-level queries. We verify the effectiveness of our method through extensive experiments and demonstrate its practical value in an interactive material assignment application.

Figures & tables

Figure 1: Material Magic Wand. Given an untextured 3D mesh (left) with existing part segmentation, which is often obtained by finding connected components of the mesh, a user can apply our tool to select a group of material-consistent parts by clicking on one single representative part. In the middle, we show example selections of petal , base , sepal , stem , leaf , and grass . For each selection, the tool automatically finds all other parts in the shape that are likely to share the same material (right) through geometric and contextual cues, accelerating the material assignment process. Colors may appear darker due to backface shading.

Figure 2: Method Overview. Left: Our view selection process renders each part with nearby context from multiple viewpoints sampled randomly over a hemisphere. We choose the one with minimal occlusion, $I^{ctx}$ , and use the same viewpoint to render the part in isolation, $I^{part}$ . $I^{full}$ captures the entire mesh. We highlight the part with a different color from the rest of the mesh. Right: For each part, its corresponding images are passed through an encoder and their embeddings are concatenated. During training, embeddings of parts with the same material are pulled together, while those with different materials are pushed apart.

Method	AUC PR	R-Prec	mAP	Recall@K				F1
				R@5	R@10	R@20	R@100
Histogram Matching [ 5 ]	26.85	25.45	30.71	4.55	8.89	15.97	45.64	23.84
SigLIP-v2 [ 43 ]	62.83	56.02	60.58	22.88	31.20	40.72	68.44	39.44
PartField [ 26 ]	75.30	67.74	70.52	26.88	37.12	47.92	71.31	56.57
DINO-v3 $\textit{small}^{\dagger}$ [ 41 ]	81.14	78.32	83.49	34.57	45.49	56.63	82.18	59.36
Ours	89.74	88.33	91.70	37.99	50.98	62.79	83.67	75.94

Table 1: Comparison of geometry-based, vision-embedding, and segmentation-based baselines on our material-aware grouping task. All metrics are macro-averaged and reported in percentage. Our method achieves consistent improvements across all evaluation metrics.

Figure 3: Ambiguities in raw material IDs. Using meshes from Objaverse directly for material grouping can introduce ambiguity in the testing benchmark due to artistic intent, noisy labeling, or coarse material assignment. For example, some shingles on the roof exhibit mixed materials, and fence or wall slats alternate between different material labels (left). To reduce such inconsistencies, we manually refine the material annotations to make the material assignment more uniform and fine-grained (right).

Figure 4: Precision–Recall curve. We sweep the similarity threshold to evaluate retrieval performance. Our method consistently maintains higher precision across all recall levels, achieving the highest AUC (89.7), followed by DINO (81.1).

Figure 5: Qualitative comparison. For each mesh, the red part denotes the query, and orange parts indicate the retrieved matches. Our method retrieves components that are both geometrically and contextually similar with the query, while baselines often miss structurally related parts or include visually similar but contextually incorrect ones.

Figure 6: Effect of changing the grouping tolerance parameter. The similarity threshold $\lambda$ controls the cut-off distance in the parts’ embedding space for part retrieval. With our method (right), at low values of $\lambda$ , only highly similar parts are selected; increasing $\lambda$ gradually expands the selection to include less similar, yet related parts. In contrast, baselines exhibit less stable behavior by either retrieving unrelated components at higher thresholds or lacking fine-grained control at lower thresholds.

Figure 7: Effect of multiple queries. Additional query parts improve retrieval by capturing a more complete set of desired components. The initial query (left) retrieves only a subset of sails, while adding an additional example (right) helps recover the rest.

Method	AUC	R-Prec	mAP	R@20
w/o isolated part	86.90	84.97	88.53	61.11
w/o part-with-context	87.30	85.37	88.91	61.09
w/o full-object	88.89	87.72	90.75	62.36
Only isolated part	86.18	81.88	85.88	59.87
Init from Dino-v2 L	86.51	84.58	88.61	61.13
Random initialization	78.54	74.52	79.62	56.25

Table 2: Ablation results. We examine the contribution of our design choices. Removing any component reduces performance, with the full model achieving the highest scores across all metrics.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Qualitative results of Histogram Matching approach. Histogram matching successfully retrieves near-identical components (e.g., duplicated branches or similar jewels) but fails to capture structurally similar yet non-identical parts, such as the rear car window or varied tree leaves.

Figure 9: Connected components as segments. Examples of part segmentation produced using connected components. Each color corresponds to a different part. $N$ denotes the number of parts in the mesh, illustrating the fine granularity of the decomposition.

Figure 10: Data sampling. (M: #materials, C: #segments, S: #sampled after balancing). Top-left (a): Within-mesh imbalance where one material (purple bricks) dominates. Top-right (b): Long-tail distribution with 76 materials but most appearing in few parts. Bottom-left (c): Insufficient samples requiring multi-view augmentation. Bottom-right (d): Mesh with 56K several near-duplicate parts requiring capping.

Inference	#Parts $\dagger$	Time(s) $\dagger$	AUC $\uparrow$	mAP $\uparrow$
No Dedup	265; 1144.2	48.91; 208.05	88.13	89.68
Dedup	60; 181.0	11.14; 33.01	89.74	91.70
$\dagger$ Reported numbers are (Median; IQR) over full test set.

Appendix

Table 3: Effect of deduplication on inference runtime.

Figure 11: Deduplication error when using similar pieces across categories (e.g., cuboid for stairs and fence ).

Method	AUC PR	R-Prec	mAP	Recall@K
				R@5	R@10	R@20	R@100
Dino v2 (small)	80.13	77.33	82.23	33.39	45.14	57.03	81.60
Dino v2 (base)	78.99	76.24	81.98	33.37	45.10	56.73	81.00
Dino v2 (large)	78.28	74.41	80.59	33.94	44.70	55.92	81.00
Dino v2 (giant)	78.30	76.51	81.78	33.57	45.15	56.29	80.47
Dino v3 (small)	81.14	78.32	83.49	34.57	45.49	56.63	82.18

Appendix

Table 4: Performance comparison of different DINO backbone variants on our material-aware part retrieval benchmark. Results show that scaling to larger models provides only marginal or inconsistent gains across metrics. All metrics in (%).

Figure 12: Additional qualitative results on Objaverse test set.

Figure 13: Results from Texverse. For each mesh in a row, the red part denotes the query and the orange parts indicate the retrieved matches. While baselines often miss non-identical instances or include unrelated parts, our method can retrieve material-consistent components (window frames, ship masts, stairs, and roof logs).

Figure 14: Geometric variation within a material. Given a query part (red), we show retrieved parts (left in each pair) versus ground truth (right in each pair). Left: Cable endpoints in the machinery exhibit different geometry from the cable body due to folding and knotting. Right: Bike cables near the down tube are tightly coiled, appearing geometrically distinct from the loosely suspended cable segments near the handlebars. Such cases can result in incomplete retrieval.

Figure 15: Geometric similarity across different materials. Given a query part (red), we show retrieved parts (left in each pair) versus ground truth (right in each pair). Left: Chess pieces have identical geometry across opposing teams, differing only in material. Right: Tail and wing feathers share similar geometric structure, leading to incorrect grouping despite different material assignments. These cases could be mitigated by training on more examples with similar contextual patterns.

Figure 16: Ambiguity arising from artistic intent. Given a query part (red), we show retrieved parts (left in each pair) versus ground truth (right in each pair). Left: Our method retrieves all pentagon-shaped cake toppings based on geometric and contextual similarity, but the actual assignments had different materials (shown in varying colors). Right: We retrieve all foliage when querying one leaf cluster, but the mesh contains intentional material variation across the tree. Such artistic choices represent inherent ambiguity that cannot be resolved from geometry and context.

3D Representation Learning

3D representation learning builds geometric features for recognizing, reconstructing, generating, and reasoning about objects and scenes.

See all 191 papers

PartMat: Material-Aware 3D Part Decomposition with a Single Global Latent
Aug 3, 2026Guangming Fu, Jin Song, Yiyun Fei +3Part-Level 3D Generation 3D Asset Generation
1Nankai University, China · 2Alibaba Group, China
MaPa: Text-driven Photorealistic Material Painting for 3D Shapes
Apr 26, 2024Shangzhan Zhang, Sida Peng, Tao Xu +73D Asset Generation Text-to-3D Generation
Zhejiang University · Ant Group · Shenzhen University
Point2Part: Unified 3D Partitioning from Point Prompts
Sep 29, 2026Hao-Tang Tsui, Yu-Rou Tuan, Xiaoxuan Ma +33D Part Segmentation
Carnegie Mellon University

Aug 3, 2026J/K move · Enter open · S save

PartMat: Material-Aware 3D Part Decomposition with a Single Global Latent

Guangming Fu, Jin Song, Yiyun Fei, Guoqiu Li+2

1Nankai University, China · 2Alibaba Group, China

Part-Level 3D Generation 3D Asset Generation 3D Part Segmentation 3D Representation Learning 3D Diffusion Models

Open paper AskPDF arXiv

Part-level 3D generation has recently attracted increasing attention for producing structured and editable 3D assets. However, existing methods typically decompose objects according to functional semantics rather than the editable material boundaries (e.g., fabric, wood, metal) required in practical 3D applications such as interior design. Additionally, current methods often generate parts independently, causing computational costs to scale linearly with the part count. To address these limitations, we present PartMat, an efficient material-aware 3D part decomposition pipeline that represents multi-part geometry with a single global latent. Given a reference image and a single whole-object geometry, PartMat decomposes the object into parts that follow material boundaries. First, we propose PartVAE to learn such a unified representation and decode all material parts in a single forward pass, thereby decoupling inference cost from the number of parts. Second, with this representation, a diffusion model is trained for part generation and refined via reinforcement learning for accurate material assignment and overlap suppression. Finally, to recover fine-grained geometric details, we introduce a sparse-voxel flow-matching model with part attention for geometry post-processing. Extensive experiments demonstrate that PartMat significantly outperforms existing baselines in material-aware decomposition accuracy and achieves comparable geometric quality, while maintaining efficient inference.

Abstract

3D Representation Learning

Explore similar work

PartMat: Material-Aware 3D Part Decomposition with a Single Global Latent

MaPa: Text-driven Photorealistic Material Painting for 3D Shapes

Point2Part: Unified 3D Partitioning from Point Prompts