CoSMo3D: Open-World Promptable 3D Semantic Part Segmentation through LLM-Guided Canonical Spatial Modeling

Li Jin; Weikai Chen; Yujie Wang; Yingda Yin; Zeyu Hu; Runze Zhang; Keyang Luo; Shengju Qian

cs.CVMar 1, 2026

CoSMo3D: Open-World Promptable 3D Semantic Part Segmentation through LLM-Guided Canonical Spatial Modeling

Authors: Li Jin, Weikai Chen, Yujie Wang, Yingda Yin, Zeyu Hu, Runze Zhang, Keyang Luo, Shengju Qian,

Organizations: SDU · LIGHTSPEED · UNC Chapel Hill

Abstract

Open-world promptable 3D semantic segmentation remains brittle as semantics are inferred in the input sensor coordinates. Yet, humans, in contrast, interpret parts via functional roles in a canonical space -- wings extend laterally, handles protrude to the side, and legs support from below. Psychophysical evidence shows that we mentally rotate objects into canonical frames to reveal these roles. To fill this gap, we propose \methodName{}, which attains canonical space perception by inducing a latent canonical reference frame learned directly from data. By construction, we create a unified canonical dataset through LLM-guided intra- and cross-category alignment, exposing canonical spatial regularities across 200 categories. By induction, we realize canonicality inside the model through a dual-branch architecture with canonical map anchoring and canonical box calibration, collapsing pose variation and symmetry into a stable canonical embedding. This shift from input pose space to canonical embedding yields far more stable and transferable part semantics. Experimental results show that \methodName{} establishes new state of the art in open-world promptable 3D segmentation.

Figures & tables

Figure 2: We propose a dual-branch framework for open-world promptable 3D semantic segmentation: the feature extraction branch encodes 3D shape features (via Point Transformer) and text semantic features (via SigLIP) to enable cross-modal part segmentation. A training-only canonical embedding branch then enforces consistent canonical space perception via semantic contrastive alignment, canonical map anchoring, and canonical box calibration losses, ensuring robust reasoning across any shape in any pose.

Figure 3: (a) Prior works perform category-level canonicalization, aligning intra-category shapes but neglecting cross-category consistency. (b) We cluster categories via LLM and align different categories relying on key semantic parts and functional consistency.

Figure 4: Handling Symmetric Objects for Canonical Map Anchoring. (a) For symmetric shapes, multiple valid poses induce ambiguous point-wise canonical labels, making direct point-wise supervision unreliable. (b) Prior works rely on category-specific processing with manual symmetry annotation, limiting open-world scalability. (c) We instead apply an order-invariant set loss on RGB-encoded canonical coordinates that matches the overall layout of semantic parts in canonical space while remaining robust to symmetric pose ambiguities.

mIoU (%)	3D feedforward	3Dcompat-Coarse				3DCompat-Fine
		Canonical		Rotated		Canonical		Rotated
		{Part} of {Obj.}	{Part}	{Part} of {Obj.}	{Part}	{Part} of {Obj.}	{Part}	{Part} of {Obj.}	{Part}
PointCLIPV2	$\times$	14.09	14.16	13.17	13.39	7.25	7.18	7.43	7.20
PartSLIP++	$\times$	6.12	17.62	5.89	15.82	3.79	5.68	3.41	4.76
Find3D	✓	31.72	25.36	32.47	26.23	12.69	12.57	12.73	12.79
Find3D ∗	✓	36.89	45.00	38.14	46.02	17.72	24.97	18.57	26.35

Table 1: Quantitative evaluation for promptable semantic segmentation (with semantic labels as reference and mean IoU reported). Top: coarse-grained dataset and fine-grained dataset; Bottom: ShapeNet-Part dataset and PartNet-E dataset. Find3D ∗ denotes retraining using the 3D data we constructed; PartSLIP++ ∗ denotes fine-tuning on the PartNet-E dataset with canonical views as input.

Figure 5: Qualitative comparison of promptable 3D part segmentation. Across challenging cases (similar geometry with different semantics, noise-prone objects, cross-category semantics, and arbitrary poses), our method produces more accurate and consistent part localizations than existing baselines.

Variant	Hard-Negative Sampling	Canon. Map Anchoring	Cross-Cat. Canon. (Data)	Canon. Box Calibration	Canonical		Rotated
Variant	Hard-Negative Sampling	Canon. Map Anchoring	Cross-Cat. Canon. (Data)	Canon. Box Calibration	$\{\text{Part}\}$ of $\{\text{Obj.}\}$	$\{\text{Part}\}$	$\{\text{Part}\}$ of $\{\text{Obj.}\}$	$\{\text{Part}\}$
A					36.89	45.00	38.14	46.02
B	✓				38.30	48.30	39.28	48.76
C	✓	✓			41.71	52.01	42.48	52.57
D	✓	✓	✓		43.34	54.15	42.63	53.70
Full Model	✓	✓	✓	✓	47.51	54.52	47.74	54.55

Table 2: Quantitative results of the ablation study. All values are mIoU scores computed over parts from all instances across all categories.

Figure 6: Shape feature analysis. (a) PartField [ 10 ] yields clear part partitions but lacks cross-shape and cross-pose consistency. (b) Find3D [ 13 ] improves consistency but produces blurry part boundaries. (c) Our method achieves both clear part separation and consistent features across shapes and poses.

3D Part Segmentation

3D part segmentation divides objects and scenes into meaningful components using geometry, appearance, language, and motion cues.

See all 36 papers

3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models
Jun 18, 2026Jintang Xue, Xinyu Wang, Yixing Wu +23D Representation Learning Multimodal Foundation Models
University of Southern California Los Angeles, California, USA · Ohio State University Columbus, Ohio, USA
PartLLM: A Unified Multimodal Foundation for 3D Part Segmentation
Sep 22, 2026Zhe Zhu, Yiheng Zhang, Peng Li +93D Part Segmentation 3D VLMs
Taiyuan University of Technology, Tencent Visvise, China · Hong Kong University of Science and Technology, Tencent Visvise, China · Taiyuan University of Technology, China +6
CANIS: Generation-Assisted 3D Canonicalization via an Image-Semantic Bridge
Aug 7, 2026Kendong Liu, Yuxin Yao, Junhui Hou3D Scene Understanding
The authors are with the Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China

Jun 18, 2026J/K move · Enter open · S save

3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models

Jintang Xue, Xinyu Wang, Yixing Wu, Jingwen Chen+1

University of Southern California Los Angeles, California, USA · Ohio State University Columbus, Ohio, USA

3D Representation Learning Multimodal Foundation Models 3D VLMs 3D Visual Grounding Multimodal Understanding

Open paper AskPDF arXiv

3D multimodal large language models (3D MLLMs) describe a 3D object as a whole but cannot address, name, or reason about its parts. Prior part-aware attempts add segmentation decoders, heavier 3D encoders, or bounding-box grammars at substantial parameter cost. We take a fundamentally different path: we reorganize the input token stream so that parts become directly addressable through the LLM's own vocabulary. Our model, 3D-PLOT-LLM, partitions the frozen point encoder's patches into K locally coherent regions and inserts, before each region's patch tokens, a learnable per-region marker and a reserved vocabulary token <part_k>; a Marker-Space Refinement (MSR) module then conditions each marker on its region's spatial statistics and adjacency neighbors. The model thus cites parts in its output and follows prompts that refer to parts by token, a capability absent from prior object-level 3D MLLMs. To probe this interface, we construct PartVerse-QA, a vocabulary-level part-QA benchmark adapted from PartVerse mesh annotations (77K training pairs and 588 held-out queries on disjoint object splits), on which 3D-PLOT-LLM reaches caption-to-slots Jaccard 0.459 and Exact-match 13.78%, with a slot-to-caption GPT-4o judge of 44.68. On the 3DCoMPaT-GrIn part-aware grounded description benchmark, 3D-PLOT-LLM outperforms PointLLM, Kestrel, PARIS3D, and SegPoint on every text-output metric, and ShapeLLM on 3 of 4, with up to +3.03 GPT-4o judge over PointLLM. On Objaverse whole-object captioning, adding PartVerse-QA at Stage 2 yields +0.65 SBERT and +1.85 GPT-4o over PointLLM, and tops PointLLM-PiSA on 4 of 5 traditional metrics (SBERT, SimCSE, BLEU-1, METEOR) despite targeting a different (part-grounded) objective. All with under 1M new trainable parameters on a frozen point encoder, an order of magnitude below prior part-aware 3D MLLMs, and no segmentation decoder or bounding-box head.

Abstract

3D Part Segmentation

Explore similar work

3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models

PartLLM: A Unified Multimodal Foundation for 3D Part Segmentation

CANIS: Generation-Assisted 3D Canonicalization via an Image-Semantic Bridge