Motivated by a problem in Generative Specification-Producing AIs (SPAIs), we focus on a problem of deep clustering of three-dimensional shapes specified by point clouds. After reviewing clustering and deep clustering, we study the fully unsupervised pipeline in which a FoldingNet autoencoder is first trained for reconstruction and then K-means++ is then applied to the resulting codewords. We report clustering accuracy (ACC), normalized mutual information (NMI) and adjusted Rand index (ARI) on ModelNet10 and ModelNet40 datasets, as a function of the number of clusters K. We also provide t-SNE visualizations of the codewords, and discuss the choice of K.
Figures & tables
Figure 1: Illustrative example of clustering scenario with K=3 clusters.
Figure 2: Some example ModelNet shapes and their labeled category.
Figure 3: Training of the FoldingNet autoencoder on ModelNet10.
Figure 4: Training of the FoldingNet autoencoder on ModelNet40.
Figure 5: t-SNE projection of FoldingNet codewords for ModelNet10, colored by KMeans++ cluster ( K=10 , left) and by true class (right). Monitor, toilet, bathtub and the sub-groups of chair, sofa and bed form compact, well separated groups, whereas desk/table and night stand/dresser overlap in the center of the projection.
# Clusters ( K )
NMI
ARI
ACC
2
0.363
0.179
0.359
4
0.591
0.429
0.549
6
0.734
0.686
0.737
8
0.758
0.757
0.789
10
0.733
0.678
0.796
12
0.721
0.627
0.801
Table 1: Autoencoding/K-Means++ Clustering Results on ModelNet10 (all 4,899 shapes). Best value per column in bold; the boxed K is the elbow (operating point).
Figure 6: NMI, ARI and ACC of KMeans++ on FoldingNet codewords vs. the number of clusters K (ModelNet10). NMI and ARI peak at K=8 ; ACC keeps increasing up to K=20 .
Figure 7: Mean codeword distortion (top) and Gaussian-mixture BIC ( 1 ) (bottom) vs. K on ModelNet10. The solid line marks the label-free knee K=8 , which coincides with the boxed elbow; ( 1 ) has no interior minimum and selects K=20 .
Figure 8: t-SNE projection of FoldingNet codewords for ModelNet40, colored by KMeans++ cluster ( K=40 , left) and by true class (right). Airplane, car, guitar, laptop, chair, toilet and monitor form compact, well separated groups, whereas the box-like furniture classes (night stand, dresser, tv stand, glass box, wardrobe, xbox) and the container-like classes (vase, cup, flower pot, plant) overlap in the center of the projection.
# Clusters ( K )
NMI
ARI
ACC
10
0.563
0.324
0.418
20
0.663
0.522
0.583
30
0.687
0.528
0.658
35
0.691
0.536
0.676
40
0.695
0.526
0.691
45
0.706
0.542
0.712
Table 2: Autoencoding/K-Means++ Clustering Results on ModelNet40 (all 12,311 shapes). Best value per column in bold; the boxed K is the elbow (operating point).
Figure 9: NMI, ARI and ACC of KMeans++ on FoldingNet codewords vs. the number of clusters K (ModelNet40). NMI and ARI peak at K=45 ; the boxed operating point remains K=30 , past which ACC improves only slowly up to K=80 .
Figure 10: Mean codeword distortion (top) and Gaussian-mixture BIC ( 1 ) (bottom) vs. K on ModelNet40. The solid line marks the label-free knee K=35 ; the dashed line is the boxed elbow K=30 . ( 1 ) has no interior minimum and selects K=80 .
We present point2vec, a self-supervised method for learning point cloud representations through masked prediction of a teacher network's latent features. Building on data2vec, we investigate a limitation of applying this approach to point clouds: positional information associated with masked patches reveals the overall object shape to the student, weakening the learning task even under heavy masking. point2vec addresses this issue by passing only visible patches to the student encoder and introducing a shallow decoder to predict representations for masked patches. The teacher processes the complete point cloud to provide contextualized training targets. These targets capture global context through the teacher's self-attention, providing supervision beyond the local geometry of individual patches. This design conceals masked-patch positions from the student and reduces its computational cost during pre-training. Our ablations show that the decoder alone provides only part of the improvement; concealing positional information is key. After pre-training on ShapeNet, point2vec outperforms the evaluated self-supervised baselines on shape classification on ModelNet40 and ScanObjectNN and few-shot classification on ModelNet40, while achieving competitive part segmentation results on ShapeNetPart.
Autoregressive models have emerged as an effective paradigm for point cloud generation. However, most existing approaches rely on heuristic tokenization strategies, such as spatial sorting or stochastic downsampling, which often disrupt intrinsic point cloud topology and weaken the structural coherence of the generated shapes. In this paper, we present PointRSP, an autoregressive framework that reformulates point cloud generation as a topology-preserving tessellation process via recursive spectral partitioning. Instead of constructing token sequences heuristically, we introduce a topology-aware partitioning autoencoder that decomposes an unstructured point cloud into a non-balanced binary tree through a hybrid recursive spectral partitioning strategy. This hierarchical representation provides a deterministic geometric blueprint that preserves topological relationships while capturing multiscale structural dependencies within a quantized latent space. To synthesize shapes in this space, we propose a dual-stream cascaded generator that jointly models structural evolution and feature synthesis. In addition, we design a geometry-calibrated positional encoding mechanism that anchors latent embeddings using multi-scale structural centers, which stabilizes cascaded generation during the early stages of structural formation. Extensive experiments show that PointRSP achieves state-of-the-art performance in generation quality and diversity, demonstrating strong generalization across complex 3D topologies.
Monan Sun, Bangzhen Liu, Huaidong Zhang +1
South China University of Technology, China · University of Chinese Academy of Sciences, China · Singapore Management University, Singapore +1
3D point cloud-language models (3D-LLMs) enable 3D understanding by pairing point cloud encoders with large language models, but existing methods rely on costly multi-modal encoders (e.g., ULIP-2) that require image-text-point cloud alignment on 8x A100-scale compute, creating high barriers for research and deployment. In this work, we systematically investigate whether low-cost self-supervised point cloud encoders, specifically PCP-MAE and Point-MAE, can serve as effective alternatives. Using MiniGPT-3D as our testbed, we evaluate 7 encoder initialization/pre-training setups (1 multi-modal baseline, 5 self-supervised, 1 random init) under frozen and unfrozen fine-tuning (12 total groups), across 2 architectures (MaskTransformer, PointTransformer), 3 objectives (PCP-MAE, Point-MAE, random init), and 2 datasets (Objaverse 660K, ShapeNet55-34 approximately 50K). Our experiments reveal three key findings: (1) The four-stage MiniGPT-3D pipeline can effectively train a 3D encoder from random initialization: an end-to-end trained random init encoder reaches 52.50% open-vocabulary accuracy and 44.45 captioning score, approaching top pre-trained variants; (2) Architecture and pre-training objective show strong crossover interaction: PCP-MAE + MaskTransformer achieves 59.00% accuracy (best self-supervised), while Point-MAE + MaskTransformer drops to 46.50%, with the pattern reversed for PointTransformer; (3) Closed-set ModelNet40 classification remains a core weakness of purely geometric encoders, reaching only ~13-18% accuracy vs. ~62% for the multi-modal baseline, even after end-to-end fine-tuning. Our results offer practical guidelines for cost-effective 3D-LLM design and reveal interaction patterns between self-supervised objectives and encoder architectures.
Yao Zheng, Tian Zhang
Beijing University of Posts and Telecommunications