Organizations: Instituto de Ciências Matemáticas e de Computação Universidade de São Paulo São Carlos, São Paulo, BR · Bowdoin College Brunswick, Maine, USA
Vision-Language Models align point clouds with image and text embeddings, enabling zero-shot recognition, retrieval, and open-vocabulary understanding of 3D shapes. Existing multimodal 3D pre-training methods produce fixed-dimensional embeddings, requiring separate models for different computational budgets. We propose 3D Matryoshka Representation Learning (3D-MRL), a multimodal 3D pre-training framework based on Matryoshka Representation Learning. 3D-MRL learns nested 3D representations by aligning point clouds with frozen CLIP image and text embeddings while applying contrastive supervision across multiple embedding dimensions. The Matryoshka objective is applied only to the 3D encoder, allowing a single model to produce representations at different dimensionalities without retraining. Experiments on the Objaverse-LVIS, ModelNet40, and ScanNet datasets show that 3D-MRL achieves competitive performance on zero-shot and few-shot 3D recognition tasks. In addition, the learned representations support retrieval across different embedding dimensions within a single model. On Objaverse-LVIS, 3D-MRL improves Top-1 accuracy from 46.8% to 50.9%. Retrieval experiments further show that different embedding dimensions yield varying levels of semantic and geometric specificity.
Figures & tables
Figure 1: Overview of 3D-MRL \xspace . 3D-MRL \xspace is a framework that aligns nested 3D embeddings with multiple dimensionalities with multimodal image and text for 3D representation learning. 3D-MRL \xspace shows competitive performance on a wide range of benchmarks.
Figure 2: Zero-shot 3D shape classification on the Objaverse-LVIS [ 8 ] (1,156 categories) and ModelNet40 [ 35 ] .
Method
Pre-Training Dataset
Objaverse-LVIS [ 8 ]
ModelNet40 [ 35 ]
ScanObjectNN [ 31 ]
Top1
Top3
Top5
Top1
Top3
Top5
Top1
Top3
Top5
PointCLIP [ 43 ]
-
1.9
4.1
5.8
19.3
28.6
34.8
10.5
20.8
30.6
PointCLIPv2 [ 47 ]
-
4.7
9.5
12.9
63.6
77.9
85.0
42.2
63.3
74.5
CLIP2Point [ 14 ]
ShapeNet
2.7
5.8
7.9
49.5
71.3
81.2
25.5
44.6
59.4
ReCon [ 27 ]
1.1
2.7
3.7
61.2
73.9
78.1
42.3
62.5
75.6
ULIP-PointBERT [ 37 ]
6.2
13.6
17.9
60.4
79.0
84.4
51.5
71.1
80.2
Table 1: Zero-shot classification on Objaverse-LVIS [ 8 ] , ModelNet40 [ 35 ] , and ScanObjectNN [ 31 ] . The performance gains from 3D-MRL \xspace are more significant on the most challenging long-tailed Objaverse-LVIS dataset. In bold, we mark the best results per pre-training method.
Table 4
Pre-Training Dataset
Method
5-way
10-way
10-shot
20-shot
10-shot
20-shot
ShapeNet
ULIP [ 37 ]
94.4 ± 3.7
93.2 ± 4.2
86.6 ± 5.3
90.6 ± 5.2
OpenShape [ 22 ]
95.3 ± 2.6
97.9 ± 3.9
89.2 ± 5.1
92.9 ± 3.9
3D-MRL \xspace ∗ (Ours)
95.2 ± 2.7
96.1 ± 2.4
88.2 ± 6.5
91.1 ± 4.6
Ensembled
OpenShape [ 22 ]
96.1 ± 2.7
95.7 ± 2.5
89.1 ± 4.6
91.8 ± 3.7
3D-MRL \xspace ∗ (Ours)
94.8 ± 2.8
97.9 ± 0.9
89.7 ± 6.0
93.2 ± 3.1
Table 4: Few-shot linear probing classification results on ModelNet40. We report the average accuracy and standard deviation of 10 independent experiments. Our 3D-MRL \xspace remains competitive across few-shot settings and improves over OpenShape in three of four Ensembled configurations.
Method
D
Avg.
Bed
Cab
Chair
Sofa
Tabl
Door
Wind
Bksf
Pic
Cntr
Desk
Curt
Fridg
Bath
Showr
Toil
Sink
CLIP2Point [ 14 ]
–
24.9
20.8
0.0
85.1
43.3
26.5
69.9
0.0
20.9
1.7
31.7
27.0
0.0
1.6
46.5
0.0
22.4
25.6
PointCLIP w/ TP. [ 43 ]
–
26.1
0.0
55.7
72.8
5.0
5.1
1.7
0.0
77.2
0.0
0.0
51.7
0.3
0.0
0.0
40.3
85.3
49.2
CLIP2Point w/ TP. [ 14 ]
–
35.2
11.8
3.0
45.1
27.6
10.5
61.5
2.6
71.9
0.3
33.6
29.9
4.7
11.5
72.2
92.4
86.1
34.0
CLIP 2 [ 39 ]
–
38.5
32.6
67.2
69.3
42.3
18.3
19.1
4.0
62.6
1.4
12.7
52.8
40.1
9.1
59.7
41.0
71.0
45.5
OpenShape † [ 22 ]
1280
45.6
66.7
3.2
75.8
83.5
37.7
49.2
47.5
64.9
48.2
1.9
66.1
70.2
1.8
50.0
57.1
45.2
7.1
3D-MRL \xspace † (Ours)
10
5.7
10.4
16.7
3.9
0.0
0.3
0.2
0.0
31.6
0.0
0.0
0.8
0.0
0.8
0.0
29.6
0.0
2.0
Table 5: Zero-shot recognition on the real-world ScanNet [ 7 ] dataset. Bold values indicate the best performance for each category. 3D-MRL \xspace achieves the best results across the granularity dimensions. Bold values indicate the best results per category.
Figure 3: Point cloud-to-3D shape retrieval results using the proposed 3D-MRL \xspace embedding space across increasing embedding dimensions. Starting from coarse semantic retrieval at low dimensions, the representations progressively refine toward fine-grained geometric similarity at higher dimensions, demonstrating the hierarchical Matryoshka property of the learned embedding space.
Figure 4: Semantic-vs-instance retrieval across embedding dimensions on Objaverse-LVIS.
Figure 5: Text-to-3D shape retrieval comparisons. Qualitative text-to-3D retrieval comparisons on Objaverse-LVIS. Given progressively more specific text prompts, 3D-MRL \xspace produces more fine-grained and semantically accurate 3D retrievals compared to OpenShape. We list the input text and the first three retrieved 3D objects.
Efficient
Objaverse-LVIS
ModelNet-40
Top-1
Top-3
Top-5
Top-1
Top-3
Top-5
True
12.4
21.6
26.5
71.5
87.1
91.1
False
12.4
22.1
27.2
71.8
86.8
91.3
Table 6: Effect of weight-tying: Efficient vs. Non-Efficient mode. We compare the parameter-free geometric truncation (True) against the use of independent scale-specific linear projection heads (False). In bold, we display the best performance per column.
Embedding Schedule
Nesting dimensions
Objaverse-LVIS
ModelNet-40
Top-1
Top-3
Top-5
Top-1
Top-3
Top-5
full
1280
12.1
22.1
27.0
70.6
86.0
90.3
linear
8, 16, 32, 64, 128, 256, 512, 1024, 1280
12.4
22.1
27.2
69.8
86.1
91.3
log-based
10, 20, 40, 80, 160, 320, 640, 1280
12.4
22.1
27.2
71.8
86.8
91.3
Table 7: Embedding dimension schedules. Comparison of zero-shot classification performance across different distribution granularities for a maximum capacity of D=1280 . Our proposed geometric log-based schedule consistently outperforms both the fixed-dimensional baseline ( full ) and the uniform sequence ( linear ), serving as an effective multi-scale optimization regularizer. In bold, we display the best performance per column.
Vision-Language Models (VLMs) are reshaping computer vision by aligning visual and textual embeddings, allowing models to recognize visual concepts and reason about them using natural language. Traditional 3D deep-learning models, however, are typically trained for specific tasks, such as classification, segmentation, or detection, and do not naturally support cross-modal retrieval from their embedding spaces using text or images as queries. To address this issue, Contrastive Language-Image Pretraining (CLIP)-based methods align 3D embeddings with pretrained image and text representations, giving rise to 3D Vision-Language Models (3D VLMs) that support zero-shot classification, cross-modal retrieval, and open-vocabulary recognition of 3D shapes. This tutorial provides an overview of 3D VLMs, ranging from basic definitions of 3D representations and their encoding into embeddings to cross-modal contrastive alignment, modern multimodal frameworks, and 3D Vision-Large Language Models (3D VLLMs). We present the main definitions of contrastive learning for multimodal embedding alignment and highlight recent advances in language-guided 3D Gaussian splatting, 3D shape generation, and embodied AI for robotics.
Open-set 3D recognition requires models that generalize to rare or unseen categories. Recent approaches address this by distilling language-vision knowledge into 3D encoders, typically relying on heavy 2D ViTs and aligning each point cloud with a single image or caption, thus anchoring representations to partial views. We propose aligning each point cloud with multiple images and textual descriptions to capture a more holistic understanding of 3D objects. To realize this idea, it is essential to design a loss function capable of jointly aligning a 3D instance with multiple matched signals, multi-view images and multiple texts, while separating positive aggregation from negative competition. We introduce such a function, termed the decoupled multi-positive contrastive loss. Our formulation enhances the loss's hardness-aware focus on challenging negatives, avoiding the "spotlight crowding" that occurs when many positives share the same softmax with all the negatives. Complementing this, we present a lightweight text adapter applied only to web captions, reducing the domain gap to curated annotations and enabling effective use of large-scale unsupervised text. Our model demonstrates state-of-the-art open-vocabulary performance on long-tail benchmarks, yielding substantial zero-shot improvements while sustaining high frame rates.
The development of 3D Vision-Language Models (VLMs), crucial for applications in robotics, autonomous driving, and augmented reality, is severely constrained by the scarcity of paired 3D-text data. Existing methods rely solely on next-token prediction loss, using only language tokens for supervision. This results in inefficient utilization of limited 3D data and leads to a significant degradation and loss of valuable geometric information in intermediate representations. To address these limitations, we propose {\mname}, a novel feature-level alignment regularization method. {\mname} explicitly supervises intermediate point cloud tokens to preserve fine-grained 3D geometric-semantic information throughout the language modeling process. Specifically, we constrain the intermediate point cloud tokens within the LLM to align with visual input tokens via a consistency loss. By training only a lightweight alignment projector and LoRA adapters, {\mname} achieves explicit feature-level supervision with minimal computational overhead, effectively preventing geometric degradation. Extensive experiments on ModelNet40 and Objaverse datasets demonstrate that our method achieves 2.08 pp improvement on average for classification tasks, with a substantial 7.50 pp gain on the challenging open-vocabulary Objaverse classification task and 4.88 pp improvement on 3D object captioning evaluated by Qwen2-72B-Instruct, validating the effectiveness of {\mname}. Code is publicly available at https://github.com/yharoldsu0627/PointAlign.
Yuanhao Su, Shaofeng Zhang, Xiaosong Jia +1
University of Science and Technology of China · Fuzhou University · Fudan University +1