Organizations: Instituto de Ciências Matemáticas e de Computação Universidade de São Paulo São Carlos, São Paulo, BR · Bowdoin College Brunswick, Maine, USA
Vision-Language Models align point clouds with image and text embeddings, enabling zero-shot recognition, retrieval, and open-vocabulary understanding of 3D shapes. Existing multimodal 3D pre-training methods produce fixed-dimensional embeddings, requiring separate models for different computational budgets. We propose 3D Matryoshka Representation Learning (3D-MRL), a multimodal 3D pre-training framework based on Matryoshka Representation Learning. 3D-MRL learns nested 3D representations by aligning point clouds with frozen CLIP image and text embeddings while applying contrastive supervision across multiple embedding dimensions. The Matryoshka objective is applied only to the 3D encoder, allowing a single model to produce representations at different dimensionalities without retraining. Experiments on the Objaverse-LVIS, ModelNet40, and ScanNet datasets show that 3D-MRL achieves competitive performance on zero-shot and few-shot 3D recognition tasks. In addition, the learned representations support retrieval across different embedding dimensions within a single model. On Objaverse-LVIS, 3D-MRL improves Top-1 accuracy from 46.8% to 50.9%. Retrieval experiments further show that different embedding dimensions yield varying levels of semantic and geometric specificity.
Figures & tables
Figure 1: Overview of 3D-MRL \xspace . 3D-MRL \xspace is a framework that aligns nested 3D embeddings with multiple dimensionalities with multimodal image and text for 3D representation learning. 3D-MRL \xspace shows competitive performance on a wide range of benchmarks.
Figure 2: Zero-shot 3D shape classification on the Objaverse-LVIS [ 8 ] (1,156 categories) and ModelNet40 [ 35 ] .
Method
Pre-Training Dataset
Objaverse-LVIS [ 8 ]
ModelNet40 [ 35 ]
ScanObjectNN [ 31 ]
Top1
Top3
Top5
Top1
Top3
Top5
Top1
Top3
Top5
PointCLIP [ 43 ]
-
1.9
4.1
5.8
19.3
28.6
34.8
10.5
20.8
30.6
PointCLIPv2 [ 47 ]
-
4.7
9.5
12.9
63.6
77.9
85.0
42.2
63.3
74.5
CLIP2Point [ 14 ]
ShapeNet
2.7
5.8
7.9
49.5
71.3
81.2
25.5
44.6
59.4
ReCon [ 27 ]
1.1
2.7
3.7
61.2
73.9
78.1
42.3
62.5
75.6
ULIP-PointBERT [ 37 ]
6.2
13.6
17.9
60.4
79.0
84.4
51.5
71.1
80.2
Table 1: Zero-shot classification on Objaverse-LVIS [ 8 ] , ModelNet40 [ 35 ] , and ScanObjectNN [ 31 ] . The performance gains from 3D-MRL \xspace are more significant on the most challenging long-tailed Objaverse-LVIS dataset. In bold, we mark the best results per pre-training method.
Table 4
Pre-Training Dataset
Method
5-way
10-way
10-shot
20-shot
10-shot
20-shot
ShapeNet
ULIP [ 37 ]
94.4 ± 3.7
93.2 ± 4.2
86.6 ± 5.3
90.6 ± 5.2
OpenShape [ 22 ]
95.3 ± 2.6
97.9 ± 3.9
89.2 ± 5.1
92.9 ± 3.9
3D-MRL \xspace ∗ (Ours)
95.2 ± 2.7
96.1 ± 2.4
88.2 ± 6.5
91.1 ± 4.6
Ensembled
OpenShape [ 22 ]
96.1 ± 2.7
95.7 ± 2.5
89.1 ± 4.6
91.8 ± 3.7
3D-MRL \xspace ∗ (Ours)
94.8 ± 2.8
97.9 ± 0.9
89.7 ± 6.0
93.2 ± 3.1
Table 4: Few-shot linear probing classification results on ModelNet40. We report the average accuracy and standard deviation of 10 independent experiments. Our 3D-MRL \xspace remains competitive across few-shot settings and improves over OpenShape in three of four Ensembled configurations.
Method
D
Avg.
Bed
Cab
Chair
Sofa
Tabl
Door
Wind
Bksf
Pic
Cntr
Desk
Curt
Fridg
Bath
Showr
Toil
Sink
CLIP2Point [ 14 ]
–
24.9
20.8
0.0
85.1
43.3
26.5
69.9
0.0
20.9
1.7
31.7
27.0
0.0
1.6
46.5
0.0
22.4
25.6
PointCLIP w/ TP. [ 43 ]
–
26.1
0.0
55.7
72.8
5.0
5.1
1.7
0.0
77.2
0.0
0.0
51.7
0.3
0.0
0.0
40.3
85.3
49.2
CLIP2Point w/ TP. [ 14 ]
–
35.2
11.8
3.0
45.1
27.6
10.5
61.5
2.6
71.9
0.3
33.6
29.9
4.7
11.5
72.2
92.4
86.1
34.0
CLIP 2 [ 39 ]
–
38.5
32.6
67.2
69.3
42.3
18.3
19.1
4.0
62.6
1.4
12.7
52.8
40.1
9.1
59.7
41.0
71.0
45.5
OpenShape † [ 22 ]
1280
45.6
66.7
3.2
75.8
83.5
37.7
49.2
47.5
64.9
48.2
1.9
66.1
70.2
1.8
50.0
57.1
45.2
7.1
3D-MRL \xspace † (Ours)
10
5.7
10.4
16.7
3.9
0.0
0.3
0.2
0.0
31.6
0.0
0.0
0.8
0.0
0.8
0.0
29.6
0.0
2.0
Table 5: Zero-shot recognition on the real-world ScanNet [ 7 ] dataset. Bold values indicate the best performance for each category. 3D-MRL \xspace achieves the best results across the granularity dimensions. Bold values indicate the best results per category.
Figure 3: Point cloud-to-3D shape retrieval results using the proposed 3D-MRL \xspace embedding space across increasing embedding dimensions. Starting from coarse semantic retrieval at low dimensions, the representations progressively refine toward fine-grained geometric similarity at higher dimensions, demonstrating the hierarchical Matryoshka property of the learned embedding space.
Figure 4: Semantic-vs-instance retrieval across embedding dimensions on Objaverse-LVIS.
Figure 5: Text-to-3D shape retrieval comparisons. Qualitative text-to-3D retrieval comparisons on Objaverse-LVIS. Given progressively more specific text prompts, 3D-MRL \xspace produces more fine-grained and semantically accurate 3D retrievals compared to OpenShape. We list the input text and the first three retrieved 3D objects.
Efficient
Objaverse-LVIS
ModelNet-40
Top-1
Top-3
Top-5
Top-1
Top-3
Top-5
True
12.4
21.6
26.5
71.5
87.1
91.1
False
12.4
22.1
27.2
71.8
86.8
91.3
Table 6: Effect of weight-tying: Efficient vs. Non-Efficient mode. We compare the parameter-free geometric truncation (True) against the use of independent scale-specific linear projection heads (False). In bold, we display the best performance per column.
Embedding Schedule
Nesting dimensions
Objaverse-LVIS
ModelNet-40
Top-1
Top-3
Top-5
Top-1
Top-3
Top-5
full
1280
12.1
22.1
27.0
70.6
86.0
90.3
linear
8, 16, 32, 64, 128, 256, 512, 1024, 1280
12.4
22.1
27.2
69.8
86.1
91.3
log-based
10, 20, 40, 80, 160, 320, 640, 1280
12.4
22.1
27.2
71.8
86.8
91.3
Table 7: Embedding dimension schedules. Comparison of zero-shot classification performance across different distribution granularities for a maximum capacity of D=1280 . Our proposed geometric log-based schedule consistently outperforms both the fixed-dimensional baseline ( full ) and the uniform sequence ( linear ), serving as an effective multi-scale optimization regularizer. In bold, we display the best performance per column.