In multi-modal 3D pre-training, aligning 3D shapes, their 2D image counterparts, and language descriptions has made tremendous progress. However, we find that supervision from rendered images is unreliable, the semantics encoded in synthetic views are distorted due to domain gap, self-occlusion, and rendering artifacts. Instead, we learn transferable 3D representations from language only. We observe that, in real world, 3D shapes are organized in a tree-like structure, ranging from general concepts to fine-grained instances with specific attributes, while language typically denotes generic concepts and subsumes these various 3D instantiations. Hence, we propose to embed 3D representations in hyperbolic space, whose geometry is well-suited for modeling tree-like data without distortion. To capture the asymmetric hierarchy between language and 3D shapes, we further impose a partial-order relation that language entails 3D shapes. Our method, Hyperbolic Contrastive Learning (HyperCL), jointly optimizes a hyperbolic geodesic-based contrastive objective and an entailment loss to learn discriminative cross-modal representations while explicitly preserving this hierarchy. We also introduce a rigorously verified benchmark for comprehensively evaluating cross-modal 3D alignment. Across two widely used benchmarks and ours, we improve the performance up to 4.6% absolute percentage on 3D shape classification and 5.5% on cross-modal text-to-3D retrieval under the same parameter budget. The code and evaluation benchmark will be made publicly available.
Figures & tables
Figure 1
Figure 2: We embed 3D shapes in hyperbolic space to preserve exponentially expanding tree-like data. In addition, we propose an entailment loss that enforces the relation in which caption entails 3D shapes .
Method
Data
OLVIS
c-OLVIS
MN40
Acc@1
Acc@3
Acc@5
Acc@1
Acc@3
Acc@5
Acc@1
Acc@3
Acc@5
ReCon Qi et al. (2023)
ShapeNet
1.1
2.7
3.7
-
-
-
61.2
73.9
78.1
CLIP2Point Huang et al. (2023)
2.7
5.8
7.9
-
-
-
49.5
71.3
81.2
ULIP Xue et al. (2023)
6.2
13.6
17.9
9.9
19.1
24.1
60.4
79.0
84.4
OpenShape Liu et al. (2023)
10.8
20.2
25.0
16.5
27.3
32.1
70.3
86.9
91.3
TAMM Zhang et al. (2024)
13.7
24.2
29.2
23.9
35.5
41.2
73.1
88.5
91.9
Table 3: Comparison on zero-shot 3D shape classification with state-of-the-art methods. To make a comprehensive comparison, we test other methods on our c-OLVIS benchmark if the model weights are released and available. Note that Uni3D-L and Uni3D-G are 3.6 × and 11.9 × more parameters than ours. The best is bold, and second-best is underlined.
Method
Data
P → T
T → P
R@1
R@3
R@5
R@1
R@3
R@5
ULIP2 Xue et al. (2024)
ShapeNet
1.0
2.4
3.6
1.0
2.7
3.9
TAMM Zhang et al. (2024)
3.9
9.4
12.8
4.9
10.5
14.4
HyperCL (Ours)
6.0
12.1
17.1
7.0
14.7
19.2
OpenShape Liu et al. (2023)
Ensembled (w/o LVIS)
19.6
35.2
45.5
20.4
37.1
46.0
Uni3D-B Zhou et al. (2024)
22.0
39.3
48.8
18.5
34.1
42.7
Table 4: Zero-shot cross-modal 3D retrieval results. For a fair comparison, when evaluating other methods, we use their sampled point clouds from 3D meshes. Recall@K is used as evaluation metric. The best is bold, second-best is underlined. P → T denote 3D-to-text retrieval, T → P is text-to-3D retrieval. Note Uni3D-L and Uni3D-G are 3.6 × and 11.9 × more parameters than ours.
Figure 5Figure 6
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameters
Value
Descriptions
Normalization
✓
normalize xyz coordinates of point cloud into [-1, 1]
Point dropout
0.875
the maximum ratio of dropped points is 0.875
Random scale
[0.8, 1.25]
uniformly scale points
Random shift
[-0.1, 0.1]
shift x, y, z coordinates independently
Random rotation
✓
small rotation jittering around x, y, z axes
Random rotation around gravity-axis
✓
rotate around gravity-axis for learning rotation invariance
Appendix
Table 6: Hyperparameters used in our HyperCL.
Speed (it/s)
Peak Memory (GB)
GFLOPs
LEP↔I+LEP↔T
3.30
46.65
72.05
LEP↔T
3.43
46.65
72.05
LHP↔T+LHent
3.36
46.65
72.05
Appendix
Table 7: Comparison of training cost.
Figure 6: Examples from the manual verification. For four categories lollipop, birdhouse, combination_lock and turkey_(food), the upper row shows 3D shapes that were correctly labelled and kept in our c-OLVIS, while the lower row shows OLVIS samples that were mislabelled and removed.
Figure 7: Statistics of our c-OLVIS. Left: category vocabulary of c-OLVIS, 846 categories in total. Middle: the rank–frequency distribution of class sizes on category-index axis. Right: the number of categories per class-size bin.
Figure 8: Qualitative visualization of retrieval results. Top: 3D → text retrieval (top-1 caption). Bottom: text → 3D retrieval (top-5 shapes). Similarities decrease from left to right.
Vision-Language Models align point clouds with image and text embeddings, enabling zero-shot recognition, retrieval, and open-vocabulary understanding of 3D shapes. Existing multimodal 3D pre-training methods produce fixed-dimensional embeddings, requiring separate models for different computational budgets. We propose 3D Matryoshka Representation Learning (3D-MRL), a multimodal 3D pre-training framework based on Matryoshka Representation Learning. 3D-MRL learns nested 3D representations by aligning point clouds with frozen CLIP image and text embeddings while applying contrastive supervision across multiple embedding dimensions. The Matryoshka objective is applied only to the 3D encoder, allowing a single model to produce representations at different dimensionalities without retraining. Experiments on the Objaverse-LVIS, ModelNet40, and ScanNet datasets show that 3D-MRL achieves competitive performance on zero-shot and few-shot 3D recognition tasks. In addition, the learned representations support retrieval across different embedding dimensions within a single model. On Objaverse-LVIS, 3D-MRL improves Top-1 accuracy from 46.8% to 50.9%. Retrieval experiments further show that different embedding dimensions yield varying levels of semantic and geometric specificity.
Márcus Lobo, Vitor Matias, Jeová Farias +1
Instituto de Ciências Matemáticas e de Computação Universidade de São Paulo São Carlos, São Paulo, BR · Bowdoin College Brunswick, Maine, USA
Image-based shape retrieval (IBSR) aims to retrieve 3D models from a database given a query image, hence addressing a classical task in computer vision, computer graphics, and robotics. Recent approaches typically rely on bridging the domain gap between 2D images and 3D shapes based on the use of multi-view renderings as well as task-specific metric learning to embed shapes and images into a common latent space. In contrast, we address IBSR through large-scale multi-modal pretraining and show that explicit view-based supervision is not required. Inspired by pre-aligned image--point-cloud encoders from ULIP and OpenShape that have been used for tasks such as 3D shape classification, we propose the use of pre-aligned image and shape encoders for zero-shot and standard IBSR by embedding images and point clouds into a shared representation space and performing retrieval via similarity search over compact single-embedding shape descriptors. This formulation allows skipping view synthesis and naturally enables zero-shot and cross-domain retrieval without retraining on the target database. We evaluate pre-aligned encoders in both zero-shot and supervised IBSR settings and additionally introduce a multi-modal hard contrastive loss (HCL) to further increase retrieval performance. Our evaluation demonstrates state-of-the-art performance, outperforming related methods on AccTop1 and AccTop10 for shape retrieval across multiple datasets, with best results observed for OpenShape combined with Point-BERT. Furthermore, training on our proposed multi-modal HCL yields dataset-dependent gains in standard instance retrieval tasks on shape-centric data, underscoring the value of pretraining and hard contrastive learning for 3D shape retrieval. The code will be made available via the project website.
Paul Julius Kühn, Cedric Spengler, Michael Weinmann +2
Fraunhofer IGD, 64283 Darmstadt, Germany · Delft University of Technology, 2628 CD Delft, Netherlands
We address a fundamental gap in 3D-LLMs: existing models focus on single-object/scene description, struggling with detailed, inter-object comparison. We propose a framework for detailed object-level reasoning across multiple objects with three components: (1) MO3D (Multi-Object in 3D), an instruction dataset requiring fine-grained multi-object comparison; (2) Multi-3DLLM, using a minimal Patch-Interaction Transformer (PIT) that models inter-/intra-object relationships while preserving local geometry; (3) Mini-apps, two application-driven benchmarks (Shape Mating, Change Captioning) that probe geometric understanding for practical use. Recent 3D-LLMs and 2D-VLMs perform poorly on these tasks, lacking both comparison-centric design and geometric awareness. In contrast, Multi-3DLLM trained on our mixture data learns geometric reasoning, surpasses all baselines on MO3D, and provides positive transfer to single-object classification.
Kohsuke Ide, Ryousuke Yamada, Yue Qiu +4
AIST · University of Tsukuba · University of Technology Nuremberg +1