Revisiting Triplet Contrastive Learning for 3D Shape Understanding

Chaoyang Zhu; Zhe Liao; Jiayi Zhang; Jie Li; Jiayi Ji; Liujuan Cao

cs.AIOct 14, 2026

Revisiting Triplet Contrastive Learning for 3D Shape Understanding

Authors: Chaoyang Zhu, Zhe Liao, Jiayi Zhang, Jie Li, Jiayi Ji, Liujuan Cao

Organizations: Xiamen University · Shanghai AI Lab

Abstract

In multi-modal 3D pre-training, aligning 3D shapes, their 2D image counterparts, and language descriptions has made tremendous progress. However, we find that supervision from rendered images is unreliable, the semantics encoded in synthetic views are distorted due to domain gap, self-occlusion, and rendering artifacts. Instead, we learn transferable 3D representations from language only. We observe that, in real world, 3D shapes are organized in a tree-like structure, ranging from general concepts to fine-grained instances with specific attributes, while language typically denotes generic concepts and subsumes these various 3D instantiations. Hence, we propose to embed 3D representations in hyperbolic space, whose geometry is well-suited for modeling tree-like data without distortion. To capture the asymmetric hierarchy between language and 3D shapes, we further impose a partial-order relation that language entails 3D shapes. Our method, Hyperbolic Contrastive Learning (HyperCL), jointly optimizes a hyperbolic geodesic-based contrastive objective and an entailment loss to learn discriminative cross-modal representations while explicitly preserving this hierarchy. We also introduce a rigorously verified benchmark for comprehensively evaluating cross-modal 3D alignment. Across two widely used benchmarks and ours, we improve the performance up to 4.6% absolute percentage on 3D shape classification and 5.5% on cross-modal text-to-3D retrieval under the same parameter budget. The code and evaluation benchmark will be made publicly available.

Figures & tables

Figure 2: We embed 3D shapes in hyperbolic space to preserve exponentially expanding tree-like data. In addition, we propose an entailment loss that enforces the relation in which caption entails 3D shapes .

Method	Data	OLVIS			c-OLVIS			MN40
Method	Data	Acc@1	Acc@3	Acc@5	Acc@1	Acc@3	Acc@5	Acc@1	Acc@3	Acc@5
ReCon Qi et al. (2023)	ShapeNet	1.1	2.7	3.7	-	-	-	61.2	73.9	78.1
CLIP2Point Huang et al. (2023)		2.7	5.8	7.9	-	-	-	49.5	71.3	81.2
ULIP Xue et al. (2023)		6.2	13.6	17.9	9.9	19.1	24.1	60.4	79.0	84.4
OpenShape Liu et al. (2023)		10.8	20.2	25.0	16.5	27.3	32.1	70.3	86.9	91.3
TAMM Zhang et al. (2024)		13.7	24.2	29.2	23.9	35.5	41.2	73.1	88.5	91.9

Table 3: Comparison on zero-shot 3D shape classification with state-of-the-art methods. To make a comprehensive comparison, we test other methods on our c-OLVIS benchmark if the model weights are released and available. Note that Uni3D-L and Uni3D-G are 3.6

\times

and 11.9

\times

more parameters than ours. The best is bold, and second-best is underlined.

Method	Data	P $\rightarrow$ T			T $\rightarrow$ P
Method	Data	R@1	R@3	R@5	R@1	R@3	R@5
ULIP2 Xue et al. (2024)	ShapeNet	1.0	2.4	3.6	1.0	2.7	3.9
TAMM Zhang et al. (2024)		3.9	9.4	12.8	4.9	10.5	14.4
HyperCL (Ours)		6.0	12.1	17.1	7.0	14.7	19.2
OpenShape Liu et al. (2023)	Ensembled (w/o LVIS)	19.6	35.2	45.5	20.4	37.1	46.0
Uni3D-B Zhou et al. (2024)		22.0	39.3	48.8	18.5	34.1	42.7

Table 4: Zero-shot cross-modal 3D retrieval results. For a fair comparison, when evaluating other methods, we use their sampled point clouds from 3D meshes. Recall@K is used as evaluation metric. The best is bold, second-best is underlined. P

\rightarrow

T denote 3D-to-text retrieval, T

\rightarrow

P is text-to-3D retrieval. Note Uni3D-L and Uni3D-G are 3.6

\times

and 11.9

\times

more parameters than ours.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameters	Value	Descriptions
Normalization	✓	normalize xyz coordinates of point cloud into [-1, 1]
Point dropout	0.875	the maximum ratio of dropped points is 0.875
Random scale	[0.8, 1.25]	uniformly scale points
Random shift	[-0.1, 0.1]	shift x, y, z coordinates independently
Random rotation	✓	small rotation jittering around x, y, z axes
Random rotation around gravity-axis	✓	rotate around gravity-axis for learning rotation invariance

Appendix

Table 6: Hyperparameters used in our HyperCL.

	Speed (it/s)	Peak Memory (GB)	GFLOPs
$\mathcal{L}_{E}^{P\leftrightarrow I}+\mathcal{L}_{E}^{P\leftrightarrow T}$	3.30	46.65	72.05
$\mathcal{L}_{E}^{P\leftrightarrow T}$	3.43	46.65	72.05
$\mathcal{L}_{H}^{P\leftrightarrow T}+\mathcal{L}_{H}^{ent}$	3.36	46.65	72.05

Appendix

Table 7: Comparison of training cost.

Figure 6: Examples from the manual verification. For four categories lollipop, birdhouse, combination_lock and turkey_(food), the upper row shows 3D shapes that were correctly labelled and kept in our c-OLVIS, while the lower row shows OLVIS samples that were mislabelled and removed.

Panel of Figure 7 — Figure 7: Statistics of our c-OLVIS. Left: category vocabulary of c-OLVIS, 846 categories in total. Middle: the rank–frequency distribution of class sizes on category-index axis. Right: the number of categories per class-size bin.

Panel of Figure 8 — Figure 8: Qualitative visualization of retrieval results. Top: 3D $\rightarrow$ text retrieval (top-1 caption). Bottom: text $\rightarrow$ 3D retrieval (top-5 shapes). Similarities decrease from left to right.

3D Representation Learning

3D representation learning builds geometric features for recognizing, reconstructing, generating, and reasoning about objects and scenes.

See all 187 papers

3D-MRL: Nested Multimodal 3D Representations via Matryoshka Representation Learning
Aug 29, 2026Márcus Lobo, Vitor Matias, Jeová Farias +13D VLMs Multimodal Pretraining
Instituto de Ciências Matemáticas e de Computação Universidade de São Paulo São Carlos, São Paulo, BR · Bowdoin College Brunswick, Maine, USA
Optimizing Multi-Modal Models for Image-Based Shape Retrieval: The Role of Pre-Alignment and Hard Contrastive Learning
Mar 7, 2026Paul Julius Kühn, Cedric Spengler, Michael Weinmann +23D Shape Retrieval Cross-Modal Retrieval
Fraunhofer IGD, 64283 Darmstadt, Germany · Delft University of Technology, 2628 CD Delft, Netherlands
Beyond Single Object: Learning 3D Relations with Large Language Models
Aug 16, 2026Kohsuke Ide, Ryousuke Yamada, Yue Qiu +43D VLMs 3D Spatial Reasoning
AIST · University of Tsukuba · University of Technology Nuremberg +1

Aug 29, 2026J/K move · Enter open · S save

3D-MRL: Nested Multimodal 3D Representations via Matryoshka Representation Learning

Márcus Lobo, Vitor Matias, Jeová Farias, Moacir Ponti

Instituto de Ciências Matemáticas e de Computação Universidade de São Paulo São Carlos, São Paulo, BR · Bowdoin College Brunswick, Maine, USA

3D VLMs Multimodal Pretraining Point Cloud Learning Matryoshka Representation Learning

Open paper AskPDF arXiv

Vision-Language Models align point clouds with image and text embeddings, enabling zero-shot recognition, retrieval, and open-vocabulary understanding of 3D shapes. Existing multimodal 3D pre-training methods produce fixed-dimensional embeddings, requiring separate models for different computational budgets. We propose 3D Matryoshka Representation Learning (3D-MRL), a multimodal 3D pre-training framework based on Matryoshka Representation Learning. 3D-MRL learns nested 3D representations by aligning point clouds with frozen CLIP image and text embeddings while applying contrastive supervision across multiple embedding dimensions. The Matryoshka objective is applied only to the 3D encoder, allowing a single model to produce representations at different dimensionalities without retraining. Experiments on the Objaverse-LVIS, ModelNet40, and ScanNet datasets show that 3D-MRL achieves competitive performance on zero-shot and few-shot 3D recognition tasks. In addition, the learned representations support retrieval across different embedding dimensions within a single model. On Objaverse-LVIS, 3D-MRL improves Top-1 accuracy from 46.8% to 50.9%. Retrieval experiments further show that different embedding dimensions yield varying levels of semantic and geometric specificity.

Abstract

3D Representation Learning

Explore similar work

3D-MRL: Nested Multimodal 3D Representations via Matryoshka Representation Learning

Optimizing Multi-Modal Models for Image-Based Shape Retrieval: The Role of Pre-Alignment and Hard Contrastive Learning

Beyond Single Object: Learning 3D Relations with Large Language Models