Optimizing Multi-Modal Models for Image-Based Shape Retrieval: The Role of Pre-Alignment and Hard Contrastive Learning

Paul Julius Kühn; Cedric Spengler; Michael Weinmann; Arjan Kuijper; Saptarshi Neil Sinha

cs.CVMar 7, 2026

Optimizing Multi-Modal Models for Image-Based Shape Retrieval: The Role of Pre-Alignment and Hard Contrastive Learning

Authors: Paul Julius Kühn, Cedric Spengler, Michael Weinmann, Arjan Kuijper, Saptarshi Neil Sinha

Organizations: Fraunhofer IGD, 64283 Darmstadt, Germany · Delft University of Technology, 2628 CD Delft, Netherlands

Abstract

Image-based shape retrieval (IBSR) aims to retrieve 3D models from a database given a query image, hence addressing a classical task in computer vision, computer graphics, and robotics. Recent approaches typically rely on bridging the domain gap between 2D images and 3D shapes based on the use of multi-view renderings as well as task-specific metric learning to embed shapes and images into a common latent space. In contrast, we address IBSR through large-scale multi-modal pretraining and show that explicit view-based supervision is not required. Inspired by pre-aligned image--point-cloud encoders from ULIP and OpenShape that have been used for tasks such as 3D shape classification, we propose the use of pre-aligned image and shape encoders for zero-shot and standard IBSR by embedding images and point clouds into a shared representation space and performing retrieval via similarity search over compact single-embedding shape descriptors. This formulation allows skipping view synthesis and naturally enables zero-shot and cross-domain retrieval without retraining on the target database. We evaluate pre-aligned encoders in both zero-shot and supervised IBSR settings and additionally introduce a multi-modal hard contrastive loss (HCL) to further increase retrieval performance. Our evaluation demonstrates state-of-the-art performance, outperforming related methods on $Acc_{Top1}$ and $Acc_{Top10}$ for shape retrieval across multiple datasets, with best results observed for OpenShape combined with Point-BERT. Furthermore, training on our proposed multi-modal HCL yields dataset-dependent gains in standard instance retrieval tasks on shape-centric data, underscoring the value of pretraining and hard contrastive learning for 3D shape retrieval. The code will be made available via the project website.

Figures & tables

Figure 1: Training pipeline with multiple modalities. Zero-shot retrieval (a) uses pre-aligned image- and shape-encoders trained with multi-modal contrastive losses [ 48 ] [ 49 ] [ 26 ] . Training or fine-tuning (b) of image- and shape-encoders using either InfoNCE or hard contrastive learning enables standard retrieval (c)

Figure 2: Comparison of (a) random sampling vs. (b) hard negative sampling [ 35 ] . Random sampling may yield out-of-class samples (e.g., couch $\bigtriangleup$ vs. airplane $\bigcirc$ ), producing overly easy negatives or even false negatives (a) , while hard negative sampling ensures true hard negatives close to the anchor $\boldsymbol{\bigcirc}$ (b) (inspired by Fig. 1 of [ 35 ] ) .

Figure 3: Von Mises-Fisher distribution on a unit hypersphere for varying $\beta$ . Higher $\beta$ increases concentration of $q_{\beta}$ and negative hardness.

Type	Dataset	Shapes	Train Img	Test Img
Shape-centered	ModelNet40 [ 46 ]	12,311	–	–
Shape-centered	Objaverse-LVIS [ 6 ]	46k	–	–
Image-centered	Pix3D [ 40 ]	322	2,648	2,470
	CompCars [ 50 ]	94	3,798	1,898
	StanfordCars [ 21 ]	134	8,144	8,041

Table 1: Dataset statistics.

Dataset	Framework	Model	Pre-train Data	$Acc_{Top1}$	$Acc_{Top10}$	$mAP@10$
Modelnet40 [ 46 ]	ULIP [ 48 ]	Point-BERT [ 53 ]	Shapenet [ 1 ]	0.3/9.5	2.1/35.9	0.7/15.6
			Ensembled [ 26 ]	1.6/22.8	6.4/61.1	2.9/30.3
	OpenShape [ 26 ]	Point-BERT(S) [ 53 ]	Shapenet [ 1 ]	18.7/71.1	55.2/90.7	29.1/73.9
		Point-BERT(L) [ 53 ]	Ensembled [ 26 ]	30.5/97.2	67.1/92.9	41.6/80.1
		SparseConv [ 4 ]	Shapenet [ 1 ]	9.5/63.8	38.0/88.2	17.1/67.4
			Ensembled [ 26 ]	17.9/18.0	51.0/52.6	27.6/27.7

Table 2: Zero-shot retrieval results for shape-centered datasets (aligned using InfoNCE loss [ 41 ] ) on ModelNet40 [ 46 ] , ObjaverseLVIS [ 6 ] . (Notation:

\text{instance}/\text{class}

)

Dataset	Framework	Model	Pre-train Data	$Acc_{Top1}$	$Acc_{Top10}$	$mAP@10$
Pix3D [ 40 ]	ULIP [ 48 ]	Point-BERT [ 53 ]	Shapenet [ 1 ]	0.4/52.5	3.2/80.7	1.0/53.9
			Ensembled [ 26 ]	0.9/59.1	4.3/85.1	1.7/60.5
	OpenShape [ 26 ]	Point-BERT(S) [ 53 ]	Shapenet [ 1 ]	32.8/88.5	81.5/98.5	47.6/89.1
		Point-BERT(L) [ 53 ]	Ensembled [ 26 ]	34.6/90.5	83.3/97.8	50.5/90.5
		SparseConv [ 4 ]	Shapenet [ 1 ]	12.4/74.5	49.6/98.6	22.1/78.9
			Ensembled [ 26 ]	16.4/89.9	61.3/99.3	28.5/88.5

Table 3: Zero-Shot retrieval results for image-centered (aligned using multi-modal InfoNCE Eq. 1 ) for IBSR datasets Pix3D [ 40 ] , CompCars [ 50 ] and [ 21 ] (Notation:

\text{instance}/\text{class}

)

Dataset	Model	Pre-train Data	$Acc_{Top1}$	$Acc_{Top10}$	$mAP@10$
Modelnet40 [ 46 ]	Point-BERT(S) [ 53 ] (ours)	Shapenet [ 1 ]	45.1/90.2	88.4/98.7	59.3/88.3
	Point-BERT(L) [ 53 ] (ours)	Ensembled [ 26 ]	57.3/92.2	92.5/99.1	69.1/88.9
	SparseConv [ 4 ] (ours)	Shapenet [ 1 ]	42.9/89.9	87.1/99.1	57.1/88.1
		Ensembled [ 26 ]	45.4/91.1	88.6/99.1	59.3/88.8
Pix3D [ 40 ]	LFD [ 12 ]	-	60.7/-	86.3/-	-
	HEG-TS [ 10 ]	-	74.9/-	95.0/-	-

Table 4: Retrieval results for OpenShape [ 26 ] models finetuned with multi-modal InfoNCE loss [ 41 ] on ModelNet [ 46 ] , Pix3D [ 40 ] , CompCars [ 50 ] , and StanfordCars [ 21 ] (Notation:

\text{instance}/\text{class}

)

Dataset	Model	$Acc_{Top1}$		$Acc_{Top10}$		$mAP@10$
Dataset	Model	Ref	HCL(ours)	Ref	HCL(ours)	Ref	HCL(ours)
Modelnet40 [ 46 ]	Point-BERT(S) [ 53 ]	24.7/73.2	24.4/72.2	62.2/91.0	61.0/91.4	35.8/74.5	35.0/73.7
	Point-BERT(L) [ 53 ]	26.3/71.6	25.6/71.0	61.5/90.2	60.1/89.7	36.6/72.8	35.8/71.6
	SparseConv [ 4 ]	24.5/71.2	24.2/73.3	61.6/91.1	61.2/91.4	35.4/74.0	35.2/74.5
Pix3D [ 40 ]	Point-BERT(S) [ 53 ]	27.4/84.6	27.8/85.9	75.6/98.5	73.9/98.5	43.4/84.6	41.8/84.8
	Point-BERT(L) [ 53 ]	30.1/84.9	29.4/81.8	79.5/87.7	73.2/98.6	46.0/84.9	43.2/81.6

Table 5: Zero-shot retrieval on ModelNet40 [ 46 ] , Pix3D [ 40 ] , CompCars [ 50 ] , and StanfordCars [ 21 ] , trained on ShapeNet [ 1 ] . InfoNCE (Ref) vs. HCL (ours). Highlighting the best model for each loss objective. (Notation:

\text{instance}/\text{class}

)

Panel of Figure 4 — Figure 4: Comparison of pre-training and loss function effects on instance retrieval.

Panel of Figure 5 — Figure 5: Qualitative results a given query image ( $I_{g}$ ) of the IKEA_EKTORP_2 sofa from [ 40 ] comparing retrieved by the scaled Point-BERT [ 53 ] model by OpenShape [ 26 ] in its pre-aligned version (a), after fine-tuning on Pix3D [ 40 ] using the InfoNCE [ 41 ] loss (b) and after fine-tuning using our multi-modal HCL loss (c).

3D Shape Retrieval

See all 8 papers

Revisiting Triplet Contrastive Learning for 3D Shape Understanding
Oct 14, 2026Chaoyang Zhu, Zhe Liao, Jiayi Zhang +33D Representation Learning Cross-Modal Alignment
Xiamen University · Shanghai AI Lab
PASR: Pose-Aware 3D Shape Retrieval from Occluded Single Views
Apr 24, 2026Jiaxin Shi, Guofeng Zhang, Wufei Ma +33D Shape Retrieval 3D Representation Learning
1Shanghai Jiao Tong University · 2Johns Hopkins University · University of California, San Diego +1
3D-MRL: Nested Multimodal 3D Representations via Matryoshka Representation Learning
Aug 29, 2026Márcus Lobo, Vitor Matias, Jeová Farias +13D VLMs Multimodal Pretraining
Instituto de Ciências Matemáticas e de Computação Universidade de São Paulo São Carlos, São Paulo, BR · Bowdoin College Brunswick, Maine, USA

Oct 14, 2026J/K move · Enter open · S save

Revisiting Triplet Contrastive Learning for 3D Shape Understanding

Chaoyang Zhu, Zhe Liao, Jiayi Zhang, Jie Li+2

Xiamen University · Shanghai AI Lab

3D Representation Learning Cross-Modal Alignment Multimodal Contrastive Learning Hyperbolic Representation Learning

Open paper AskPDF arXiv

In multi-modal 3D pre-training, aligning 3D shapes, their 2D image counterparts, and language descriptions has made tremendous progress. However, we find that supervision from rendered images is unreliable, the semantics encoded in synthetic views are distorted due to domain gap, self-occlusion, and rendering artifacts. Instead, we learn transferable 3D representations from language only. We observe that, in real world, 3D shapes are organized in a tree-like structure, ranging from general concepts to fine-grained instances with specific attributes, while language typically denotes generic concepts and subsumes these various 3D instantiations. Hence, we propose to embed 3D representations in hyperbolic space, whose geometry is well-suited for modeling tree-like data without distortion. To capture the asymmetric hierarchy between language and 3D shapes, we further impose a partial-order relation that language entails 3D shapes. Our method, Hyperbolic Contrastive Learning (HyperCL), jointly optimizes a hyperbolic geodesic-based contrastive objective and an entailment loss to learn discriminative cross-modal representations while explicitly preserving this hierarchy. We also introduce a rigorously verified benchmark for comprehensively evaluating cross-modal 3D alignment. Across two widely used benchmarks and ours, we improve the performance up to 4.6% absolute percentage on 3D shape classification and 5.5% on cross-modal text-to-3D retrieval under the same parameter budget. The code and evaluation benchmark will be made publicly available.

Abstract

3D Shape Retrieval

Explore similar work

Revisiting Triplet Contrastive Learning for 3D Shape Understanding

PASR: Pose-Aware 3D Shape Retrieval from Occluded Single Views

3D-MRL: Nested Multimodal 3D Representations via Matryoshka Representation Learning