Image-based shape retrieval (IBSR) aims to retrieve 3D models from a database given a query image, hence addressing a classical task in computer vision, computer graphics, and robotics. Recent approaches typically rely on bridging the domain gap between 2D images and 3D shapes based on the use of multi-view renderings as well as task-specific metric learning to embed shapes and images into a common latent space. In contrast, we address IBSR through large-scale multi-modal pretraining and show that explicit view-based supervision is not required. Inspired by pre-aligned image--point-cloud encoders from ULIP and OpenShape that have been used for tasks such as 3D shape classification, we propose the use of pre-aligned image and shape encoders for zero-shot and standard IBSR by embedding images and point clouds into a shared representation space and performing retrieval via similarity search over compact single-embedding shape descriptors. This formulation allows skipping view synthesis and naturally enables zero-shot and cross-domain retrieval without retraining on the target database. We evaluate pre-aligned encoders in both zero-shot and supervised IBSR settings and additionally introduce a multi-modal hard contrastive loss (HCL) to further increase retrieval performance. Our evaluation demonstrates state-of-the-art performance, outperforming related methods on AccTop1 and AccTop10 for shape retrieval across multiple datasets, with best results observed for OpenShape combined with Point-BERT. Furthermore, training on our proposed multi-modal HCL yields dataset-dependent gains in standard instance retrieval tasks on shape-centric data, underscoring the value of pretraining and hard contrastive learning for 3D shape retrieval. The code will be made available via the project website.
Figures & tables
Figure 1: Training pipeline with multiple modalities. Zero-shot retrieval (a) uses pre-aligned image- and shape-encoders trained with multi-modal contrastive losses [ 48 ] [ 49 ] [ 26 ] . Training or fine-tuning (b) of image- and shape-encoders using either InfoNCE or hard contrastive learning enables standard retrieval (c)
Figure 2: Comparison of (a) random sampling vs. (b) hard negative sampling [ 35 ] . Random sampling may yield out-of-class samples (e.g., couch △ vs. airplane ◯ ), producing overly easy negatives or even false negatives (a) , while hard negative sampling ensures true hard negatives close to the anchor ◯ (b) (inspired by Fig. 1 of [ 35 ] ) .
Figure 3: Von Mises-Fisher distribution on a unit hypersphere for varying β . Higher β increases concentration of qβ and negative hardness.
Type
Dataset
Shapes
Train Img
Test Img
Shape-centered
ModelNet40 [ 46 ]
12,311
–
–
Objaverse-LVIS [ 6 ]
46k
–
–
Image-centered
Pix3D [ 40 ]
322
2,648
2,470
CompCars [ 50 ]
94
3,798
1,898
StanfordCars [ 21 ]
134
8,144
8,041
Table 1: Dataset statistics.
Dataset
Framework
Model
Pre-train Data
AccTop1
AccTop10
mAP@10
Modelnet40 [ 46 ]
ULIP [ 48 ]
Point-BERT [ 53 ]
Shapenet [ 1 ]
0.3/9.5
2.1/35.9
0.7/15.6
Ensembled [ 26 ]
1.6/22.8
6.4/61.1
2.9/30.3
OpenShape [ 26 ]
Point-BERT(S) [ 53 ]
Shapenet [ 1 ]
18.7/71.1
55.2/90.7
29.1/73.9
Point-BERT(L) [ 53 ]
Ensembled [ 26 ]
30.5/97.2
67.1/92.9
41.6/80.1
SparseConv [ 4 ]
Shapenet [ 1 ]
9.5/63.8
38.0/88.2
17.1/67.4
Ensembled [ 26 ]
17.9/18.0
51.0/52.6
27.6/27.7
Table 2: Zero-shot retrieval results for shape-centered datasets (aligned using InfoNCE loss [ 41 ] ) on ModelNet40 [ 46 ] , ObjaverseLVIS [ 6 ] . (Notation: instance/class )
Dataset
Framework
Model
Pre-train Data
AccTop1
AccTop10
mAP@10
Pix3D [ 40 ]
ULIP [ 48 ]
Point-BERT [ 53 ]
Shapenet [ 1 ]
0.4/52.5
3.2/80.7
1.0/53.9
Ensembled [ 26 ]
0.9/59.1
4.3/85.1
1.7/60.5
OpenShape [ 26 ]
Point-BERT(S) [ 53 ]
Shapenet [ 1 ]
32.8/88.5
81.5/98.5
47.6/89.1
Point-BERT(L) [ 53 ]
Ensembled [ 26 ]
34.6/90.5
83.3/97.8
50.5/90.5
SparseConv [ 4 ]
Shapenet [ 1 ]
12.4/74.5
49.6/98.6
22.1/78.9
Ensembled [ 26 ]
16.4/89.9
61.3/99.3
28.5/88.5
Table 3: Zero-Shot retrieval results for image-centered (aligned using multi-modal InfoNCE Eq. 1 ) for IBSR datasets Pix3D [ 40 ] , CompCars [ 50 ] and [ 21 ] (Notation: instance/class )
Dataset
Model
Pre-train Data
AccTop1
AccTop10
mAP@10
Modelnet40 [ 46 ]
Point-BERT(S) [ 53 ] (ours)
Shapenet [ 1 ]
45.1/90.2
88.4/98.7
59.3/88.3
Point-BERT(L) [ 53 ] (ours)
Ensembled [ 26 ]
57.3/92.2
92.5/99.1
69.1/88.9
SparseConv [ 4 ] (ours)
Shapenet [ 1 ]
42.9/89.9
87.1/99.1
57.1/88.1
Ensembled [ 26 ]
45.4/91.1
88.6/99.1
59.3/88.8
Pix3D [ 40 ]
LFD [ 12 ]
-
60.7/-
86.3/-
-
HEG-TS [ 10 ]
-
74.9/-
95.0/-
-
Table 4: Retrieval results for OpenShape [ 26 ] models finetuned with multi-modal InfoNCE loss [ 41 ] on ModelNet [ 46 ] , Pix3D [ 40 ] , CompCars [ 50 ] , and StanfordCars [ 21 ] (Notation: instance/class )
Dataset
Model
AccTop1
AccTop10
mAP@10
Ref
HCL(ours)
Ref
HCL(ours)
Ref
HCL(ours)
Modelnet40 [ 46 ]
Point-BERT(S) [ 53 ]
24.7/73.2
24.4/72.2
62.2/91.0
61.0/91.4
35.8/74.5
35.0/73.7
Point-BERT(L) [ 53 ]
26.3/71.6
25.6/71.0
61.5/90.2
60.1/89.7
36.6/72.8
35.8/71.6
SparseConv [ 4 ]
24.5/71.2
24.2/73.3
61.6/91.1
61.2/91.4
35.4/74.0
35.2/74.5
Pix3D [ 40 ]
Point-BERT(S) [ 53 ]
27.4/84.6
27.8/85.9
75.6/98.5
73.9/98.5
43.4/84.6
41.8/84.8
Point-BERT(L) [ 53 ]
30.1/84.9
29.4/81.8
79.5/87.7
73.2/98.6
46.0/84.9
43.2/81.6
Table 5: Zero-shot retrieval on ModelNet40 [ 46 ] , Pix3D [ 40 ] , CompCars [ 50 ] , and StanfordCars [ 21 ] , trained on ShapeNet [ 1 ] . InfoNCE (Ref) vs. HCL (ours). Highlighting the best model for each loss objective. (Notation: instance/class )
Figure 4: Comparison of pre-training and loss function effects on instance retrieval.
Figure 5: Qualitative results a given query image ( Ig ) of the IKEA_EKTORP_2 sofa from [ 40 ] comparing retrieved by the scaled Point-BERT [ 53 ] model by OpenShape [ 26 ] in its pre-aligned version (a), after fine-tuning on Pix3D [ 40 ] using the InfoNCE [ 41 ] loss (b) and after fine-tuning using our multi-modal HCL loss (c).
In multi-modal 3D pre-training, aligning 3D shapes, their 2D image counterparts, and language descriptions has made tremendous progress. However, we find that supervision from rendered images is unreliable, the semantics encoded in synthetic views are distorted due to domain gap, self-occlusion, and rendering artifacts. Instead, we learn transferable 3D representations from language only. We observe that, in real world, 3D shapes are organized in a tree-like structure, ranging from general concepts to fine-grained instances with specific attributes, while language typically denotes generic concepts and subsumes these various 3D instantiations. Hence, we propose to embed 3D representations in hyperbolic space, whose geometry is well-suited for modeling tree-like data without distortion. To capture the asymmetric hierarchy between language and 3D shapes, we further impose a partial-order relation that language entails 3D shapes. Our method, Hyperbolic Contrastive Learning (HyperCL), jointly optimizes a hyperbolic geodesic-based contrastive objective and an entailment loss to learn discriminative cross-modal representations while explicitly preserving this hierarchy. We also introduce a rigorously verified benchmark for comprehensively evaluating cross-modal 3D alignment. Across two widely used benchmarks and ours, we improve the performance up to 4.6% absolute percentage on 3D shape classification and 5.5% on cross-modal text-to-3D retrieval under the same parameter budget. The code and evaluation benchmark will be made publicly available.
Single-view 3D shape retrieval is a fundamental yet challenging task that is increasingly important with the growth of available 3D data. Existing approaches largely fall into two categories: those using contrastive learning to map point cloud features into existing vision-language spaces and those that learn a common embedding space for 2D images and 3D shapes. However, these feed-forward, holistic alignments are often difficult to interpret, which in turn limits their robustness and generalization to real-world applications. To address this problem, we propose Pose-Aware 3D Shape Retrieval (PASR), a framework that formulates retrieval as a feature-level analysis-by-synthesis problem by distilling knowledge from a 2D foundation model (DINOv3) into a 3D encoder. By aligning pose-conditioned 3D projections with 2D feature maps, our method bridges the gap between real-world images and synthetic meshes. During inference, PASR performs a test-time optimization via analysis-by-synthesis, jointly searching for the shape and pose that best reconstruct the patch-level feature map of the input image. This synthesis-based optimization is inherently robust to partial occlusion and sensitive to fine-grained geometric details. PASR substantially outperforms existing methods on both clean and occluded 3D shape retrieval datasets by a wide margin. Additionally, PASR demonstrates strong multi-task capabilities, achieving robust shape retrieval, competitive pose estimation, and accurate category classification within a single framework.
Jiaxin Shi, Guofeng Zhang, Wufei Ma +3
1Shanghai Jiao Tong University · 2Johns Hopkins University · University of California, San Diego +1
Vision-Language Models align point clouds with image and text embeddings, enabling zero-shot recognition, retrieval, and open-vocabulary understanding of 3D shapes. Existing multimodal 3D pre-training methods produce fixed-dimensional embeddings, requiring separate models for different computational budgets. We propose 3D Matryoshka Representation Learning (3D-MRL), a multimodal 3D pre-training framework based on Matryoshka Representation Learning. 3D-MRL learns nested 3D representations by aligning point clouds with frozen CLIP image and text embeddings while applying contrastive supervision across multiple embedding dimensions. The Matryoshka objective is applied only to the 3D encoder, allowing a single model to produce representations at different dimensionalities without retraining. Experiments on the Objaverse-LVIS, ModelNet40, and ScanNet datasets show that 3D-MRL achieves competitive performance on zero-shot and few-shot 3D recognition tasks. In addition, the learned representations support retrieval across different embedding dimensions within a single model. On Objaverse-LVIS, 3D-MRL improves Top-1 accuracy from 46.8% to 50.9%. Retrieval experiments further show that different embedding dimensions yield varying levels of semantic and geometric specificity.
Márcus Lobo, Vitor Matias, Jeová Farias +1
Instituto de Ciências Matemáticas e de Computação Universidade de São Paulo São Carlos, São Paulo, BR · Bowdoin College Brunswick, Maine, USA