Cross-modal scene and instance retrieval aims to retrieve a corresponding 3D environment or object given query 2D images, serving as a foundational mechanism for spatial reasoning in robotics and AR/VR. Existing methods achieve this by learning alignment only at the level of whole scenes or objects, which provides no supervision for the fine-grained local structure within them. In this work, we show that supervision on a set of sparse 2D-3D correspondences, without scene- or instance-level contrastive objectives, is sufficient to train a single model for both scene- and instance-level retrieval. Specifically, we propose CrossWeave, a novel framework for cross-modal retrieval that learns sparse patch-point alignment, which alone outperforms existing methods. An optimal transport-based aggregator over the aligned features gives further gains by clustering structurally redundant regions, freeing descriptor capacity for semantically distinct local features. CrossWeave achieves state-of-the-art performance on both scene- and instance-level retrieval on ScanNet and 3RScan with a single model, whereas existing methods require separate models for each task. Our results suggest that whole-scene encoding is unnecessary for cross-modal retrieval and that sparse local correspondence supervision is a more effective and flexible alternative.
Figures & tables
Figure 1: Overview of CrossWeave. CrossWeave learns cross-modal retrieval representations from sparse image patch-3D point correspondences rather than scene- or instance-level labels. Scene Sampling: Given calibrated RGB-D observations and a reconstructed 3D map, visible patch-point pairs are obtained by projecting 3D points into the image plane, filtering occluded points using rendered depth, quantizing valid pixels into DINOv2 patch tokens, and selecting a compact semantically diverse subset using feature-space farthest point sampling. Feature Extraction: Frozen encoders extract local patch and point features, which are projected by learnable projection matrices Wpatch and Wpoint into a shared embedding space. Feature Aggregation: A shared SALAD aggregator then maps variable-sized sets of local features into fixed-dimensional descriptors. Training combines a local patch-point Lp2p contrastive loss with an aggregated image-point-cloud contrastive loss Li2pc , preserving fine-grained alignment while organizing local evidence into retrieval-ready descriptors. At inference, the same encoders and projection heads serve both tasks: SALAD descriptors are used for scene retrieval and mean-pooled local features for instance retrieval.
ScanNet
3RScan
Scene Recall
Scene Recall
Method
Top-1
Top-5
Top-10
Top-1
Top-5
Top-10
Scene CrossOver
22.44
61.86
75.32
19.11
58.60
77.07
Scene CrossOver †
19.87
58.01
74.36
17.83
51.59
73.25
Scene CrossOver (Updated)
40.38
90.06
96.47
23.57
66.88
84.71
Ours
61.54
99.36
100.00
46.50
85.99
92.36
Table 1: Comparison of scene retrieval performance on ScanNet and 3RScan. † Trained on additional datasets (ArkitScenes [ 2 ] + MultiScan [ 18 ] ). Best and Second-best results.
Figure 2: Qualitative comparison of cross-modal scene retrieval on ScanNet. Given a set of query RGB images, each model retrieves the most similar 3D scenes from the database. The ground-truth scene is shown on the left, and retrieved scenes are ordered by similarity from left to right. CrossOver and CrossOver (Updated) fail to retrieve the correct scene within the top five results, while CrossWeave retrieves it at rank 1. The remaining retrieved scenes are also semantically consistent, indicating that CrossWeave learns descriptors that capture both local cross-modal correspondences and global scene context.
Table 2: Comparison of within-scene instance retrieval performance on ScanNet and 3RScan. † Trained on additional datasets (ArkitScenes [ 2 ] + MultiScan [ 18 ] ). Best and Second-best results.
Figure 3: Qualitative instance retrieval on ScanNet. For each query view with its instance mask (left), CrossWeave ranks the instances of the same scene (middle) by cosine similarity. The Top-3 retrievals are shown with their similarity scores, and the correct match is outlined in green.
10 candidates
50 candidates
Method
Top-1
Top-5
Top-10
Top-1
Top-5
Top-10
SceneGraphLoc ‡ [ 19 ]
53.60
81.90
92.80
30.20
50.20
61.20
CrossOver [ 27 ]
46.00
77.97
90.58
18.69
39.16
51.62
Ours
53.60
93.10
100.00
27.50
60.20
76.50
Table 3: Scene retrieval on 3RScan against SceneGraphLoc and CrossOver. Each query image is ranked against N candidate scenes; Top- k : percentage of queries with the correct scene in the top k . ‡ Uses ground-truth scene graphs at inference. Best and second-best results are highlighted.
ScanNet
3RScan
Scene
Instance
Scene
Instance
Agg.
Li2pc
Lp2p
Feats
Top-1
Top-1
R@75%
Top-1
Top-1
R@75%
Mean
✗
✓
local
47.12
78.96
70.51
27.39
73.30
47.13
Mean
✓
✓
local
52.24
79.63
71.15
30.57
73.62
49.04
SALAD
✓
✗
salad
57.37
73.33
56.09
43.95
66.98
38.22
SALAD
✓
✗
local
38.14
73.31
52.88
19.75
67.43
36.31
Table 4: Ablation of training losses, aggregation and retrieval features. Agg. : aggregation used during training. Feats : salad (aggregated descriptors), local (mean-pooled patch and point features), or hybrid ( salad for scene, local for instance retrieval). Best and second-best results are highlighted; the main model is shaded and marked with ⋆ .
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Scene retrieval performance (Top-1 recall) on ScanNet and 3RScan as a function of training data volume. Performance is stable across data regimes, with even 10% of the full training set sufficient to outperform the previous state of the art.
Figure 5: Scene retrieval performance (Top-1 recall) as a function of the number of query camera views. The main results use 10 views.
Instance Retrieval
Scene-Level Recall
Dataset
Mask condition
Top-1
Top-3
R@25%
R@50%
R@75%
ScanNet
GT
78.77
94.05
98.40
88.78
67.95
2D noisy
75.75
93.02
97.44
88.14
62.50
3D noisy
75.33
92.91
97.12
87.50
63.46
2D + 3D noisy
72.29
91.86
97.12
85.90
55.45
3RScan
GT
73.50
92.73
98.09
92.99
49.68
Appendix
Table 5: Instance retrieval under noisy instance masks, using the main model with local features.
Figure 6: Visualizations for the effect of FPS sampling. Images with red dots denote random sampling. Images with blue dots denote FPS sampled points. It can clearly be observed that FPS enables us to avoid over-sampling on the often visually dominant but geometrically uninformative areas in the images such as walls, floors and other flat surfaces, thereby boosting feature diversity within each training batch. Note: We have visualized a large number of pixel-point pairs in these images; in our training framework we only use 32 patch-point pairs. Refer to Section 3.1 for more details.
Figure 7: Comparisons of Top-5 retrievals on a ScanNet room scene. Retrievals are ordered from most to least similar moving left to right, with correct matches highlighted in green. CrossWeave successfully retrieves the correct scene at rank 1, while CrossOver (Updated) retrieves it at rank 2, and the base model of CrossOver fails within the top 5.
Figure 8: Additional qualitative results on a ScanNet office scene. CrossWeave retrieves the ground truth point cloud as its top match, whereas both CrossOver (Updated) and CrossOver only identify the correct scene at rank 5.
Figure 9: Qualitative results on a 3RScan dining area scene. CrossWeave successfully retrieves the ground truth point cloud as its top match, while CrossOver (Updated) ranks it fourth and the base CrossOver model fails to retrieve it within the top 5.
Figure 10: Retrieval results on a 3RScan kitchen and dining scene. CrossWeave demonstrates superior discriminative ability by retrieving the exact scene at rank 1 from highly localized query images, whereas both baseline models completely fail to place the correct match within their top 5 retrievals.
Figure 12: Qualitative instance retrieval on 3RScan. For each query view with its instance mask (left), CrossWeave ranks the instances of the same scene (middle) by cosine similarity. The Top-3 retrievals are shown with their similarity scores, and the correct match is outlined in green.
Figure 13: Failure case on a 3RScan living room scene. All three models miss the ground-truth scene in the top 5, as the retrieved scenes are visually and structurally similar to the query (a brown sectional sofa with a wooden coffee table). CrossWeave’s top matches still share the furniture arrangement (sofas and central coffee tables), whereas CrossOver (Updated) retrieves structurally different scenes.
Figure 14: Instance retrieval failures on ScanNet and 3RScan. In both cases the query is a chair and the Top-1 retrieval (red) is another chair of the same type in the same scene, while the correct instance (green) is ranked second with a similar score.
Efficiently retrieving specific 3D instances from large-scale scenes via natural language prompts remains a formidable challenge in multimedia analysis. Existing approaches predominantly follow a "scene-level embedding" paradigm, which requires distilling high-dimensional semantic features into every 3D primitive. This strategy suffers from a fundamental architectural bottleneck: memory and computational costs scale linearly with scene complexity, inevitably triggering out-of-memory (OOM) failures in city-scale environments. To address this barrier, we propose QueryGaussian, a training-free framework for expeditious and scalable open-vocabulary 3D instance retrieval. Unlike holistic semantic distillation, QueryGaussian employs an instance-level query mechanism that decouples semantic understanding from geometric representation. Specifically, we leverage pre-trained 2D vision models to interpret user prompts and lift segmentation masks into 3D via a concurrent maximum-weight association strategy, ensuring semantic-visual consistency. To mitigate projection ambiguity, we introduce a temporal fusion module with multi-stage adaptive density clustering. Experimental results demonstrate that QueryGaussian not only matches the accuracy of state-of-the-art methods but also delivers a decisive efficiency leap, reducing GPU memory usage by over 70% and accelerating inference by 180x. Crucially, QueryGaussian enables expeditious instance retrieval on city-scale scenes containing tens of millions of Gaussians using consumer-grade hardware.
Xiuyuan Zhu, Ke Lu, Zijie Yang +3
University of Chinese Academy of Sciences, Beijing, China · 3Peng Cheng Laboratory, Shenzhen, China · 2State Key Laboratory of Communication Content Cognition, Beijing, China
Learning reliable correspondences between images and point clouds is fundamental for 2D-3D matching. Despite recent progress in detection-free methods, existing approaches primarily optimize matching within a single model and often struggle to maintain reliable correspondences under challenging conditions such as noisy inputs, low overlap, and ambiguous structures. In this work, we propose TeaMatch, a novel framework that introduces teachability as a criterion for cross-modal representation learning. We define teachability as the ability of a representation to be effectively recovered by weak learners under degraded inputs, reflecting its structural consistency and robustness. To this end, we construct a set of task-specific weak students that simulate common failure modes and train them to imitate the teacher on a training split while evaluating their recoverability on a disjoint meta split. The teacher is then optimized to improve the students' ability to recover reliable correspondences, guided by correspondence-level and geometry-aware constraints. Our framework can be seamlessly integrated into existing coarse-to-fine matching pipelines without additional inference cost. Extensive experiments demonstrate that TeaMatch improves matching robustness and achieves state-of-the-art performance on challenging 2D-3D matching benchmarks.
Chongjian Wang, Junjie Gao
Shandong Women’s University · Jinan, China · Shandong University of Science and Technology +1
Visual correspondence across image-to-image (2D-2D), image-to-point cloud (2D-3D), and point cloud-to-point cloud (3D-3D) geometric matching forms the foundation for numerous 3D vision tasks. Despite sharing a similar problem structure, current methods use task-specific designs with separate models for each modality combination. We present UniCorrn, the first correspondence model with shared weights that unifies geometric matching across all three tasks. Our key insight is that Transformer attention naturally captures cross-modal feature similarity. We propose a dual-stream decoder that maintains separate appearance and positional feature streams. This design enables end-to-end learning through stack-able layers while supporting flexible query-based correspondence estimation across heterogeneous modalities. Our architecture employs modality-specific backbones followed by shared encoder and decoder components, trained jointly on diverse data combining pseudo point clouds from depth maps with real 3D correspondence annotations. UniCorrn achieves competitive performance on 2D-2D matching and surpasses prior state-of-the-art by 8% on 7Scenes (2D-3D) and 10% on 3DLoMatch (3D-3D) in registration recall. Project website: https://neu-vi.github.io/UniCorrn