cs.CVOct 29, 2026
SaveCrossWeave: Cross-Modal Scene and Instance Retrieval from Sparse 2D-3D Alignment
Organizations: IIIT Hyderabad · University of Heidelberg · MBZUAI · IIT Bombay
Abstract
Cross-modal scene and instance retrieval aims to retrieve a corresponding 3D environment or object given query 2D images, serving as a foundational mechanism for spatial reasoning in robotics and AR/VR. Existing methods achieve this by learning alignment only at the level of whole scenes or objects, which provides no supervision for the fine-grained local structure within them. In this work, we show that supervision on a set of sparse 2D-3D correspondences, without scene- or instance-level contrastive objectives, is sufficient to train a single model for both scene- and instance-level retrieval. Specifically, we propose , a novel framework for cross-modal retrieval that learns sparse patch-point alignment, which alone outperforms existing methods. An optimal transport-based aggregator over the aligned features gives further gains by clustering structurally redundant regions, freeing descriptor capacity for semantically distinct local features. CrossWeave achieves state-of-the-art performance on both scene- and instance-level retrieval on ScanNet and 3RScan with a , whereas existing methods require separate models for each task. Our results suggest that whole-scene encoding is unnecessary for cross-modal retrieval and that sparse local correspondence supervision is a more effective and flexible alternative.
Figures & tables
Figure 1: Overview of CrossWeave. CrossWeave learns cross-modal retrieval representations from sparse image patch-3D point correspondences rather than scene- or instance-level labels. Scene Sampling: Given calibrated RGB-D observations and a reconstructed 3D map, visible patch-point pairs are obtained by projecting 3D points into the image plane, filtering occluded points using rendered depth, quantizing valid pixels into DINOv2 patch tokens, and selecting a compact semantically diverse subset using feature-space farthest point sampling. Feature Extraction: Frozen encoders extract local patch and point features, which are projected by learnable projection matrices and into a shared embedding space. Feature Aggregation: A shared SALAD aggregator then maps variable-sized sets of local features into fixed-dimensional descriptors. Training combines a local patch-point contrastive loss with an aggregated image-point-cloud contrastive loss , preserving fine-grained alignment while organizing local evidence into retrieval-ready descriptors. At inference, the same encoders and projection heads serve both tasks: SALAD descriptors are used for scene retrieval and mean-pooled local features for instance retrieval.
| ScanNet | 3RScan | |||||
|---|---|---|---|---|---|---|
| Scene Recall | Scene Recall | |||||
| Method | Top-1 | Top-5 | Top-10 | Top-1 | Top-5 | Top-10 |
| Scene CrossOver | 22.44 | 61.86 | 75.32 | 19.11 | 58.60 | 77.07 |
| Scene CrossOver † | 19.87 | 58.01 | 74.36 | 17.83 | 51.59 | 73.25 |
| Scene CrossOver (Updated) | 40.38 | 90.06 | 96.47 | 23.57 | 66.88 | 84.71 |
| Ours | 61.54 | 99.36 | 100.00 | 46.50 | 85.99 | 92.36 |
Table 1: Comparison of scene retrieval performance on ScanNet and 3RScan. † Trained on additional datasets (ArkitScenes [ 2 ] + MultiScan [ 18 ] ). Best and Second-best results.
Figure 2: Qualitative comparison of cross-modal scene retrieval on ScanNet. Given a set of query RGB images, each model retrieves the most similar 3D scenes from the database. The ground-truth scene is shown on the left, and retrieved scenes are ordered by similarity from left to right. CrossOver and CrossOver (Updated) fail to retrieve the correct scene within the top five results, while CrossWeave retrieves it at rank 1. The remaining retrieved scenes are also semantically consistent, indicating that CrossWeave learns descriptors that capture both local cross-modal correspondences and global scene context.
Table 2: Comparison of within-scene instance retrieval performance on ScanNet and 3RScan. † Trained on additional datasets (ArkitScenes [ 2 ] + MultiScan [ 18 ] ). Best and Second-best results.
Figure 3: Qualitative instance retrieval on ScanNet. For each query view with its instance mask (left), CrossWeave ranks the instances of the same scene (middle) by cosine similarity. The Top-3 retrievals are shown with their similarity scores, and the correct match is outlined in green.
| 10 candidates | 50 candidates | |||||
|---|---|---|---|---|---|---|
| Method | Top-1 | Top-5 | Top-10 | Top-1 | Top-5 | Top-10 |
| SceneGraphLoc ‡ [ 19 ] | 53.60 | 81.90 | 92.80 | 30.20 | 50.20 | 61.20 |
| CrossOver [ 27 ] | 46.00 | 77.97 | 90.58 | 18.69 | 39.16 | 51.62 |
| Ours | 53.60 | 93.10 | 100.00 | 27.50 | 60.20 | 76.50 |
Table 3: Scene retrieval on 3RScan against SceneGraphLoc and CrossOver. Each query image is ranked against candidate scenes; Top- : percentage of queries with the correct scene in the top . ‡ Uses ground-truth scene graphs at inference. Best and second-best results are highlighted.
| ScanNet | 3RScan | ||||||||
| Scene | Instance | Scene | Instance | ||||||
| Agg. | Feats | Top-1 | Top-1 | R@75% | Top-1 | Top-1 | R@75% | ||
| Mean | ✗ | ✓ | local | 47.12 | 78.96 | 70.51 | 27.39 | 73.30 | 47.13 |
| Mean | ✓ | ✓ | local | 52.24 | 79.63 | 71.15 | 30.57 | 73.62 | 49.04 |
| SALAD | ✓ | ✗ | salad | 57.37 | 73.33 | 56.09 | 43.95 | 66.98 | 38.22 |
| SALAD | ✓ | ✗ | local | 38.14 | 73.31 | 52.88 | 19.75 | 67.43 | 36.31 |
Table 4: Ablation of training losses, aggregation and retrieval features. Agg. : aggregation used during training. Feats : salad (aggregated descriptors), local (mean-pooled patch and point features), or hybrid ( salad for scene, local for instance retrieval). Best and second-best results are highlighted; the main model is shaded and marked with ⋆ .
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Scene retrieval performance (Top-1 recall) on ScanNet and 3RScan as a function of training data volume. Performance is stable across data regimes, with even 10% of the full training set sufficient to outperform the previous state of the art.
Figure 5: Scene retrieval performance (Top-1 recall) as a function of the number of query camera views. The main results use 10 views.
| Instance Retrieval | Scene-Level Recall | |||||
|---|---|---|---|---|---|---|
| Dataset | Mask condition | Top-1 | Top-3 | R@25% | R@50% | R@75% |
| ScanNet | GT | 78.77 | 94.05 | 98.40 | 88.78 | 67.95 |
| 2D noisy | 75.75 | 93.02 | 97.44 | 88.14 | 62.50 | |
| 3D noisy | 75.33 | 92.91 | 97.12 | 87.50 | 63.46 | |
| 2D + 3D noisy | 72.29 | 91.86 | 97.12 | 85.90 | 55.45 | |
| 3RScan | GT | 73.50 | 92.73 | 98.09 | 92.99 | 49.68 |
Table 5: Instance retrieval under noisy instance masks, using the main model with local features.
Figure 6: Visualizations for the effect of FPS sampling. Images with red dots denote random sampling. Images with blue dots denote FPS sampled points. It can clearly be observed that FPS enables us to avoid over-sampling on the often visually dominant but geometrically uninformative areas in the images such as walls, floors and other flat surfaces, thereby boosting feature diversity within each training batch. Note: We have visualized a large number of pixel-point pairs in these images; in our training framework we only use 32 patch-point pairs. Refer to Section 3.1 for more details.
Figure 7: Comparisons of Top-5 retrievals on a ScanNet room scene. Retrievals are ordered from most to least similar moving left to right, with correct matches highlighted in green. CrossWeave successfully retrieves the correct scene at rank 1, while CrossOver (Updated) retrieves it at rank 2, and the base model of CrossOver fails within the top 5.
Figure 8: Additional qualitative results on a ScanNet office scene. CrossWeave retrieves the ground truth point cloud as its top match, whereas both CrossOver (Updated) and CrossOver only identify the correct scene at rank 5.
Figure 9: Qualitative results on a 3RScan dining area scene. CrossWeave successfully retrieves the ground truth point cloud as its top match, while CrossOver (Updated) ranks it fourth and the base CrossOver model fails to retrieve it within the top 5.
Figure 10: Retrieval results on a 3RScan kitchen and dining scene. CrossWeave demonstrates superior discriminative ability by retrieving the exact scene at rank 1 from highly localized query images, whereas both baseline models completely fail to place the correct match within their top 5 retrievals.
Figure 12: Qualitative instance retrieval on 3RScan. For each query view with its instance mask (left), CrossWeave ranks the instances of the same scene (middle) by cosine similarity. The Top-3 retrievals are shown with their similarity scores, and the correct match is outlined in green.
Figure 13: Failure case on a 3RScan living room scene. All three models miss the ground-truth scene in the top 5, as the retrieved scenes are visually and structurally similar to the query (a brown sectional sofa with a wooden coffee table). CrossWeave’s top matches still share the furniture arrangement (sofas and central coffee tables), whereas CrossOver (Updated) retrieves structurally different scenes.
Figure 14: Instance retrieval failures on ScanNet and 3RScan. In both cases the query is a chair and the Top-1 retrieval (red) is another chair of the same type in the same scene, while the correct instance (green) is ranked second with a similar score.
More in this topic
Cross-Modal Retrieval
Cross-modal retrieval matches queries and items across different data types, such as text, images, video, audio, and sensor signals.
See all 187 papers