Large-scale instance-level training data is scarce, so models are typically trained on domain-specific datasets. Yet in real-world retrieval, they must handle diverse domains, making generalization to unseen data critical. We introduce ELViS, an image-to-image similarity model that generalizes effectively to unseen domains. Unlike conventional approaches, our model operates in similarity space rather than representation space, promoting cross-domain transfer. It leverages local descriptor correspondences, refines their similarities through an optimal transport step with data-dependent gains that suppress uninformative descriptors, and aggregates strong correspondences via a voting process into an image-level similarity. This design injects strong inductive biases, yielding a simple, efficient, and interpretable model. To assess generalization, we compile a benchmark of eight datasets spanning landmarks, artworks, products, and multi-domain collections, and evaluate ELViS as a re-ranking method. Our experiments show that ELViS outperforms competing methods by a large margin in out-of-domain scenarios and on average, while requiring only a fraction of their computational cost. Code available at: https://github.com/pavelsuma/ELViS/
Figures & tables
Figure 1: Performance vs. time. Average performance across 8 datasets and multiple domains for fixed numbers of re-ranked images indicated with text labels. All models are trained on the landmarks domain (GLDv2). Runtime is estimated from model latencies reported in Table 5 .
Figure 2: Detailed overview of ELViS. The similarity matrix is refined using optimal transport with descriptor-dependent dustbin gains. The strongest local similarities per descriptor are then selected and transformed element-wise by a learned function f , before being sum-aggregated into a scalar global similarity. During training, a modified BCE loss with a learnable function g reshapes the penalty curve; g is used only for training and is expandable at inference.
Figure 3: Shape of the learned univariate functions f (left) and g (right). Although parameterized as MLPs, both functions learn well-behaved scalar transformations that effectively separate matching and non-matching distributions. The distributions of input values are visualized separately for positive and negative image pairs, sampled during training.
Figure 4: Visualization of the 25 strongest correspondences (votes) among si,sj (left) and si′,sj′ (right) before (left) and after (right) refinement with optimal transport. Red (yellow) represents high (low) similarity. Raw similarity values in S (left) and values in S′ after passing them through f (right) are used. Heatmaps represent the dustbins values by evaluating h densely for all patches in both images; bright values indicate large dustbin gain and uninformative descriptors.
Method
R OP+1M
GLDv2
ILIAS
INSTRE
MET
Prod1M
RP2K
SOP-1k
ID
OOD
avg
landmarks domain (GLDv2)
No re-ranking
57.7
27.3
9.4
65.3
61.6
24.7
39.0
33.7
42.5
38.9
39.8
Chamfer †
56.7
23.8
6.2
55.0
37.3
17.8
55.8
48.6
40.2
36.8
37.6
Chamfer+OT †
60.6
23.8
14.3
76.0
74.0
35.1
55.4
46.2
42.2
50.2
48.2
RRT ⋆
69.2
33.1
13.1
72.4
64.1
29.3
60.7
52.1
51.1
48.6
49.2
R2 Former †
68.5
32.6
15.2
77.6
72.0
35.6
47.7
43.7
50.6
48.6
49.1
Table 1: Domain generalization performance (mAP) . Training performed either on landmarks (GLDv2) or household items (SOP). Results reported per dataset and as average over in-domain (ID), out-of-domain (OOD), and all datasets (avg). Local descriptors extracted with DINOv2 ( Oquab et al., 2024 ) . Gray indicates in-domain results. Green ( Red ) highlights gain (loss) of ELViS over the second best method. † , ⋆ indicate similarity-based and descriptor-based models, respectively.
Method ID OOD avg No re-ranking 49.6 60.5 57.8 Chamfer 42.3 44.5 44.0 Chamfer+OT 41.7 51.4 49.0 RRT 57.1 56.2 56.4 R2 Former 56.0 63.9 61.9 AMES 58.6 62.8 61.8 ELViS 56.9 (-1.7) 67.4 (+3.5) 64.8 (+2.9)
Method ID OOD avg No re-ranking 25.4 57.8 49.7 Chamfer 32.8 58.7 52.2 Chamfer+OT 29.3 62.3 54.1 RRT 37.5 58.4 53.2 R2 Former 35.2 63.0 56.0 AMES 37.1 62.7 56.3 ELViS 36.4 (-1.1) 68.7 (+5.7) 60.6 (+4.3)
(a) DINOv3 ( Siméoni et al., 2025 )
(b) SigLIP2 ( Tschannen et al., 2025 )
Table 2: Performance (mAP) comparison using local descriptors from additional foundational models . Training performed on landmarks (GLDv2). Results reported per dataset and as averages over in-domain (ID), out-of-domain (OOD), and all datasets (avg). Green ( Red ) highlights gain (loss) of ELViS over the second best method.
Method
ID
OOD
avg
No re-ranking
42.5
38.9
39.8
ELViS
50.5
55.0
53.9
w/o dustbin
23.1
(-27.4)
32.8
(-22.2)
30.4
(-23.5)
w/o descriptor-dependent gain
48.8
(-1.7)
52.4
(-2.6)
51.5
(-2.4)
w/o f function
50.8
(+0.3)
53.8
(-1.2)
53.1
(-0.8)
w/o g function
45.6
(-4.9)
49.5
(-5.5)
48.5
(-5.4)
Table 3: Ablation study on method components . Average, ID, and OOD performance, when training on landmarks. Green ( Red ) highlights gain (loss) over ELViS.
Table 8
Figure 5: Effectiveness with small training sets. Performance of ELViS and AMES when trained on subsets of the full training set (754k images), ranging from 0.2% to 100%. No re-ranking performance is indicated at 0 training set size. Training hyperparameters are tuned for each subset size based on validation performance.
dataset
train set
validation set
test set
domain
evaluation metric
queries
database
queries
database
GLDv2
754K
379
762K
750
762K
landmark
mAP@100
SOP
48.9K
1K
10.6K
1K
60.5K
household
mAP@100
R Oxford
–
–
–
70
5K+1M
landmark
mAP
R Paris
–
–
–
70
6.3K+1M
landmark
mAP
Product1M
–
–
–
6.2K
38.7K
retail
mAP@100
Table F: Dataset statistics and metrics used . For each dataset, we use the most commonly used metric. Train and validation set statistics are reported only for the two training datasets that are used in this work.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure F: Benchmark examples . Three random samples are shown for each of the 9 datasets. All images are resized to a square aspect ratio for better visualization.
Method
Alps
California
Gobi Desert
Amazon
Toshka Lakes
R@1
R@10
R@100
R@1
R@10
R@100
R@1
R@10
R@100
R@1
R@10
R@100
R@1
R@10
R@100
AnyLoc
40.7
70.8
92.0
48.7
75.0
91.6
28.7
57.0
81.7
38.6
63.8
86.2
63.7
84.5
96.3
AstroLoc
98.1
99.5
99.8
97.4
99.2
99.8
94.6
99.2
99.9
93.0
96.9
99.1
99.0
99.6
99.9
No re-ranking
35.7
65.6
91.1
50.1
78.2
93.9
26.2
53.3
84.0
31.4
61.1
83.9
52.4
78.1
94.9
AMES
44.0
74.5
93.7
57.4
83.0
95.7
36.0
64.7
87.0
40.4
69.2
87.8
61.5
85.4
96.8
ELViS
55.1
80.0
94.6
62.0
82.7
94.7
49.7
71.2
86.9
54.4
76.2
89.0
77.0
90.9
97.6
Appendix
Table G: Generalization to extreme domain shifts. Evaluation on retrieval for Astronaut Photography Localization (APL) ( Berton et al., 2024 ) . Performance is measured in Recall@ k (R@ k ). APL covers different geographical areas of extreme visual environments. Top: reference methods AnyLoc ( Keetha et al., 2023 ) , a universal Visual Place Recognition (VPR) approach and the specialist AstroLoc ( Berton et al., 2025 ) trained for the task. Bottom: Re-ranking 400 images on top of a DINOv3 backbone. Bold indicates the best performance.
Method
R OP+1M
GLDv2
ILIAS
INSTRE
MET
Prod1M
RP2K
SOP-1k
ID
OOD
avg
No re-ranking
57.7
27.3
9.4
65.3
61.6
24.7
39.0
33.7
42.5
38.9
39.8
Spatial verification
58.5
27.6
12.3
70.4
68.7
28.7
49.7
37.3
43.3
43.7
43.6
SuperGlue
61.3
28.3
15.1
73.7
72.9
38.8
56.2
41.5
44.8
49.7
48.5
OmniGlue
59.1
27.8
13.6
73.0
70.8
19.4
44.4
41.1
43.4
43.7
43.7
ELViS
68.8
32.2
18.8
80.4
77.9
41.5
59.2
52.3
50.5
55.0
53.9
Appendix
Table H: Comparison with image matching methods . Landmark datasets are treated as in-domain (ID) for all methods, while the remaining datasets are evaluated as out-of-domain (OOD).
λ
ID
OOD
avg
0.01
46.4
47.7
47.4
0.1
50.5
55.0
53.9
0.5
50.7
52.1
51.7
1.0
48.9
48.2
48.4
learnable
51.2
54.9
54.0
Appendix
Table I: Ablation study of the OT hyperparameters . Models are trained on the Landmarks (GLDv2) domain. We report average performance over the in-domain (ID), out-of-domain (OOD), and all benchmarks (avg). We assess the impact of the regularization parameter λ and the number of OT iterations, varying one hyperparameter at a time while keeping all others at their default values.
Figure G: Average precision per query. Each marker corresponds to the AP of a single query evaluated on different test datasets using DINOv2 as a representation model. x-axis: performance using ELViS trained on SOP dataset. y-axis: performance using ELViS trained on GLDv2 dataset.
Figure H: Average precision per query. Each marker corresponds to the AP of a single query evaluated on INSTRE using DINOv2 as a representation model. y-axis: performance using ELViS trained on SOP dataset. x-axis: performance of the global descriptor without any re-ranking.
Method
R Oxford
R Paris
GLDv2
ILIAS
INSTRE
MET
Prod1M
RP2K
SOP-1k
ID
OOD
avg
DINOv2 descriptors ( Oquab et al., 2024 )
No re-ranking
47.3
67.9
27.3
9.4
65.3
61.6
24.7
39.0
33.7
42.5
38.9
39.8
Chamfer
43.0
54.8
23.8
6.2
55.0
37.3
17.8
55.8
48.6
40.2
36.8
37.6
Chamfer+OT
55.6
65.5
23.8
14.3
76.0
74.0
35.1
55.4
46.2
42.2
50.2
48.2
RRT
64.1 ±0.6
74.4 ±0.0
33.1 ±0.2
13.1 ±0.6
72.4 ±1.1
64.1 ±0.7
29.3 ±1.7
60.7 ±0.9
52.1 ±0.6
51.1 ±0.3
48.6 ±0.9
49.2 ±0.8
R2 Former
63.7 ±0.2
73.4 ±0.1
32.6 ±0.0
15.2 ±0.1
77.6 ±0.1
72.0 ±0.3
35.6 ±0.3
47.7 ±0.6
43.7 ±0.1
50.6 ±0.1
48.6 ±0.2
49.1 ±0.2
Appendix
Table J: mAP mean and standard deviation for training on landmarks (GLDv2). Three different backbones used as a representation model.
Method
R Oxford
R Paris
GLDv2
ILIAS
INSTRE
MET
Prod1M
RP2K
SOP-1k
ID
OOD
avg
No re-ranking
47.3
67.9
27.3
9.4
65.3
61.6
24.7
39.0
33.7
33.7
40.7
39.8
Chamfer
29.7
62.6
15.4
6.7
63.2
45.2
24.7
50.3
50.2
50.2
35.9
37.7
Chamfer+OT
43.7
60.8
18.6
11.7
75.9
71.6
37.5
52.7
45.8
45.8
45.7
45.7
RRT
28.2 ±1.7
58.7 ±1.3
10.8 ±1.1
12.2 ±1.5
68.4 ±1.8
25.5 ±2.8
32.2 ±1.8
46.9 ±3.9
57.1 ±0.3
57.1 ±0.3
34.2 ±2.0
37.1 ±1.5
R2 Former
43.4 ±1.0
67.5 ±0.3
23.7 ±0.5
12.9 ±0.3
73.4 ±0.4
59.3 ±0.7
32.8 ±1.9
42.0 ±0.7
51.1 ±0.3
51.1 ±0.3
42.8 ±0.7
43.8 ±0.7
AMES
46.2 ±1.0
65.1 ±1.5
17.2 ±1.1
12.4 ±0.5
72.2 ±1.0
44.2 ±1.8
36.7 ±1.9
51.8 ±2.5
56.7 ±0.6
56.7 ±0.6
41.4 ±1.5
43.3 ±1.2
Appendix
Table K: mAP mean and standard deviation for training on household items (SOP). DINOv2 used as a representation model.
Figure I: The impact of re-ranking with ELViS. We show a single query and its database positives per row for each test set, with the exception of the MET dataset where three queries are shown. The text above each query denotes the change in Average Precision. The text above the positive images denotes the number of negative images ranked before the positive using only global similarity (1) and after re-ranking with ELViS (2), denoted as (1)→(2). The positives are ordered based on the difference (1)-(2) in descending order.
Figure J: Visualization of strong correspondences before (left) and after (right) refinement with optimal transport.
Figure K: Visualization of similarity matrices before (left) and after (right) refinement with optimal transport with individual values passed through function f . A subset of 100 descriptors is used. These examples correspond to the ones of Figure J .
Generalizing image classification across domains remains challenging in critical tasks such as fundus image-based diabetic retinopathy (DR) grading and resting-state fMRI seizure onset zone (SOZ) detection. When domains differ in unknown causal factors, achieving cross-domain generalization is difficult, and there is no established methodology to objectively assess such differences without direct metadata or protocol-level information from data collectors, which is typically inaccessible. We first introduce domain conformal bounds (DCB), a theoretical framework to evaluate whether domains diverge in unknown causal factors. Building on this, we propose GenEval, a multimodal Vision Language Models (VLM) approach that combines foundational models (e.g., MedGemma-4B) with human knowledge via Low-Rank Adaptation (LoRA) to bridge causal gaps and enhance single-source domain generalization (SDG). Across eight DR and two SOZ datasets, GenEval achieves superior SDG performance, with average accuracy of 69.2% (DR) and 81% (SOZ), outperforming the strongest baselines by 9.4% and 1.8%, respectively.
Recent advances in vision-language models (VLMs) have demonstrated remarkable zero-shot capabilities, yet adapting these models to specialized domains remains a significant challenge. Building on recent theoretical insights suggesting that independently trained VLMs are related by a canonical transformation, we extend this understanding to the concept of domains. We hypothesize that image features across disparate domains are related by a canonicalized geometric transformation that can be recovered using a small set of anchors. Few-shot classification provides a natural setting for this alignment, as the limited labeled samples serve as the anchors required to estimate this transformation. Motivated by this hypothesis, we introduce BiCLIP, a framework that applies a targeted transformation to multimodal features to enhance cross-modal alignment. Our approach is characterized by its extreme simplicity and low parameter footprint. Extensive evaluations across 11 standard benchmarks, including EuroSAT, DTD, and FGVCAircraft, demonstrate that BiCLIP consistently achieves state-of-the-art results. Furthermore, we provide empirical verification of existing geometric findings by analyzing the orthogonality and angular distribution of the learned transformations, confirming that structured alignment is the key to robust domain adaptation. Code is available at https://github.com/QuantitativeImagingLaboratory/BilinearCLIP
Pranav Mantini, Shishir K. Shah
University of Houston, Department of Computer Science · The University of Oklahoma, School of Computer Science
Multimodal Large Language Models (MLLMs) have demonstrated strong cross-modal reasoning capabilities, yet their potential for vision-only tasks remains underexplored. We investigate MLLMs as training-free similarity estimators for instance-level image-to-image retrieval. Our approach prompts the model with paired images and converts next-token probabilities into similarity scores, enabling zero-shot re-ranking within large-scale retrieval pipelines. This design avoids specialized architectures and fine-tuning, leveraging the rich visual discrimination learned during multimodal pre-training. We address scalability by combining MLLMs with memory-efficient indexing and top-k candidate re-ranking. Experiments across diverse benchmarks show that MLLMs outperform task-specific re-rankers outside their native domains and exhibit superior robustness to clutter, occlusion, and small objects. Despite strong results, we identify failure modes under severe appearance changes, highlighting opportunities for future research. Our findings position MLLMs as a promising alternative for open-world large-scale image retrieval.
Bahey Tharwat, Giorgos Kordopatis-Zilos, Pavel Suma +2
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · VRG, FEE, Czech Technical University in Prague, Prague, Czech Republic