Good Rankings, Poor Choices? Rethinking Language-Only Vision-Language Model Selection

Hyuk Kwon; Jaebyeong Jeon; Hyeonseo Jang; Kibok Lee

cs.CVOct 26, 2026

Good Rankings, Poor Choices? Rethinking Language-Only Vision-Language Model Selection

Authors: Hyuk Kwon, Jaebyeong Jeon, Hyeonseo Jang, Kibok Lee

Organizations: Yonsei University

Abstract

Language-Only Vision-Language Model Selection (LOVM) aims to identify the Vision-Language Model (VLM) with the highest zero-shot accuracy on a target task using only textual information. We revisit the existing LOVM evaluation protocol and argue that ranking metrics, such as top-5 recall and Kendall's rank correlation, do not directly assess the performance of the selected model. We therefore adopt simple regret (SR), defined as the performance gap between the best and selected models, and introduce Outlier-Robust Normalized Regret (ORNR) to account for the varying performance ranges across tasks while reducing sensitivity to outliers. Building on this revised evaluation protocol, we propose Target-Conditioned Expected Source Regret (TC-ESR), a training-free method that selects the model with the lowest similarity-weighted mean regret across source tasks. Across 22 target tasks and 43 VLMs, TC-ESR achieves the lowest mean SR and ORNR among all compared methods. Separately, we identify two sources of target performance leakage in the prior ranking-based evaluation protocol and provide corrections that remove both.

Figures & tables

Figure 1: Example based on the top-5 rankings produced by SWAB and TC-ESR on GTSRB. SWAB attains higher

R_{5}

and

\tau_{5}

, whereas TC-ESR selects the best-performing model.

Figure 2: Motivation for ORNR. (a) Similar SR can occupy different fractions of the accuracy range across tasks. (b) An unusually weak model can inflate the full range and distort normalization.

Figure 3: TC-ESR framework. Text-only task specifications are mapped to three-axis source–target similarities using frozen vocabularies and text embeddings. $\lambda_{t}^{\star}$ is selected by using cross validation over source tasks, and the resulting weights aggregate each model’s source task simple regrets. The model with the lowest TC-ESR score is selected.

Table 4

Table 5

Ranking metric	Rank corr. with SR	Disagreement (%)	Mean SR loss (%p)	Median SR loss (%p)
$R_{5}$	$-0.83$	25.7	3.32	1.82
$\tau_{5}$	$-0.47$	43.1	2.46	1.05

Table 5: Agreement between ranking metrics and SR over six methods and 22 targets.

Leakage removed	ModelGPT		SWAB		Avg. change
Leakage removed	$R_{5}$ $\uparrow$	$\tau_{5}$ $\uparrow$	$R_{5}$ $\uparrow$	$\tau_{5}$ $\uparrow$	Avg. change
None (original)	0.443	0.267	0.504	0.320	–
Feature selection	0.435	0.252	0.496	0.302	-3.7%
Exclude IN-1K	0.418	0.233	0.482	0.294	-7.7%
Both (final)	0.411	0.180	0.474	0.283	-14.3%

Table 6: Effect of protocol corrections on

R_{5}

and

\tau_{5}

for ModelGPT and SWAB.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Classes	Task	Domain
Imagenet ( Deng et al., 2009 )	1000	classification	natural image
SUN397 ( Xiao et al., 2010 )	397	scene und.	natural image
Country211 ( Radford et al., 2021 )	211	geolocation	natural image
Stanford Cars ( Krause et al., 2013 )	196	classification	natural image
Flowers102 ( Nilsback & Zisserman, 2008 )	102	classification	natural image
CIFAR100 ( Krizhevsky & Hinton, 2009 )	100	classification	natural image

Appendix

Table 7: The task information follows the LOVM and SWAB benchmarks ( Zohar et al., 2023 ; Yi et al., 2024 ) .

ID	Model	Name	Dataset	Name
1	RN50	RN50	openai	WIT
2	RN101	RN101	openai	WIT
3	RN50x4	RN50x4	openai	WIT
4	RN50x16	RN50x16	openai	WIT
5	RN50x64	RN50x64	openai	WIT
6	ViT-B-32	ViT-B/32	laion400m_e31	L400m

Appendix

Table 8: Detailed information on the 43 candidate models used in the LOVM benchmark. The model information follows the LOVM and SWAB benchmarks ( Zohar et al., 2023 ; Yi et al., 2024 ) .

Field	Category labels
Domain family	Object centered natural photographs; scene centered natural photographs; texture centered photographs; document or glyph images; medical images; aerial or satellite images; synthetic or rendered images
Label scale type	Unordered semantic taxonomy; binary state; ordered level; numeric count; symbol or code identity; multi factor composition

Appendix

Table 9: Category labels in the shared vocabularies for the two task metadata fields.

Task	Domain family	Agreement	Label scale type	Agreement	$\eta_{d}$
Stanford Cars	Object centered natural photographs	5/5	Unordered semantic taxonomy	5/5	1
CIFAR100	Object centered natural photographs	5/5	Unordered semantic taxonomy	5/5	1
CLEVR closest distance	Synthetic or rendered images	5/5	Ordered level	5/5	0
CLEVR count all	Synthetic or rendered images	5/5	Numeric count	5/5	0
Country211	Scene centered natural photographs	5/5	Unordered semantic taxonomy	5/5	1
Diabetic Retinopathy	Medical images	5/5	Ordered level	5/5	0

Appendix

Table 10: Task metadata for the 23 tasks. Agreement denotes the number of the five runs that assign the selected category. We set

\eta_{d}=1

only when the label scale type indicates a taxonomy of semantic categories and

n_{d}\geq 3

, and set

\eta_{d}=0

otherwise. The number of classes for each task is reported in Table 7 . ImageNet-1K is included during metadata construction and as a source task, but is excluded from the evaluation targets.

Table 12

Method	SR (%p) $\downarrow$	ORNR (%) $\downarrow$	$R_{5}$ (%) $\uparrow$	$\tau_{5}$ (%) $\uparrow$
INB	7.23	24.66	41.8	23.3
AvgRank	7.35	25.69	41.8	23.0
ModelGPT	7.11	24.25	41.1	18.0
SWAB	7.39	25.81	47.4	28.3
ESR	$\underline{5.83}$	$\underline{22.56}$	31.8	18.2
TC-ESR	4.66	17.81	40.0	22.7

Appendix

Table 14: Mean results over 22 target tasks under the revised LOVM evaluation protocol. SR is reported in percentage points, whereas ORNR,

R_{5}

, and

\tau_{5}

are reported as percentages. For ModelGPT and SWAB, each target result is averaged over ten seeds before macro averaging. Bold and underline denote the best and second-best mean results, respectively.

Method	Source Regret	SR (%p) $\downarrow$	ORNR (%) $\downarrow$
ESR	SR	5.83	22.56
ESR	ORNR	7.03	27.19
TC-ESR	SR	4.66	17.81
TC-ESR	ORNR	4.70	18.97

Appendix

Table 15: Effect of the regret used for source aggregation.

Table 15

	Method	cars	cifar100	clevr-distance	clevr-count	country211	retinopathy	dmlab	dtd	eurosat	fer2013	flowers	gtsrb	kitti	mnist	pcam	pets	renderedsst2	resisc45	stl10	sun397	svhn	voc2007	Avg.
SR ( $\%$ p) $\downarrow$	INB	0.00	0.10	5.37	11.68	3.56	49.81	9.06	0.00	5.87	2.70	0.49	8.73	12.52	13.03	9.09	0.41	14.39	0.00	0.49	1.27	10.56	0.00	7.23
	AvgRank	0.46	0.00	9.59	12.85	4.61	42.93	7.75	1.22	0.00	8.16	0.00	0.00	32.49	12.47	4.62	0.00	7.08	0.57	0.96	0.09	12.38	3.37	7.35
	ModelGPT	0.00	0.10	5.79	11.68	3.56	49.81	8.93	0.00	5.87	3.24	0.49	6.11	12.52	13.03	8.64	0.37	13.64	0.00	0.53	1.27	10.56	0.34	7.11
	SWAB	0.00	0.02	9.59	11.68	3.56	49.13	9.06	0.00	5.87	2.70	0.49	8.73	12.52	13.03	8.64	0.41	10.29	0.00	0.49	1.27	12.38	2.69	7.39
	ESR	0.29	1.99	3.27	9.58	7.93	20.57	5.60	0.21	0.04	2.68	3.17	0.17	32.35	10.12	10.46	0.79	15.32	0.95	0.86	1.82	0.00	0.13	5.83
	TC-ESR	0.00	0.10	3.27	15.16	4.61	2.22	5.60	0.00	5.87	2.70	0.49	0.00	12.52	10.12	5.94	0.41	14.94	0.57	0.49	0.09	17.52	0.00	4.66

Appendix

Table 18: Per-task results under the revised LOVM evaluation protocol. SR is reported in percentage points, whereas ORNR,

R_{5}

, and

\tau_{5}

are reported as percentages. For ModelGPT and SWAB, each per-task result is averaged over ten seeds. Bold and underline denote the best and second-best means in the Avg. column, respectively.

Figure 4: Task similarity matrix used by TC-ESR. Rows denote target tasks and columns denote source tasks, and each cell reports $S_{t}(s)$ . The score combines visual domain, label semantics, and label granularity when applicable. Crosses mark excluded pairs with $s=t$ , while ImageNet-1K appears only as a source task under the revised evaluation protocol. Because the inclusion of label granularity depends on the label scale type of the target task, the matrix need not be symmetric.

Type	Denominator	INB	AvgRank	ModelGPT	SWAB	ESR	TC-ESR
One-sided	Min–max	21.67	22.26	21.35	22.16	19.29	15.97
	$P_{5}$	26.99	28.65	26.61	28.78	23.78	19.52
	$P_{10}$	29.35	31.19	28.97	31.26	26.58	21.47
	1.0 IQR	26.05	28.48	25.78	28.07	23.63	19.01
	1.5 IQR (ORNR)	24.66	25.69	24.25	25.81	22.56	17.81
	2.0 IQR	24.20	25.19	23.79	25.23	21.82	17.41

Appendix

Table 19: Mean normalized regret over 22 targets under alternative denominators. ORNR corresponds to 1.5 IQR. Lower is better; bold and underline denote the best and second-best results, respectively.

Perturbation	Min–max	ORNR	Reduction (%)
Weakest candidate $\rightarrow$ 50%	4.58	1.19	74.0
Weakest candidate $\rightarrow$ 0	6.54	1.17	82.0
Bottom 5% removed	3.48	1.82	47.7
Bottom 10% removed	5.36	3.62	32.5
Bottom 20% removed	7.84	6.00	23.5

Appendix

Table 20: Mean absolute change in normalized regret when weak candidates are perturbed.

Method	Feature selection	SR	ORNR	$R_{5}$	$\tau_{5}$
ModelGPT	$R_{5}+\tau_{5}$	7.11	24.25	41.1	18.0
	SR	7.14	24.16	42.1	17.1
	ORNR	7.12	24.18	42.5	16.6
SWAB	$R_{5}+\tau_{5}$	7.39	25.81	47.4	28.3
	SR	8.58	28.40	34.3	10.2
	ORNR	8.10	26.42	36.6	10.9

Appendix

Table 21: ModelGPT and SWAB with feature selection retuned by SR or ORNR.

Method	Distinct candidate models	Share of the most frequent (%)
INB	1	100.0
AvgRank	1	100.0
ModelGPT	3	95.0
SWAB	5	80.9
ESR	2	95.5
TC-ESR	8	45.5

Appendix

Table 22: Distribution of the selected candidate models over 22 target tasks. For ModelGPT and SWAB, selections are pooled over ten seeds.

	SR (%p) $\downarrow$		ORNR (%) $\downarrow$
Method	w/ PCam	w/o PCam	w/ PCam	w/o PCam
INB	49.81	49.81	70.37	70.37
AvgRank	42.93	42.93	60.65	60.65
ModelGPT	49.81	49.81	70.37	70.37
SWAB	49.13	49.13	69.40	69.40
ESR	20.57	20.57	29.06	29.06

Appendix

Table 23: SR and ORNR on Retinopathy with and without PCam in the source tasks.

Domain family	Target	#Sources	$N_{\mathrm{eff},t}$
Object centered natural photographs	Stanford Cars	8	4.24
	CIFAR100	8	3.93
	FER2013	8	3.63
	Flowers102	8	5.25
	KITTI closest distance	8	6.24
	Oxford Pets	8	4.27

Appendix

Table 24: Effective number of source tasks

N_{\mathrm{eff},t}

of TC-ESR for each target, grouped by domain family. #Sources denotes the number of source tasks sharing the target’s domain family, counting ImageNet-1K. For every target with at least one such source, these sources receive more than 99.3% of the total weight.

Method	SR (%p) $\downarrow$	ORNR (%) $\downarrow$
INB	5.20	22.48
AvgRank	5.65	24.03
ModelGPT	5.08	22.06
SWAB	5.40	23.73
ESR	5.13	22.25
TC-ESR	4.78	18.51

Appendix

Table 25: Mean SR and ORNR over 21 target tasks excluding Retinopathy. Bold and underline denote the best and second-best LOVM methods.

Model Selection

Model selection chooses models, features, or configurations that balance predictive fit, generalization, complexity, and deployment constraints.

See all 96 papers

Model Specific Task Similarity for Vision Language Model Selection via Layer Conductance
Feb 1, 2026Wei Yang, Hong Xie, Tao Tan +3VLM Evaluation Vision-Language Models
School of AI and Data Science USTC
Can Argus Judge Them All? Comparing VLMs Across Domains
Jun 23, 2025Harsh Joshi, Gautam Siddharth Kashyap, Rafiq Ali +5VLM Evaluation VLM Robustness
Bharati Vidyapeeth, New Delhi, India · Macquarie University, Sydney, Australia · DSEU-Okhla, New Delhi, India +3
An Effective Router for Vision-Language Model Selection
Jun 8, 2026Can Wang, Shengwei Wang, Bolin Zhang +2Vision-Language Models
Harbin Institute of Technology Harbin, China · Harbin Institute of Technology Shandong Key Laboratory of Digital Service Computing Technology and Systems Harbin, China

Feb 1, 2026J/K move · Enter open · S save

Model Specific Task Similarity for Vision Language Model Selection via Layer Conductance

Wei Yang, Hong Xie, Tao Tan, Xin Li+2

School of AI and Data Science USTC

VLM Evaluation Vision-Language Models Task Transferability Estimation Transfer Learning

Open paper AskPDF arXiv

While open sourced Vision-Language Models (VLMs) have proliferated, selecting the optimal pretrained model for a specific downstream task remains challenging. Exhaustive evaluation is often infeasible due to computational constraints and data limitations in few shot scenarios. Existing selection methods fail to fully address this: they either rely on data-intensive proxies or use symmetric textual descriptors that neglect the inherently directional and model-specific nature of transferability. To address this problem, we propose a framework that grounds model selection in the internal functional dynamics of the visual encoder. Our approach represents each task via layer wise conductance and derives a target-conditioned block importance distribution through entropy regularized alignment. Building on this, we introduce Directional Conductance Divergence (DCD), an asymmetric metric that quantifies how effectively a source task covers the target's salient functional blocks. This allows for predicting target model rankings by aggregating source task ranks without direct inference. Experimental results on 48 VLMs across 21 datasets demonstrate that our method outperforms state-of-the-art baselines, achieving a 14.7% improvement in NDCG@5 over SWAB.

Abstract

Model Selection

Explore similar work

Model Specific Task Similarity for Vision Language Model Selection via Layer Conductance

Can Argus Judge Them All? Comparing VLMs Across Domains

An Effective Router for Vision-Language Model Selection