Large Language Model Selection with Limited Annotations

Yavuz Durmazkeser; Patrik Okanovic; Andreas Kirsch; Torsten Hoefler; Nezihe Merve Gürel

cs.CLOct 10, 2025

Large Language Model Selection with Limited Annotations

Authors: Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch, Torsten Hoefler, Nezihe Merve Gürel

Organizations: TU Delft · ETH Zurich

Abstract

Choosing a Large Language Model (LLM) for a given task requires comparing many strong candidates, yet standard evaluation relies on costly annotations over fixed evaluation sets. To address this challenge, we develop SELECT-LLM, the first framework for active model selection of LLMs. SELECT-LLM aims to find a small set of queries whose annotations are most informative for identifying the best LLM for a given task. To this end, we introduce a query selection rule based on expected information gain, computed from pairwise similarities between candidate model outputs. Because this rule only uses generated model responses, SELECT-LLM can be applied across candidate models without assumptions about their architecture or access to model weights. This makes it suitable for both open-weight and black-box LLMs. We evaluate SELECT-LLM across 23 datasets, 156 evaluated models, diverse task families, and multiple text evaluation metrics. Across all experiments, SELECT-LLM improves over the strongest baseline in every setting, with annotation cost reductions up to 81.8% for best model selection and up to 84.78% for near-best model selection.

Figures & tables

Figure 1: An overview of Select-LLM . For an arbitrary pool of $n$ queries and a set of candidate language models, Select-LLM adaptively annotates most informative $b\ll n$ queries for identifying the best language model for the pool.

Figure 2: Comparison of Select-LLM and the baselines in terms of best model identification probability across 23 datasets. In each plot, the horizontal arrow and percentage indicate Select-LLM ’s labeling efficiency relative to the strongest baseline, and the dashed vertical lines mark the corresponding budgets. Each plot is shown until the strongest baseline reaches

100\%

identification probability.

Dataset	$\delta=0.1\%$	$\delta=0.5\%$	$\delta=1\%$
bAbI	$\downarrow\mathbf{50.74}\%$	$\downarrow\mathbf{47.64}\%$	$\downarrow\mathbf{50.25}\%$
BBH	$\downarrow\mathbf{28.71}\%$	$\downarrow\mathbf{39.19}\%$	$\downarrow\mathbf{35.10}\%$
CNN/DailyMail (ROUGE-L)	$\downarrow\mathbf{69.15}\%$	$\downarrow\mathbf{69.15}\%$	$\downarrow\mathbf{69.15}\%$
CNN/DailyMail (BERTScore)	$\downarrow\mathbf{78.99}\%$	$\downarrow\mathbf{77.14}\%$	$\downarrow\mathbf{72.73}\%$
Flickr30k	$\downarrow\mathbf{38.05}\%$	$\downarrow\mathbf{38.05}\%$	$\downarrow\mathbf{38.05}\%$
GPQA	$\downarrow\mathbf{43.78}\%$	$\downarrow\mathbf{43.66}\%$	$\downarrow\mathbf{43.43}\%$

Table 1: Annotation efficiency for near-best model selection across datasets. Each

\delta

denotes a relative gap from the best model. Bolded

\downarrow

values indicate fewer labels than the strongest baseline;

\uparrow

indicates more labels.

Dataset	Select-LLM	Random	Margin	Min Agreement	VMA	AMC
Ident. prob.	(70%/80%/90%/100%)	(70%/80%/90%/100%)	(70%/80%/90%/100%)	(70%/80%/90%/100%)	(70%/80%/90%/100%)	(70%/80%/90%/100%)
bAbI	$\mathbf{2.00}$ / $\mathbf{1.40}$ / $\mathbf{0.60}$ / $\mathbf{0.00}$	$2.60$ / $2.60$ / $2.60$ / $2.20$	$2.60$ / $2.60$ / $2.60$ / $2.20$	$2.60$ / $2.60$ / $2.60$ / $2.20$	$\underline{2.40}$ / $\underline{2.40}$ / $\underline{2.20}$ / $\underline{2.00}$	$2.60$ / $\underline{2.40}$ / $2.40$ / $2.20$
BBH	$3.00$ / $2.10$ / $\mathbf{0.80}$ / $\mathbf{0.00}$	$2.90$ / $2.60$ / $2.30$ / $0.90$	$2.80$ / $2.60$ / $2.40$ / $0.80$	$\mathbf{2.10}$ / $\underline{2.00}$ / $1.80$ / $\underline{0.10}$	$\mathbf{2.10}$ / $\mathbf{1.90}$ / $\underline{1.60}$ / $0.20$	$\underline{2.20}$ / $\underline{2.00}$ / $\underline{1.60}$ / $0.30$
CNN/DailyMail (ROUGE-L)	$\mathbf{1.25}$ / $\mathbf{1.25}$ / $\mathbf{1.05}$ / $\mathbf{0.00}$	$7.50$ / $7.50$ / $2.92$ / $1.91$	$7.62$ / $7.62$ / $3.44$ / $3.02$	$\underline{3.21}$ / $\underline{3.21}$ / $\underline{2.19}$ / $\underline{1.34}$	$7.47$ / $7.47$ / $3.13$ / $1.95$	$7.57$ / $7.57$ / $3.01$ / $1.83$
CNN/DailyMail (BERTScore)	$\mathbf{2.63}$ / $\mathbf{1.64}$ / $\mathbf{1.18}$ / $\mathbf{0.00}$	$7.55$ / $7.37$ / $7.18$ / $\mathbf{0.00}$	$5.75$ / $4.49$ / $2.63$ / $\mathbf{0.00}$	$\underline{3.24}$ / $\underline{3.28}$ / $\underline{2.27}$ / $\mathbf{0.00}$	$7.39$ / $6.89$ / $6.27$ / $\mathbf{0.00}$	$7.44$ / $7.34$ / $6.97$ / $\mathbf{0.00}$
Flickr30k	$\mathbf{1.63}$ / $\mathbf{1.37}$ / $\mathbf{1.13}$ / $\mathbf{0.00}$	$2.54$ / $2.12$ / $\underline{1.80}$ / $\mathbf{0.00}$	$3.16$ / $2.51$ / $2.33$ / $\mathbf{0.00}$	$\underline{1.89}$ / $\underline{1.70}$ / $2.13$ / $\mathbf{0.00}$	$2.59$ / $2.05$ / $1.91$ / $\mathbf{0.00}$	$2.53$ / $2.06$ / $1.91$ / $\mathbf{0.00}$

Table 2: 95th percentile performance gap (%) at the budget where Select-LLM reaches identification probabilities 70%/80%/90%/100% on each dataset. Best values are bold; second-best are underlined.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

2 models
$\max P_{j}$	Top-1	Top-5%	Spearman	Pairwise
$0.50$	$1.000$	$1.000$	$0.996$	$0.984$
$0.60$	$1.000$	$1.000$	$0.995$	$0.981$
$0.70$	$1.000$	$1.000$	$0.996$	$0.982$
$0.80$	$1.000$	$1.000$	$0.995$	$0.982$
$0.90$	$1.000$	$1.000$	$0.995$	$0.981$

Appendix

Table 3: Synthetic validation in

\{0,1\}^{8}

with

n_{\mathrm{syn}}=100

candidate queries per seed,

\tau=1

, and

10{,}000

independent seeds. The mutual information

\mathbb{I}(F;R\mid A_{t},q_{i})

is computed by summing over all

2^{8}=256

binary-valued vectors. We write

P_{j}=\mathbb{P}(F=f_{j}\mid A_{t})

; the first column reports the largest current model posterior probability. The remaining columns report top-1 recall, top-5% recall, Spearman rank correlation, and pairwise order accuracy. For

m>2

, the remaining posterior mass is distributed uniformly over the other

m-1

models.

Figure 3: Query-rank comparison between the selection rule and exact mutual information in the synthetic binary-valued vector validation. Each plot corresponds to a different number of candidate models. The $x$ -axis gives the rank induced by Equation 5 , and the $y$ -axis gives the rank induced by $\mathbb{I}(F;R\mid A_{t},q_{i})$ , with rank $1$ at the top. Points near the diagonal indicate agreement between the two rankings.

Figure 4: Distribution of average model scores for the

23

datasets used in our experiments. Each histogram bins the candidate models by their average oracle score on the corresponding task, using the same task-specific metric as in the main experiments. Scores below zero are included in the first bin.

Dataset	Selected $\tau$	Dataset size	Realization size $n$	Avg. score range	Models	Metric
bAbI	2.0	1,000	500	0.49–0.71	28	Exact match
BBH	3.0	5,761	1,000	0.59–0.73	15	Exact match
CNN/DailyMail (ROUGE-L)	0.5	1,000	500	0.09–0.27	17	ROUGE-L
CNN/DailyMail (BERTScore)	0.1	1,000	500	0.21–0.30	17	BERTScore
Flickr30k	0.1	1,000	500	0.00–0.31	8	ROUGE-L
GPQA	3.0	1,192	500	0.24–0.39	12	Exact match

Appendix

Table 4: Summary of the datasets and model collections used in the experiments. Dataset size is the number of instances available in the oracle pool, and realization size

n

is the number of instances sampled for each experimental realization. Score ranges report the minimum and maximum average model scores under the dataset’s evaluation metric.

Figure 5: Sensitivity of Select-LLM to the temperature parameter

\tau

across the

23

datasets.

Explore similar work

Large Language Model Selection with Limited Annotations
May 24, 2026Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch +2Large Language Models(Llms Evaluation Metrics
TU Delft · ETH Zurich
Which LLM to pick? Online Active Model Selection for Large Language Models
Oct 1, 2026Alessandro Turrin, Patrik Okanovic, Torsten Hoefler +1Model Selection Streaming
TU Delft · ETH Zurich
Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorization
May 11, 2026Elad Tolochinsky, Yaniv Tenzer, Yaniv RomanoLarge Language Model Evaluation Model Selection
Department of Computer Science, Technion – Israel Institute of Technology · Department of Electrical and Computer Engineering, Technion – Israel Institute of Technology

May 24, 2026, cs.CLJ/K move · Enter open · S save