Beyond Scalar Leaderboards: Adaptive Sampling for Pareto Frontier Identification in Multi-Objective LLM Arenas

Liang Twist Shan; Jiahao Ai; Zhimei Ren

stat.MLOct 20, 2026

Beyond Scalar Leaderboards: Adaptive Sampling for Pareto Frontier Identification in Multi-Objective LLM Arenas

Authors: Liang Twist Shan, Jiahao Ai, Zhimei Ren

Organizations: Department of Statistics, University of Wisconsin-Madison · Department of Statistics and Data Science, University of Pennsylvania

Abstract

The evaluation of large language models (LLMs) is inherently multi-objective, as different models may offer different tradeoffs across performance dimensions such as math, code, and writing. Because such evaluations are often based on costly and time-consuming human feedback, it is important to draw reliable conclusions using as few comparisons as possible. In this paper, we present Vector-Borda Empirical-Gap Elimination (VB-EGE), a fixed-confidence adaptive sampling algorithm for efficiently identifying models on the Pareto frontier in arena-style evaluation, where each comparison evaluates a pair of models on a prompt with respect to a selected performance dimension. VB-EGE progressively eliminates models from an active set once sufficient statistical evidence has been accumulated to certify their Pareto status. Under a multi-objective Bradley-Terry model, we show that VB-EGE returns the correct Pareto frontier with high probability and derive an instance-dependent upper bound on its sample complexity. We further establish a worst-case lower bound showing that VB-EGE is minimax-optimal up to logarithmic factors. Simulations and experiments on three real-world Arena datasets (Arena, SearchArena, and VisionArena) demonstrate VB-EGE's empirical effectiveness.

Figures & tables

Figure 1: Three-dimensional comparison along the Code, Math, and Instruction-Following objectives from the Arena-Human-Preference-140K dataset ( Chiang et al., 2024 ) . Diamonds denote the strict Pareto frontier.

Study	Parameter grid
Scaling in $K$	$K\in\{16,32,64,128\}$ at $d=4,\Delta=1$
Scaling in $d$	$d\in\{2,4,10\}$ at $K=64,\Delta=1$
Scaling in $\Delta$	$\Delta\in\{0.5,0.75,1,1.25,1.5\}$ at $K=64,d=10$
Scaling in $\delta$	$\delta\in\{0.2,0.1,0.05,0.02,0.01,0.005,0.002,0.001\}$ at $K=64,d=10,\Delta=1$

Table 1: Parameters used in the scaling experiments.

Figure 2: Mean stopping time (mean

\pm

one standard error) on logarithmic vertical axes. (a) Scaling in

K

d

\Delta

, and

\log(1/\delta)

. (b) Eight heterogeneous synthetic Pareto benchmark settings; S and M denote small and medium. (c) Three realistic Arena datasets, with stopping times measured in comparisons.

Dataset	Objectives $d$	Candidates	Retained $K$	Rows	$\lvert\mathcal{P}_{\rm emp}\rvert$
Arena-140K	5	52	35	$73{,}135$	3
Search-Arena-24K	9	13	8	$4{,}305$	5
VisionArena-Battle	8	17	12	$23{,}458$	3

Table 2: Main realistic Arena data settings. Rows are retained pair–objective records; the VisionArena-Battle records arise from

12{,}937

unique multi-label comparisons.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Generator	$K$	$d$	$\|\mathcal{P}\|$	Parameters
Convex-2D	Convex frontier	60	2	15	$s=15$ , margins $[0.03,0.18]$
Convex-3D	Convex frontier	60	3	15	$s=15$ , simplex $\alpha=1$ , margins $[0.03,0.18]$
Witness-4	Unique witness	40	4	8	$s=8,q=4$ , margins $[0.04,0.16]$ , $\alpha=0.7$
Witness-10	Unique witness	80	10	16	$s=16,q=4$ , margins $[0.035,0.14]$ , $\alpha=0.7$
Arena-4 small	Synthetic Arena	32	4	8	$s=8$ , margins $[0.08,0.25]$ , $\alpha=0.7$
Arena-4 medium	Synthetic Arena	64	4	12	$s=12$ , margins $[0.08,0.25]$ , $\alpha=0.7$

Appendix

Table 3: Parameters used in the synthetic benchmark experiments.

Sweep	Value	VB mean $\tau$ (SE)	Focal/VB	MLE/VB	Plug-in/VB
$K$	16	$3.63\times 10^{5}$ (678)	1.02	0.51	9.03
	32	$7.85\times 10^{5}$ (411)	1.00	1.08	19.1
	64	$1.66\times 10^{6}$ (457)	1.00	2.29	40.1
	128	$3.51\times 10^{6}$ (301)	1.00	4.80	83.2
$d$	2	$7.65\times 10^{5}$ ( $1.63\times 10^{3}$ )	1.03	2.38	41.7
	4	$1.66\times 10^{6}$ (405)	1.00	2.29	40.1

Appendix

Table 4: Fixed-confidence scaling results. Parentheses contain one standard error for VB-EGE ; remaining columns report each baseline’s mean comparison count divided by that of VB-EGE.

Setting	VB mean $\tau$ (SE)	Focal/VB (SE)	MLE/VB (SE)	Plug-in/VB (SE)
Convex-2D	$1.21\times 10^{8}$ ( $9.57\times 10^{5}$ )	3.55 (0.10)	6.90 (0.21)	118 (3.6)
Convex-3D	$4.46\times 10^{9}$ ( $3.86\times 10^{9}$ )	28.1 (1.6)	62.5 (3.7)	1015 (60)
Witness-4	$2.28\times 10^{8}$ ( $7.27\times 10^{5}$ )	1.25 (0.0040)	1.79 (0.0057)	29.7 (0.094)
Witness-10	$1.60\times 10^{9}$ ( $3.26\times 10^{6}$ )	1.81 (0.060)	2.97 (0.0061)	49.0 (0.10)
Arena-4 small	$3.83\times 10^{10}$ ( $3.30\times 10^{10}$ )	15.7 (0.35)	18.3 (0.80)	302 (14)
Arena-4 medium	$2.20\times 10^{10}$ ( $9.61\times 10^{9}$ )	31.6 (0.25)	69.0 (5.7)	1224 (9.6)

Appendix

Table 5: Mean comparison counts for VB-EGE and ratios of mean comparison counts (baseline/VB-EGE). Parentheses contain one standard error. Ratio standard errors use the paired delta method, with instance-level averages as the independent units when multiple trials share the same latent instance.

Table 8

Figure 3: Arena-10 medium mean stopping time (mean

\pm

one standard error) versus

\log(1/\delta)

Panel of Figure 4 — Figure 4: Mean stopping-time with respect to difference choices of $(C_{\mathrm{samp}},C_{\mathrm{thr}})$ . Each cell reports stopping time (mean $\pm$ one standard error). The black rectangle corresponds to the default choice.

$d$	$\|\mathcal{P}\|$	VB mean $\tau$ (SE)	Focal/VB	MLE/VB	Plug-in/VB
4	4	$9.44\times 10^{7}$ ( $7.02\times 10^{5}$ )	2.60	6.11	99.1
4	8	$1.16\times 10^{8}$ ( $4.91\times 10^{6}$ )	10.0	22.9	387
4	16	$2.52\times 10^{8}$ ( $5.70\times 10^{7}$ )	21.8	39.7	640
4	32	$4.42\times 10^{8}$ ( $2.22\times 10^{7}$ )	11.8	28.2	462
10	4	$5.21\times 10^{8}$ ( $2.79\times 10^{6}$ )	2.41	5.66	93.5
10	8	$5.55\times 10^{8}$ ( $3.11\times 10^{6}$ )	2.27	5.32	87.9

Appendix

Table 8: Sensitivity to Pareto-set size. Parentheses contain one standard error for VB-EGE ; other entries report each baseline’s mean comparison count divided by that of VB-EGE.

Panel of Figure 5 — Figure 5: Mean stopping time (mean $\pm$ one standard error) as the true Pareto set grows.

Figure 6: Total VB-EGE allocation to each model versus its Borda classification gap on representative Arena-4 medium and Two-group-10 instances. Smaller-gap models receive more samples.

Dataset	Source	Decisive	Assigned comparisons	Objective records	Candidates	$K$	Rows
Arena-140K	$135{,}609$	$98{,}332$	$98{,}332$	$98{,}332$	52	35	$73{,}135$
Search-Arena-24K	$24{,}069$	$8{,}309$	$8{,}309$	$8{,}309$	13	8	$4{,}305$
VisionArena-Battle	$29{,}849$	$18{,}728$	$15{,}966$	$28{,}925$	17	12	$23{,}458$

Appendix

Table 9: Preprocessing counts for the three Arena datasets. “Assigned comparisons” counts source comparisons assigned to at least one evaluation objective; “Objective records” counts the resulting pair–objective records. As in the main text, “Rows” counts pair–objective records involving only retained models. The

23{,}458

retained VisionArena-Battle records correspond to

12{,}937

unique source comparisons.

Dataset	Pareto members
Arena-35	chatgpt-4o-latest-20250326 ; gemini-2.5-pro ; grok-3-preview-02-24
SearchArena-8	gemini-2.5-pro-exp-03-25-grounding ; sonar ; sonar-pro-high ; sonar-reasoning ; sonar-reasoning-pro-high
VisionArena-12	claude-3-5-sonnet-20240620 ; gemini-1.5-pro-exp-0801 ; gpt-4o-2024-05-13

Appendix

Table 10: Full-dimensional strict empirical Pareto sets on the three retained model sets.

Method	Arena-35	SearchArena-8	VisionArena-12
VB-EGE	$86.82\;(1.00\times)$	$124.34\;(1.00\times)$	$8.80\;(1.00\times)$
Focal Borda	$1{,}435.70\;(16.54\times)$	$492.74\;(3.96\times)$	$40.39\;(4.59\times)$
BT-MLE	$1{,}586.26\;(18.27\times)$	$149.06\;(1.20\times)$	$14.77\;(1.68\times)$
Borda plug-in	$26{,}164.10\;(301.36\times)$	$1{,}890.10\;(15.20\times)$	$251.62\;(28.60\times)$

Appendix

Table 11: Comparison of the four methods on the three Arena datasets. Entries are mean stopping times in millions of sampled comparisons (including repeated records), with each method’s mean divided by the VB-EGE mean in parentheses. Every method stops and recovers the exact frontier in all 100 trials on each dataset.

Figure 7: Arena-35: empirical Borda profiles on all five objectives.

Figure 8: SearchArena-8: empirical Borda profiles on all nine objectives.

Figure 9: VisionArena-12: empirical Borda profiles on all eight objectives.

Panel of Figure 10 — Figure 10: Arena-35 illustrative projections.

Panel of Figure 11 — Figure 11: SearchArena-8 illustrative projections.

Panel of Figure 12 — Figure 12: VisionArena-12 illustrative projections.

Dataset	Decisive	tie	both_bad	Non-decisive fraction
Arena-35	$73{,}135$	$16{,}093$	$11{,}594$	$27.5\%$
SearchArena-8	$4{,}305$	$1{,}148$	$676$	$29.8\%$
VisionArena-12	$23{,}458$	$6{,}346$	$8{,}031$	$38.0\%$

Appendix

Table 12: Outcome counts on the retained models, before non-decisive labels are merged. Each record is one comparison assigned to one objective, and the last column is the fraction of records labeled tie or both_bad .

Figure 13: Arena-35: Borda profiles of the models in

\mathcal{P}_{D}\cup\mathcal{P}_{H}

under each target; minimax-m1 enters the strict Pareto frontier under

H

. Left: decisive-only target

D

; right: direct half-credit target

H

Figure 14: SearchArena-8: Borda profiles of the models in

\mathcal{P}_{D}\cup\mathcal{P}_{H}

under each target; frontier membership is unchanged. Left: decisive-only target

D

; right: direct half-credit target

H

Figure 15: VisionArena-12: Borda profiles of the models in

\mathcal{P}_{D}\cup\mathcal{P}_{H}

under each target; frontier membership is unchanged. Left: decisive-only target

D

; right: direct half-credit target

H

Dataset	Candidate $k_{\rm tie}$	Selected $k_{\rm tie}$ by objective
Arena-35	$\{0,1,2,3,5\}$	$(2,1,0,0,0)$
SearchArena-8	$\{0,1,2\}$	$(2,0,2,0,0,1,2,1,0)$
VisionArena-12	$\{0,1,2\}$	$(2,2,1,2,1,0,1,2)$

Appendix

Table 13: Selection of the tie-factor size

k_{\rm tie}

on the fixed split described above. Selected sizes are listed in the objective order of Appendix D.1 .

Figure 16: Arena-35: held-out NLL for each candidate tie-factor size

k_{\rm tie}

; open circles mark the selected size for each objective. The candidates

0,1,2,3,5

are plotted at their numerical positions.

Figure 17: SearchArena-8: held-out NLL for each candidate tie-factor size

k_{\rm tie}

; open circles mark the selected size for each objective.

Figure 18: VisionArena-12: held-out NLL for each candidate tie-factor size

k_{\rm tie}

; open circles mark the selected size for each objective.

Figure 19: Arena-35: Borda profiles of the models in

\mathcal{P}_{D}\cup\mathcal{P}_{F}

under each target; minimax-m1 enters and chatgpt-4o-latest-20250326 leaves the strict Pareto frontier under

F

. Left: decisive-only target

D

; right: fitted Davidson target

F

Figure 20: SearchArena-8: Borda profiles of the models in

\mathcal{P}_{D}\cup\mathcal{P}_{F}

under each target; frontier membership is unchanged. Left: decisive-only target

D

; right: fitted Davidson target

F

Figure 21: VisionArena-12: Borda profiles of the models in

\mathcal{P}_{D}\cup\mathcal{P}_{F}

under each target; frontier membership is unchanged. Left: decisive-only target

D

; right: fitted Davidson target

F

Dataset	$D/H/F$	$\operatorname{Jac}(D,H)$	$\operatorname{Jac}(D,F)$	Mean $\lvert b^{H}-b^{D}\rvert$	Mean $\lvert b^{F}-b^{D}\rvert$	$\rho_{\min}(D,H)$	$\rho_{\min}(D,F)$
Arena-35	$3/4/3$	$0.750$	$0.500$	$0.0244$	$0.0267$	$0.993$	$0.973$
SearchArena-8	$5/5/5$	$1.000$	$1.000$	$0.0333$	$0.0352$	$0.929$	$0.810$
VisionArena-12	$3/3/3$	$1.000$	$1.000$	$0.0529$	$0.0524$	$0.958$	$0.951$

Appendix

Table 14: Target sensitivity on the fixed sets of retained models used in the main experiments.

D/H/F

gives the strict-frontier sizes under the decisive-only, direct half-credit, and fitted Davidson targets.

\operatorname{Jac}(D,H)

and

\operatorname{Jac}(D,F)

are Pareto-set Jaccard overlaps. Mean coordinate shifts are averaged over all model–objective coordinates, and

\rho_{\min}

is the minimum objective-wise Spearman correlation with the decisive-only target.

Dataset	$\Delta_{\min}^{D}$	$\Delta_{\min}^{H}$	$\Delta_{\min}^{F}$	$\mathcal{H}_{D}/10^{3}$	$\mathcal{H}_{H}/10^{3}$	$\mathcal{H}_{F}/10^{3}$
Arena-35	0.00441	0.00520	0.00391	111.72	101.03	147.38
SearchArena-8	0.00460	0.00539	0.00506	94.97	69.93	79.71
VisionArena-12	0.02171	0.00649	0.00225	5.38	49.25	397.08

Appendix

Table 15: Gap summaries and stopping times under

D

H

, and

F

. Panel (a) gives the minimum gaps and

\mathcal{H}_{X}=\sum_{i}\Delta_{i}(b^{X})^{-2}

, with the latter in thousands. Panel (b) gives mean stopping times in millions of comparisons, with one standard error in parentheses, and ratios of means. In panel (c), each entry gives a final phase followed by its trial count in parentheses. Every trial recovers its construction’s strict frontier.

Quantity	Notation	Value
Threshold regularization	$\lambda_{\mathrm{reg}}$	$0.1$
Bernoulli variance bound	$\bar{\sigma}^{2}$	$1/4$
Score box bound	$S_{\max}$	$2$
Fitting tolerance; maximum iterations		$10^{-7};\ 1{,}000$
Initial samples per cell	$n_{0}$	$2$
Check-growth factor	$a_{\mathrm{batch}}$	$1.8$

Appendix

Table 16: Constants used by Pareto BT-GLR T&S.

Figure 22: Scaling comparison at

\delta=0.05

. Pareto BT-GLR T&S has a smaller capped mean stopping time than VB-EGE only at

K=16

and a larger one in the other 11 settings. Curves show capped mean stopping times

\pm

one standard error.

Figure 23: Benchmark comparison at

\delta=0.05

. Bars show capped mean stopping times

\pm

one standard error. Labels above the bars give the numbers of capped Pareto BT-GLR T&S trials (10 of 500 on Convex-3D and 9 of 500 on Two-group-10). The two capped means are dominated by these trials under the capping convention above.

Figure 24: Pareto-set size sensitivity at

\delta=0.05

. Pareto BT-GLR T&S has a larger capped mean stopping time than VB-EGE in all eight settings, with a geometric-mean ratio of

20.6

. Curves show capped mean stopping times

\pm

one standard error.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorization
May 11, 2026Elad Tolochinsky, Yaniv Tenzer, Yaniv RomanoModel Selection Multi-Armed Bandits
Department of Computer Science, Technion – Israel Institute of Technology · Department of Electrical and Computer Engineering, Technion – Israel Institute of Technology
Why Global LLM Leaderboards Are Misleading: Small Portfolios for Heterogeneous Supervised ML
May 7, 2026Jai Moondra, Ayela Chughtai, Bhargavi Lanka +1Multilingual Language Model Evaluation LLM Evaluation
Carnegie Mellon University · MIT Sloan School of Management
The Capability Frontier: Benchmarks Miss 82% of Model Performance
Jun 25, 2026Bradley Fowler, Ryan Smith, Daniel Thi Graviet +8LLM Evaluation Language Model Generation Evaluation
1Martian · 3ThoughtWorks · University of Oxford

May 11, 2026J/K move · Enter open · S save

Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorization

Elad Tolochinsky, Yaniv Tenzer, Yaniv Romano

Department of Computer Science, Technion – Israel Institute of Technology · Department of Electrical and Computer Engineering, Technion – Israel Institute of Technology

Model Selection Multi-Armed Bandits LLM Evaluation Robust Matrix Completion Best-Arm Identification Doubly Robust Estimation

Open paper AskPDF arXiv

Selecting the best large language model (LLM) for a fixed benchmark is often expensive, since exhaustive evaluation requires running every model on every example. Multi-armed bandit (MAB) algorithms can reduce the number of LLM calls by sequentially selecting the next model-example pair to evaluate, thereby avoiding wasted evaluations on clearly underperforming models. Further savings can be achieved by predicting model scores from the partially observed model-example score matrix using low-rank factorization. However, such predictions are not ground truth: they can be biased and may therefore lead to incorrect identification of the best model. In this work, we propose a principled framework that combines MAB with cheap predicted scores without compromising statistical validity. Specifically, we derive doubly robust estimators of each model's performance that use the low-rank predictions to reduce variance. This enables the construction of valid finite-sample confidence intervals in our setting, where models are selected adaptively and examples are sampled without replacement. Empirical results on real-world benchmarks show that our approach reduces the number of required evaluations, yielding meaningful savings in compute and cost while accurately identifying the best-performing model.

Abstract

LLM Evaluation

Explore similar work

Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorization

Why Global LLM Leaderboards Are Misleading: Small Portfolios for Heterogeneous Supervised ML

The Capability Frontier: Benchmarks Miss 82% of Model Performance