Robust AI Evaluation through Maximal Lotteries

Hadi Khalaf; Serena L. Wang; Daniel Halpern; Itai Shapira; Flavio du Pin Calmon; Ariel D. Procaccia

cs.LGFeb 24, 2026

Robust AI Evaluation through Maximal Lotteries

Authors: Hadi Khalaf, Serena L. Wang, Daniel Halpern, Itai Shapira, Flavio du Pin Calmon, Ariel D. Procaccia

Organizations: Harvard University

Abstract

The standard way to evaluate language models on subjective tasks is through pairwise comparisons: an annotator chooses the "better" of two responses to a prompt. Leaderboards aggregate these comparisons into a single Bradley-Terry (BT) ranking, forcing heterogeneous preferences into a total order and violating basic social-choice desiderata. In contrast, social choice theory provides an alternative approach called maximal lotteries, which aggregates pairwise preferences without imposing any assumptions on their structure. However, we show that maximal lotteries are highly sensitive to preference heterogeneity and can favor models that severely underperform on specific tasks or user subpopulations. We introduce robust lotteries that optimize worst-case performance under plausible shifts in the preference data. On large-scale preference datasets, robust lotteries provide more reliable win rate guarantees across the annotator distribution and recover a stable set of top-performing models. By moving from rankings to pluralistic sets of winners, robust lotteries offer a principled step toward an ecosystem of complementary AI systems that serve the full spectrum of human preferences.

Figures & tables

Figure 1: Robust lotteries improve worst-case win rate across user subpopulations. We consider a fixed set of models (red points) and partition the LMArena votes according to the language of the prompt. We then evaluate any lottery (a probability distribution over models) by its worst-case performance on LMArena (

y

-axis): the minimum win rate the lottery achieves across the four user subpopulations. Because the guarantee is

\min_{q}\Pr(p\succ q)

in a symmetric zero-sum game, it is at most 50%. The

x

-axis reports the lottery’s expected inference cost per 1M input tokens. The orange curve traces the cost–performance frontier obtained by solving for the maximal lottery under an expected-cost budget. The blue curve traces the corresponding frontier for robust lotteries , which instead optimize the worst-case guarantee using a robust linear program. Across budgets, robust lotteries achieve substantially higher worst-case performance than the standard maximal lottery.

Panel of Figure 2 — Figure 2: Maximal lotteries are sensitive to population shifts. Each simplex point is a lottery $p=(p_{1},p_{2},p_{3})$ over models $\{\text{1, 2, 3}\}$ with vertices being deterministic choices. Edge arrows indicate the direction of preference. In each panel, the dashed lines are zero-margin boundaries $f_{j}(p)=p^{\top}Me_{j}=0$ , i.e., mixtures that tie pure opponent $j$ under the stratum margin matrix $M$ . Left: English stratum has a Condorcet winner, so the maximal lottery collapses to the winner ( $p^{\star}_{\mathrm{EN}}$ at a vertex). Right: Spanish stratum exhibits a 3-cycle, so $p^{\star}_{\mathrm{ES}}$ lies in the interior, balancing the worst-case opponent. Middle: two nearby population mixtures (weights $\alpha$ and $\alpha^{\prime}$ over strata) induce two nearby matrices and thus two sets of zero-margin lines; their intersections give $p^{\star}_{\mathrm{mix}}$ and $p^{\star}_{\mathrm{mix}}{}^{\prime}$ , illustrating that even with the same majority directions, small shifts in mixture weights can move the maximal lottery. This sensitivity motivates robust maximal lotteries, which optimize a worst-case guarantee over a set of plausible population mixtures.

Panel of Figure 3 — Figure 3: Robust lotteries improve win rate guarantees across subpopulations. We compute robust lotteries for varying radius values $\rho$ and evaluate each lottery on held-out votes (20% split). We show bootstrap means of win rates achieved on the overall population and each subgroup with standard errors (200 samples). As $\rho$ increases, robust lotteries improve the win rate guarantees for the lowest-performing groups with a modest decrease for the highest-performing groups, illustrating a robustness–accuracy trade-off. Left: LMArena, with groups defined by prompt language. Right: HUMAINE, with groups defined by annotator’s ethnic group.

Figure 4: Robust lotteries diversify the lottery to handle preference tradeoffs among subpopulations. We present the estimated probability assigned to each model with its standard error from LMArena (200 samples). At

\rho=0

, the lottery concentrates on the top aggregate performer (Gemini 2.5 Pro). As

\rho

increases, probability mass shifts toward additional strong models (e.g., o3 and Llama 4 Maverick), reflecting the need to hedge to improve win rate guarantees as seen in Figure 3 .

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Table 5

Panel of Figure 6 — Figure B.1: Weekly (left) and cumulative (right) mixture distributions of vote categories in LMArena. The weekly plot shows the fraction of votes collected per category each week, while the cumulative plot shows the category composition of all votes collected up to a given week.

Figure B.2: Pairwise preferences exhibit substantial heterogeneity across groups in LMArena. We compute the probability that two distinct groups (sampled proportional to their comparison counts) yield opposite preferences for

(i,j)

, i.e., their margins

M_{ij}

have opposite signs.

Pair $(i,j)$	Rev. prob.	Prefer $i$	Prefer $j$
Gemini 2.0 Flash, Opus 4	0.467	ZH (5), PL (17), RU (10)	EN (54)
Llama 4 Maverick, Gemma 3	0.453	ZH (8), PL (13), RU (13)	EN (64)
Mistral Medium, Llama 4 Maverick	0.443	EN (127)	ZH (19), PL (30), RU (14)
o4 Mini, Gemini 2.5 Pro	0.419	EN (115)	ZH (10), PL (28), RU (11)
Qwen3 30B, Sonnet 3.7	0.415	ZH (10), PL (20)	EN (64), RU (8)

Appendix

Table B.3: Highest reversal-rate pairs in LMArena. For each model pair

(i,j)

, we report the reversal probability and the languages which prefer

i

over

j

, along with the number of comparisons available in each language (ZH = Chinese, PL = Polish, RU = Russian, EN = English).

Figure B.3: Robust lotteries reduce overfitting in LMArena. We compute robust lotteries for varying radius values

\rho

and evaluate each lottery on the overall population’s training votes and held-out votes from LMArena (80%-20% split). We report the generalization gap (train win rate minus test win rate) as a function of

\rho

(200 samples). As

\rho

increases, the gap shrinks from roughly

6\%

\rho=0

– which corresponds to a standard maximal lottery – to about

2\%

\rho=1

, indicating that robustness regularizes the learned mixture and yields more stable performance.

Claude 3.7 Sonnet	Gemini 2.0 Flash	o1	Grok 3
Claude Opus 4	Gemini 2.5 Flash	o3 Mini	Grok 4
Claude Sonnet 4	Gemini 2.5 Pro	o4 Mini	DeepSeek R1
Command A	Command R7B	Mistral Nemo

Appendix

Table B.4: Models evaluated in HUMAINE dataset.

Table B.5: Vote counts in HUMAINE by group type. Left. Group votes by ethnic group. Right. Group votes by political affiliation.

Panel of Figure 12 — Figure B.4: Pairwise preferences exhibit substantial heterogeneity across groups in the HUMAINE dataset. We compute the probability that two distinct groups (sampled proportional to their comparison counts) yield opposite preferences for $(i,j)$ , i.e., their margins $M_{ij}$ have opposite signs. Left. Groups defined by annotator’s ethnic group. Right. Groups defined by annotator’s political affiliation.

Panel of Figure 13 — Figure B.5: Win rate achieved by robust lottery on training margin matrices in the HUMAINE dataset. We compute robust lotteries for varying radius values $\rho$ and evaluate each lottery on the training votes (80% split). We show bootstrap means of win rates achieved on the overall population and each subgroup with standard errors (200 samples). Left. Groups defined by annotator’s ethnic group. Right. Groups defined by annotator’s political affiliation.

Panel of Figure 14 — Figure B.6: Win rate achieved by robust lottery on held-out votes in the HUMAINE dataset. We compute robust lotteries for varying radius values $\rho$ and evaluate each lottery on held-out votes (20% split). We show bootstrap means of win rates achieved on the overall population and each subgroup with standard errors (200 samples). Left. Groups defined by annotator’s ethnic group. Right. Groups defined by annotator’s political affiliation.

Panel of Figure 15 — Figure B.7: Robust lotteries reduce overfitting in the HUMAINE dataset. We compute robust lotteries for varying radius values $\rho$ and evaluate each lottery on the overall population’s training votes and held-out votes from HUMAINE (80%-20% split). We report the generalization gap (train win rate minus test win rate) as a function of $\rho$ (200 samples).

Figure B.8: Robust lotteries diversify the lottery to handle preference tradeoffs among ethnic groups in the HUMAINE dataset. We present the estimated probability assigned to each model with its standard error from HUMAINE when annotators are grouped based on ethnic group (200 samples).

Figure B.9: Robust lotteries diversify the lottery to handle preference tradeoffs among political groups in the HUMAINE dataset. We present the estimated probability assigned to each model with its standard error from HUMAINE when annotators are grouped based on political affiliation (200 samples).

Gemini 2.0 Flash	Gemini 2.5 Flash	Gemini 2.5 Pro	Gemini 2.5 Pro (no search)	GPT-4o Mini
GPT-4o	GPT-4o High	GPT-4o High Loc	Sonar	Sonar Pro
Sonar Pro High	Sonar Reasoning	Sonar Reasoning Pro High	–	–

Appendix

Table B.6: Models evaluated in Search Arena dataset.

Table B.7: Vote counts in Search Arena by group type. Left. Group votes by language. Right. Group votes by user intent.

Panel of Figure 20 — Figure B.10: Win rate achieved by robust lottery on training margin matrices in the Search Arena dataset. We compute robust lotteries for varying radius values $\rho$ and evaluate each lottery on the training votes (80% split). We show bootstrap means of win rates achieved on the overall population and each subgroup with standard errors (200 samples). Left. Groups defined by prompt language. Right. Groups defined by annotator’s primary intent.

Panel of Figure 21 — Figure B.11: Win rate achieved by robust lottery on held-out votes in the Search Arena dataset. We compute robust lotteries for varying radius values $\rho$ and evaluate each lottery on held-out votes (20% split). We show bootstrap means of win rates achieved on the overall population and each subgroup with standard errors (200 samples). Left. Groups defined by prompt language. Right. Groups defined by annotator’s primary intent.

Panel of Figure 22 — Figure B.12: Robust lotteries reduce overfitting in the Search Arena dataset. We compute robust lotteries for varying radius values $\rho$ and evaluate each lottery on the overall population’s training votes and held-out votes from Search Arena (80%-20% split). We report the generalization gap (train win rate minus test win rate) as a function of $\rho$ (200 samples). Left. Groups defined by prompt language. Right. Groups defined by annotator’s primary intent.

Figure B.13: Robust lotteries diversify the lottery to handle preference tradeoffs among languages in the Search Arena dataset. We present the estimated probability assigned to each model with its standard error from Search Arena when annotators are grouped based on language (200 samples).

Figure B.14: Robust lotteries diversify the lottery to handle preference tradeoffs among tasks in the Search Arena dataset. We present the estimated probability assigned to each model with its standard error from Search Arena when annotators are grouped based on primary intent in the task (200 samples).

Claude	Gemini	Mistral	Cerebras	Gemma
GPT Neo	GPT 3.5	Llama 3	Olmo	OPT
Phi 3	Pythia	Qwen	–	–

Appendix

Table B.8: Models evaluated in Open LLM dataset.

Group	Votes
MMLU	226,540
ARC	100,061
MedMCQA	47,767
WinoGrande	41,227
CommonsenseQA	19,303

Appendix

Table B.9: Vote counts in Open LLM by group type

Panel of Figure 27 — Figure B.15: We compute robust lotteries for varying radius values $\rho$ and evaluate each lottery on the overall population’s training votes and held-out votes from Open LLM (80%-20% split). Left: Win rate achieved on training margin matrices. Right: Win rate achieved on held-out margin matrices.

Figure B.16: Robust lotteries reduce overfitting in the Open LLM dataset. We compute robust lotteries for varying radius values

\rho

and evaluate each lottery on the overall population’s training votes and held-out votes from Open LLM when annotators are grouped based on benchmark (80%-20% split). We report the generalization gap (train win rate minus test win rate) as a function of

\rho

(200 samples).

Figure B.17: Robust lotteries diversify the lottery to handle preference tradeoffs among languages in the Open LLM dataset. We present the estimated probability assigned to each model with its standard error from Open LLM when annotators are grouped based on benchmark (200 samples).

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings
Aug 16, 2025Jenny Y. Huang, Yunyi Shen, Dennis Wei +1Pairwise Preference Learning LLM Evaluation
Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology · MIT-IBM Watson AI Lab · IBM Research
Why Global LLM Leaderboards Are Misleading: Small Portfolios for Heterogeneous Supervised ML
May 7, 2026Jai Moondra, Ayela Chughtai, Bhargavi Lanka +1Multilingual Language Model Evaluation LLM Evaluation
Carnegie Mellon University · MIT Sloan School of Management
Multiple latent orderings better predict language model preferences
Aug 26, 2026Aviral Chawla, William H. W. Thompson, Jean-Gabriel YoungPreference Modeling Pairwise Preference Learning
Vermont Complex Systems Institute University of Vermont Burlington, Vermont 05403 · Vermont Complex Systems Institute Department of Statistics University of Vermont Burlington, Vermont 05403

Aug 16, 2025J/K move · Enter open · S save

Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings

Jenny Y. Huang, Yunyi Shen, Dennis Wei, Tamara Broderick

Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology · MIT-IBM Watson AI Lab · IBM Research

Pairwise Preference Learning LLM Evaluation Pairwise Comparison Pairwise Preference Evaluation Learning to Rank Pairwise Ranking+1

Open paper AskPDF arXiv

We propose a method for evaluating the robustness of widely used LLM ranking systems -- variants of a Bradley--Terry model -- to dropping a worst-case very small fraction of preference data. Our approach is computationally fast and easy to adopt. When we apply our method to matchups from popular LLM ranking platforms, including Chatbot Arena and derivatives, we find that the rankings of top-performing models can be remarkably sensitive to the removal of a small fraction of preferences; for instance, dropping just 0.003% of human preferences can change the top-ranked model on Chatbot Arena. Our robustness check identifies the specific preferences most responsible for such ranking flips, allowing for inspection of these influential preferences. We observe that the rankings derived from MT-bench preferences are notably more robust than those from Chatbot Arena, likely due to MT-bench's use of expert annotators and carefully constructed prompts. Finally, we find that neither rankings based on crowdsourced human evaluations nor those based on LLM-as-a-judge preferences are systematically more sensitive than the other.

Abstract

LLM Evaluation

Explore similar work

Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings

Why Global LLM Leaderboards Are Misleading: Small Portfolios for Heterogeneous Supervised ML

Multiple latent orderings better predict language model preferences