Leaderboard Incentives: Model Rankings under Strategic Post-Training

Yatong Chen; Guanhua Zhang; Moritz Hardt

cs.GTMar 9, 2026

Leaderboard Incentives: Model Rankings under Strategic Post-Training

Authors: Yatong Chen, Guanhua Zhang, Moritz Hardt

Organizations: Max Planck Institute for Intelligent Systems, Tübingen and Tübingen AI Center

Abstract

Influential benchmarks incentivize competing model developers to strategically allocate post-training resources toward improvements on the leaderboard, a phenomenon dubbed benchmaxxing or training on the test task. In this work, we initiate a principled study of the incentive structure that benchmarks induce. We model benchmarking as a Stackelberg game between a benchmark designer who chooses an evaluation protocol and multiple model developers who compete simultaneously in a subgame given by the designer's choice. Each competitor has a model of unknown latent quality and can inflate its observed score by allocating resources to benchmark-specific improvements. First, we prove that current benchmarks induce games for which no Nash equilibrium between model developers exists. This result suggests one explanation for why current practice leads to misaligned incentives, prompting model developers to strategize in opaque ways. However, we prove that under mild conditions, a recently proposed evaluation protocol, called tune-before-test, induces a benchmark with a unique Nash equilibrium that ranks models by latent quality. This positive result demonstrates that benchmarks need not set bad incentives, even if current evaluations do.

Figures & tables

Figure 1: Left: Continued post-training trajectories of Qwen2.5 models of different sizes on Winogrande . Here, we use model size as a proxy for the model’s latent capability

\theta

. The

x

-axis denotes the amount of post-training steps (each step corresponds to 8 data points), reflecting post-training effort

e

. The

y

-axis denotes accuracy on the validation set, i.e.,

v(\theta,e)

. For each model, we fit a curve following Equation 3 . The empirical results align with the assumptions of monotonicity in capability , diminishing returns and saturation in effort , and non-decreasing effort gaps in Assumption 4.2 . See Appendix E for additional details and results for the other eight benchmarks. Right: For each tune-before-test level

\Delta^{\textit{tbt}}

(the amount of benchmark-specific finetuning steps,

x

-axis), we calculate the minimal additional steps required (

y

-axis) to change the ranking for at least one model, i.e.,

\min_{r\in\{2,\ldots,n\}}{e}^{\text{req}}_{r}(\Delta^{\textit{tbt}})

, based on the fitted curves on the left. With

\Delta^{\textit{tbt}}=3,000

, at least 384,668 training steps are needed to change the ranking of one model.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
$\theta\in\mathbbm{R}_{\geq 0}$	one-dimensional latent model capability
$e\in\mathbbm{R}_{\geq 0}$	benchmark-specific training effort
$v(\theta,e)\in[0,1]$	post-effort score (benchmark performance)
$c(e)$	cost of exerting effort $e$
$(R_{j})_{j=1}^{n}$	rank-based reward scheme: the model ranked $j$ receives reward $R_{j}$
$\Delta^{\textit{tbt}}\in\mathbbm{R}_{\geq 0}$	tune-before-test (TbT) baseline chosen by the leaderboard designer

Appendix

Table 1: Primary notation.

Figure 2: Illustration of the notations and functions used in the proof of Proposition 4.3 . The post-effort score function

v(\theta,e)

satisfies the conditions in Assumption 4.2 . Here,

e_{i}^{*}

and

e_{j}^{*}

denote the equilibrium efforts of model developers

i

and

j

, respectively. The counterfactual efforts

\tilde{e}_{i}

and

\tilde{e}_{j}

are defined as the efforts each model developer would need to match the other model developer’s post-effort score given a TbT effort level

\Delta^{\textit{tbt}}

, i.e.,

v(\theta_{i},\tilde{e}_{i}+\Delta^{\textit{tbt}})=v(\theta_{j},e_{j}^{*}+\Delta^{\textit{tbt}}),v(\theta_{j},\tilde{e}_{j}+\Delta^{\textit{tbt}})=v(\theta_{i},e_{i}^{*}+\Delta^{\textit{tbt}}).

Figure 3: Continued post-training trajectories of Qwen models of different sizes on nine benchmarks. Here, we use model size as a proxy for the model’s latent capability

\theta

. The

x

-axis denotes the number of post-training steps, reflecting post-training effort

e

. The

y

-axis denotes accuracy on the validation set, i.e.,

v(\theta,e)

. For each model, we fit a curve following Equation equation 3 .

Benchmark Design

Benchmark design creates tasks, datasets, metrics, and protocols that make AI capabilities measurable and comparisons credible.

See all 391 papers

On Benchmark Hacking in ML Contests: Modeling, Insights and Design
Apr 24, 2026Xiaoyun Qiu, Yang Yu, Haifeng XuBenchmark Design
Department of Economics, Dartmouth College. · Sloan & CSAIL, Massachusetts Institute of Technology. · Department of Computer Science, University of Chicago.
You Don't Need to Run Every Eval
Jun 22, 2026Yuchen Zeng, Dimitris PapailiopoulosLLM Evaluation Robust Matrix Completion
Microsoft Research, AI Frontiers
Dynamically Allocating Evaluation Effort for Model Ranking
Aug 4, 2026Vilém Zouhar, Julia Kreutzer, Alon Lavie +4Model Selection Multi-Armed Bandits
ETH Zurich · Cohere Labs · Carnegie Mellon University +3

Apr 24, 2026J/K move · Enter open · S save

On Benchmark Hacking in ML Contests: Modeling, Insights and Design

Xiaoyun Qiu, Yang Yu, Haifeng Xu

Department of Economics, Dartmouth College. · Sloan & CSAIL, Massachusetts Institute of Technology. · Department of Computer Science, University of Chicago.

Benchmark Design

Open paper AskPDF arXiv

Benchmark hacking refers to tuning a machine learning model to score highly on certain evaluation criteria without improving true generalization or faithfully solving the intended problem. We study this phenomenon in a generic machine learning contest, where each contestant chooses two types of effort: creative effort that improves model capability as desired by the contest host, and mechanistic effort that only improves the model's fitness to the particular task in contest without contributing to true generalization. We establish the existence of a symmetric monotone pure strategy equilibrium in this competition game. It also provides a natural definition of benchmark hacking in this strategic context by comparing a player's equilibrium effort allocation to that of a single-agent baseline scenario. Under our definition, contestants with types below certain threshold (low types) always engage in benchmark hacking, whereas those above the threshold do not. Furthermore, we show that more skewed reward structures (favoring top-ranked contestants) can elicit more desirable contest outcomes. We also provide empirical evidence to support our theoretical predictions.

Abstract

Benchmark Design

Explore similar work

On Benchmark Hacking in ML Contests: Modeling, Insights and Design

You Don't Need to Run Every Eval

Dynamically Allocating Evaluation Effort for Model Ranking