cs.AINov 10, 2026

Leaderboard Hacking: Preference-Based Model Evaluations are Vulnerable to Manipulation

Authors: Eve Fleisig, Joachim Baumann, Dylan Hadfield-Menell, Dirk Hovy

Organizations: UC Berkeley · Università Bocconi · Stanford University · MIT

Abstract

Arena-style rankings of language models are a widely used evaluation framework in online leaderboards, research papers, and model evaluation in production. These rankings aim to realistically assess model performance using pairwise comparisons of anonymous models on user prompts, aggregating votes via the Bradley-Terry model to produce the final ranking. However, we find that these leaderboards are highly vulnerable to several manipulation strategies that unscrupulous providers could use to boost a model's rank: strategic voting (individual votes that boost a model's rank, including votes on irrelevant models), strategic prompting (choosing prompts that favor a model), and strategic nomination (boosting the rank of a model by inserting irrelevant models). We demonstrate these effects on public leaderboard data, analyze the circumstances that cause these vulnerabilities, and describe approaches to safeguard against each attack. Notably, these issues hold for any preference-based evaluation using the Bradley-Terry model that allows open prompting, voting, or model submission. These manipulations boost the rank of nearly every tested model; each attack can boost a model by up to 15-20 places in a ranking and, combined, they can boost a model by up to 45 places.

Figures & tables

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. A Unified Perturbation Framework for Analyzing Leaderboard Stability and Manipulation

    May 15, 2026Hosna Oyarhoseini, Jimmy Lin, Amir-Hossein KarimiLLM EvaluationLearning to Rank

  2. Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings

    Aug 16, 2025Jenny Y. Huang, Yunyi Shen, Dennis Wei +1Pairwise Preference LearningLLM Evaluation

  3. SOTA or Luck? The Winner's Curse in LLM Leaderboards

    Nov 17, 2026Sikun Xu, Raphael Thomadsen, Dennis J. Zhang +1LLM EvaluationSelection Bias