stat.MLNov 10, 2026

Differentially Private Statistical Inference for Rankings from Pairwise Comparisons

Authors: Xintao Xia, Jia Gu

Organizations: Center for Data Science, Zhejiang University

Abstract

Reliable inference for rankings of large language models based on human pairwise comparisons must preserve the privacy of individual preference records. To meet this need, we develop methods for score and rank inference under the Bradley-Terry model with edge differential privacy. For an objective-perturbed estimator, a new leave-one-out analysis establishes finite-sample bounds on Euclidean and entrywise estimation errors. For sufficiently dense comparison graphs, the error bounds extend to smaller privacy budgets, including regimes in which its privacy term dominates its statistical term. A uniform asymptotic expansion yields asymptotically valid confidence intervals for individual scores and pairwise score differences. To improve finite-sample performance, we account for both regularization bias and privacy-induced randomness by combining a plug-in ridge-bias correction with Gaussian-Laplace calibration. We further establish a bootstrap procedure for simultaneous rank inference, providing asymptotically valid confidence sets for individual ranks and tests of top-KK membership. When the comparison probability is public and known, all proposed inferential procedures require no additional privacy budget. We evaluate finite-sample performance through simulations and apply the proposed methods to rank large language models for creative writing.

Figures & tables

Explore similar work

CardsList
  1. LLM Evaluation as Tensor Completion: Low Rank Structure and Semiparametric Efficiency

    Apr 7, 2026Jiachun Li, David Simchi-Levi, Will Wei SunSemiparametric InferenceLLM Evaluation

  2. SOTA or Luck? The Winner's Curse in LLM Leaderboards

    Nov 17, 2026Sikun Xu, Raphael Thomadsen, Dennis J. Zhang +1LLM EvaluationSelection Bias

  3. Privacy Without Regret: Differentially Private Inference-Time Alignment

    Aug 26, 2026Ishi Jain, Nandini Bhattad, Sayak Ray ChowdhuryInference-Time Language Model AlignmentBest-of-N Sampling