Prediction-Powered LLM Ranking with Adaptively Collected Human Feedback
Authors: Gabriel Sargent, Will Wei Sun, Zhengwu Zhang, Yufeng Liu
Organizations: Department of Statistics and Operations Research, University of North Carolina at Chapel Hill. · Daniels School of Business, Purdue University. · Department of Statistics, University of Michigan.
Ranking large language models (LLMs) through human evaluations is essential for downstream decision-making. Because human feedback is expensive, evaluation platforms often collect it adaptively, prioritizing informative model comparisons using previously observed feedback. To further improve precision with limited human feedback, evaluators can draw on abundant unrated responses and relatively inexpensive LLM judges; however, judge bias and adaptive dependence challenge standard i.i.d. inference frameworks. We propose Prediction-Powered Ranking (PP-Rank), a framework that combines LLM-judge predictions with human feedback to improve precision while providing valid inference under adaptive sampling. Data-driven prediction weights minimize the asymptotic covariance trace within the proposed class. In theory, we establish asymptotic normality for general evaluation targets and simultaneous ranking inference. A joint martingale analysis handles dependence between adaptive human feedback and auxiliary data. We further derive a rule for reducing exploration that prioritizes lower-cost comparisons while matching a conservative variance bound, and characterize allocation limits for regular T-rate estimation of pairwise means. Semi-synthetic experiments on UltraFeedback-Extended show improvements over human-only baselines in score estimation, confidence interval width, and ranking certification. Synthetic experiments demonstrate robustness to judge quality and adaptive allocation, while showing that PP-Rank can substantially reduce evaluation costs without sacrificing precision.
Figures & tables
Figure 1: Human feedback is often too scarce to precisely compare models, leaving decision-makers unable to certify conclusions about their relative abilities. PP-Rank additionally leverages unlabeled tasks and LLM-judge feedback to reduce uncertainty, enabling such conclusions to be certified. (Model responses shown are hypothetical.)
Uniform
Targeted
PP-Rank
Top-1 set certified
0.38
0.52
0.74
Table 1: Proportion of trials in which the top-1 set is correctly certified. Uniform and Targeted use the labeled-only IPW estimator under uniform sampling and an adaptive policy targeted at the top-1 question, respectively. PP-Rank uses the same adaptive policy. We set T=10,000 and run 200 matched trials per arm.
Figure 2: Left: minimum coverage across the ten models’ win rates against a uniformly chosen opponent, at nominal 90%. Right: average width of the same intervals for the two estimators that remain valid. For each method we run 1000 trials with T=10,000 . Only the inverse-propensity-weighted estimators remain valid as exploration decreases, and PP-Rank’s intervals are narrower at every exploration rate.
Labeled-only
PP-Rank
Improvement (%)
MSE ( ↓ )
0.0112
0.0072
35.6%
Avg. CI width ( ↓ )
0.341
0.276
19.0%
Avg. rank-set width ( ↓ )
2.074
1.867
10.0%
Exact ranking recovered ( ↑ )
0.286
0.357
24.8%
Table 2: Comparison of PP-Rank with the labeled-only IPW baseline. T=10,000 , ϵ=0.1 , 1,000 trials. Arrows indicate whether smaller ( ↓ ) or larger ( ↑ ) values are better. Improvement is the relative percentage improvement compared with the labeled-only baseline.
Figure 3: Left: percent of trials in which each model’s rank set contains its true rank; the shaded region marks coverage of at least the nominal 90%. Right: average 90% rank sets for each model. T=4,000 , ϵ=0.1 , 1,000 trials. Both methods achieve nominal coverage, and PP-Rank’s rank sets are narrower.
Policy
Labeled-only (no IPW)
Labeled-only (IPW)
Judge-only
PPI++
PP-Rank
MSE reduction ( ↑ )
Greedy
0.40
0.88
0.00
0.00
0.88
36%
Uncertainty
0.50
0.87
0.00
0.00
0.88
33%
Arena2024
0.88
0.88
0.00
0.86
0.87
30%
Arena2025
0.72
0.88
0.00
0.17
0.88
34%
Table 3: Results under four sampling policies. The first five numerical columns report minimum coverage of nominal 90% confidence intervals across all 45 pairwise Bradley–Terry score contrasts, for which the target is 0.90. The two IPW estimators stay near nominal coverage, while methods designed for i.i.d. data undercover under nonuniform sampling. The final column reports PP-Rank’s MSE reduction relative to the labeled-only IPW baseline. T=10,000 , ϵ=0.1 , 1,000 trials. PP-Rank reduces MSE under every policy.
Figure 4: Left: minimum coverage across the four models’ win rates at nominal 90%. Right: average width of the same intervals for the two estimators that remain valid. Only the inverse-propensity-weighted estimators remain valid as exploration decreases, and PP-Rank’s intervals are narrower at every exploration rate. We set T=5000 and τ=0.5 .
Method
Cost reduction ( ↑ )
MSE ratio
CI-width ratio
Rank-set width ratio
Labeled-only
—
1.00
1.00
1.00
PP-Rank, τ=0
52.6%
0.95
0.96
0.98
PP-Rank, τ=0.25
40.7%
0.98
0.98
0.99
PP-Rank, τ=0.5
14.3%
1.00
1.00
1.00
PP-Rank, τ=0.75
2.0%
1.00
1.00
1.00
PP-Rank, τ=1
0.2%
1.00
1.00
1.00
Table 4: PP-Rank reduces evaluation costs while maintaining precision comparable to the labeled-only IPW baseline, as its MSE, CI-width, and rank-set-width ratios relative to the baseline are all near 1. Results are averaged over 1,000 trials with T=5,000 and N=50,000 .
With PRECISE, we extended Prediction-Powered Inference to produce bias-corrected estimates of ranking evaluation metrics by combining a small human-labeled set with a large LLM-judged set. PPI is provably unbiased regardless of the LLM judge's error profile. We make it applicable to hierarchical metrics like Precision@K, where annotations are per-document but the metric is per-query, by reducing the output-space computation from O(2^|C|) to O(2^K). On the ESCI benchmark, augmenting 30 human annotations with Claude 3 Sonnet judgments reduces the standard error of Precision@4 estimates from 4.45 to 3.50 (a 21% relative reduction). In a production system, our framework correctly identified the best of three system variants from 100 human labels and 2 hours of domain-expert annotation; A/B testing confirmed this ranking with +407 bps in daily sales.
Selecting the best large language model (LLM) for a fixed benchmark is often expensive, since exhaustive evaluation requires running every model on every example. Multi-armed bandit (MAB) algorithms can reduce the number of LLM calls by sequentially selecting the next model-example pair to evaluate, thereby avoiding wasted evaluations on clearly underperforming models. Further savings can be achieved by predicting model scores from the partially observed model-example score matrix using low-rank factorization. However, such predictions are not ground truth: they can be biased and may therefore lead to incorrect identification of the best model. In this work, we propose a principled framework that combines MAB with cheap predicted scores without compromising statistical validity. Specifically, we derive doubly robust estimators of each model's performance that use the low-rank predictions to reduce variance. This enables the construction of valid finite-sample confidence intervals in our setting, where models are selected adaptively and examples are sampled without replacement. Empirical results on real-world benchmarks show that our approach reduces the number of required evaluations, yielding meaningful savings in compute and cost while accurately identifying the best-performing model.
Elad Tolochinsky, Yaniv Tenzer, Yaniv Romano
Department of Computer Science, Technion – Israel Institute of Technology · Department of Electrical and Computer Engineering, Technion – Israel Institute of Technology
Computerized Adaptive Testing (CAT) has proven effective for efficient LLM evaluation on multiple-choice benchmarks, but modern LLM evaluation increasingly relies on generation tasks where outputs are scored continuously rather than marked correct/incorrect. We present a principled extension of IRT-based adaptive testing to continuous bounded scores (ROUGE, BLEU, LLM-as-a-Judge) by replacing the Bernoulli response distribution with a heteroskedastic normal distribution. Building on this, we introduce an uncertainty aware ranker with adaptive stopping criteria that achieves reliable model ranking while testing as few items and as cheaply as possible. We validate our method on five benchmarks spanning n-gram-based, embedding-based, and LLM-as-judge metrics. Our method improves ranking correlation by 0.13 τ over random sampling and has 99% accuracy on confident predictions while using 2% of the items after a one-time calibration step.
Esma Balkır, Alice Pernthaller, Marco Basaldella +2
Trismik · Leverhulme Centre for the Future of Intelligence, University of Cambridge · Universitat Politècnica de València +1