Prediction-Powered LLM Ranking with Adaptively Collected Human Feedback
Authors: Gabriel Sargent, Will Wei Sun, Zhengwu Zhang, Yufeng Liu
Organizations: Department of Statistics and Operations Research, University of North Carolina at Chapel Hill. · Daniels School of Business, Purdue University. · Department of Statistics, University of Michigan.
Ranking large language models (LLMs) through human evaluations is essential for downstream decision-making. Because human feedback is expensive, evaluation platforms often collect it adaptively, prioritizing informative model comparisons using previously observed feedback. To further improve precision with limited human feedback, evaluators can draw on abundant unrated responses and relatively inexpensive LLM judges; however, judge bias and adaptive dependence challenge standard i.i.d. inference frameworks. We propose Prediction-Powered Ranking (PP-Rank), a framework that combines LLM-judge predictions with human feedback to improve precision while providing valid inference under adaptive sampling. Data-driven prediction weights minimize the asymptotic covariance trace within the proposed class. In theory, we establish asymptotic normality for general evaluation targets and simultaneous ranking inference. A joint martingale analysis handles dependence between adaptive human feedback and auxiliary data. We further derive a rule for reducing exploration that prioritizes lower-cost comparisons while matching a conservative variance bound, and characterize allocation limits for regular T-rate estimation of pairwise means. Semi-synthetic experiments on UltraFeedback-Extended show improvements over human-only baselines in score estimation, confidence interval width, and ranking certification. Synthetic experiments demonstrate robustness to judge quality and adaptive allocation, while showing that PP-Rank can substantially reduce evaluation costs without sacrificing precision.
Figures & tables
Figure 1: Human feedback is often too scarce to precisely compare models, leaving decision-makers unable to certify conclusions about their relative abilities. PP-Rank additionally leverages unlabeled tasks and LLM-judge feedback to reduce uncertainty, enabling such conclusions to be certified. (Model responses shown are hypothetical.)
Uniform
Targeted
PP-Rank
Top-1 set certified
0.38
0.52
0.74
Table 1: Proportion of trials in which the top-1 set is correctly certified. Uniform and Targeted use the labeled-only IPW estimator under uniform sampling and an adaptive policy targeted at the top-1 question, respectively. PP-Rank uses the same adaptive policy. We set T=10,000 and run 200 matched trials per arm.
Figure 2: Left: minimum coverage across the ten models’ win rates against a uniformly chosen opponent, at nominal 90%. Right: average width of the same intervals for the two estimators that remain valid. For each method we run 1000 trials with T=10,000 . Only the inverse-propensity-weighted estimators remain valid as exploration decreases, and PP-Rank’s intervals are narrower at every exploration rate.
Labeled-only
PP-Rank
Improvement (%)
MSE ( ↓ )
0.0112
0.0072
35.6%
Avg. CI width ( ↓ )
0.341
0.276
19.0%
Avg. rank-set width ( ↓ )
2.074
1.867
10.0%
Exact ranking recovered ( ↑ )
0.286
0.357
24.8%
Table 2: Comparison of PP-Rank with the labeled-only IPW baseline. T=10,000 , ϵ=0.1 , 1,000 trials. Arrows indicate whether smaller ( ↓ ) or larger ( ↑ ) values are better. Improvement is the relative percentage improvement compared with the labeled-only baseline.
Figure 3: Left: percent of trials in which each model’s rank set contains its true rank; the shaded region marks coverage of at least the nominal 90%. Right: average 90% rank sets for each model. T=4,000 , ϵ=0.1 , 1,000 trials. Both methods achieve nominal coverage, and PP-Rank’s rank sets are narrower.
Policy
Labeled-only (no IPW)
Labeled-only (IPW)
Judge-only
PPI++
PP-Rank
MSE reduction ( ↑ )
Greedy
0.40
0.88
0.00
0.00
0.88
36%
Uncertainty
0.50
0.87
0.00
0.00
0.88
33%
Arena2024
0.88
0.88
0.00
0.86
0.87
30%
Arena2025
0.72
0.88
0.00
0.17
0.88
34%
Table 3: Results under four sampling policies. The first five numerical columns report minimum coverage of nominal 90% confidence intervals across all 45 pairwise Bradley–Terry score contrasts, for which the target is 0.90. The two IPW estimators stay near nominal coverage, while methods designed for i.i.d. data undercover under nonuniform sampling. The final column reports PP-Rank’s MSE reduction relative to the labeled-only IPW baseline. T=10,000 , ϵ=0.1 , 1,000 trials. PP-Rank reduces MSE under every policy.
Figure 4: Left: minimum coverage across the four models’ win rates at nominal 90%. Right: average width of the same intervals for the two estimators that remain valid. Only the inverse-propensity-weighted estimators remain valid as exploration decreases, and PP-Rank’s intervals are narrower at every exploration rate. We set T=5000 and τ=0.5 .
Method
Cost reduction ( ↑ )
MSE ratio
CI-width ratio
Rank-set width ratio
Labeled-only
—
1.00
1.00
1.00
PP-Rank, τ=0
52.6%
0.95
0.96
0.98
PP-Rank, τ=0.25
40.7%
0.98
0.98
0.99
PP-Rank, τ=0.5
14.3%
1.00
1.00
1.00
PP-Rank, τ=0.75
2.0%
1.00
1.00
1.00
PP-Rank, τ=1
0.2%
1.00
1.00
1.00
Table 4: PP-Rank reduces evaluation costs while maintaining precision comparable to the labeled-only IPW baseline, as its MSE, CI-width, and rank-set-width ratios relative to the baseline are all near 1. Results are averaged over 1,000 trials with T=5,000 and N=50,000 .
Department of Computer Science, Technion – Israel Institute of Technology · Department of Electrical and Computer Engineering, Technion – Israel Institute of Technology