cs.LGOct 21, 2026

Alternative Strategies for Active Learning with Bayesian Bradley Terry Model

Authors: Saman Jayasinghe, Sean Walton, Alma Rahat, Andy Gray, Chris Wheadon

Organizations: Swansea University, United Kingdom · Loughborough University, United Kingdom · Bath Spa University, United Kingdom · No More Marking Ltd, United Kingdom

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Comparative judgement uses pairwise comparisons to rank items, but recovering accurate rankings from a limited number of judgements depends on both the inference method and the pairs selected. We developed a Bayesian Bradley-Terry model (BBTM) that uses Polya-Gamma data augmentation for efficient Gibbs sampling and quantifies uncertainty in latent abilities and ranks. We propose three active learning strategies based on the knowledge gradient (KG) principle. We define our acquisition functions using Kendall's tau distance, posterior rank entropy, and posterior separation. Approximate one-step updates to posterior moments reduce the computational cost of evaluating candidate pairs. We also introduce Posterior Separation Reliability (PSR), which measures the spread of posterior mean abilities relative to posterior uncertainty. It provides an internal indicator of ranking reliability without requiring a reference ranking. In synthetic experiments covering different item counts, comparison budgets, and noise levels, BBTM generally recovers ranking more accurately than preference-based Bayesian comparative judgement (PBCJ). At low to moderate noise levels, the pairwise entropy pair selection strategy performs best in ranking recovery. However, at the highest noise levels, the KG strategy based on posterior separation produces better point rankings. PSR increases alongside improvements in ranking accuracy as comparisons accumulate. Robustness experiments identify the proportion of informative outcomes needed to meet specified accuracy targets within a fixed comparison budget. In a moderate noise setting, replacing 30% of outcomes with random choices has little effect on ranking performance. These findings demonstrate the value of Bayesian inference and active learning pair selection for recovering ranks from limited, noisy paired comparisons.

Figures & tables

Figure 1: Convergence of BBTM and PBCJ across six pair selections, for σ=5.0\sigma=5.0 , N=10N=10 , K=10K=10 . Each subplot shows the median tau distance across 5050 seeds against the number of comparisons. The shaded region indicates the interquartile range. Results for σ=1.25\sigma=1.25 and σ=10.0\sigma=10.0 are given in Appendix A .
Figure 2: Performance comparison at σ=5.0\sigma=5.0 . Each panel corresponds to one combination of a ranking model (BBTM, top row; PBCJ, bottom row) and a pair-selection method (columns), evaluated over the N×KN\times K grid. The colour scale shows the number of losses V(i)V(i) , that is, the number of competing combinations that significantly outperform the given one under the Wilcoxon signed-rank test, with the darkest cells indicating a combination not beaten by any other ( V(i)=0V(i)=0 ). The number in white indicates the median final normalised Kendall’s τ\tau distance over 50 repeats for that configuration, with lower values indicating better ranking recovery. Heatmaps for the remaining noise levels are provided in Appendix B .
Figure 3: Pair coverage, the fraction of unique pairs sampled, against the number of comparisons, for four greedy selectors ( N=15N=15 , K=30K=30 , σ=5.0\sigma=5.0 , 5050 seeds). Each panel shows the coverage across exploration levels ε∈{0,0.05,0.10,0.20}\varepsilon\in\{0,0.05,0.10,0.20\} . Coverage for the remaining noise levels ( σ∈{0.0001,1.25,2.5,10.0}\sigma\in\{0.0001,1.25,2.5,10.0\} ) is given in Appendix F .
Figure 4: Convergence plots, Normalised Kendall’s τ\tau distance against the number of comparisons, for four greedy selectors ( N=15N=15 , K=30K=30 , σ=5.0\sigma=5.0 , 5050 seeds). Each panel shows the median τ\tau distance across exploration levels ε∈{0,0.05,0.10,0.20}\varepsilon\in\{0,0.05,0.10,0.20\} , with shaded bands indicating the interquartile range. Convergence for the remaining noise levels ( σ∈{0.0001,1.25,2.5,10.0}\sigma\in\{0.0001,1.25,2.5,10.0\} ) is given in Appendix F .
Figure 5: BBTM rank-distribution recovery at σ=5.0\sigma=5.0 . Each panel corresponds to one pair-selection method, evaluated over the N×KN\times K grid. The colour scale shows the number of losses V(i)V(i) under the Wilcoxon signed-rank test, with the darkest cells indicating a method not beaten by any other ( V(i)=0V(i)=0 ). The number in white reports the median worst-matched JSD over the repeats, with lower values indicating better recovery of the target rank distributions. JSD heatmaps for the remaining noise levels are given in Appendix C .
Figure 6: A comparison between target rank densities and the estimated rank densities from the BBTM after N×K=5×10=50N\times K=5\times 10=50 pairwise comparisons. The top row shows the target rank distributions, whilst the bottom row shows the posterior rank densities. The expected ranks of items marked by the red dashed line and the per-item JSD value is given.
Figure 7: PSR as a Bayesian reliability measure in BBTM. Results shown for KG-Entropy with N = 10 items, K = 10, σ\sigma = 5.0 (a) Median PSR versus Median normalised Kendall’s Tau distance across 50 runs (colour indicates the comparison index). (b) Symlog transformation of the PSR values. (c) Median PSR values over comparisons.
Figure 8: BBTM robustness to the true skill-based and coin toss outcomes for the entropy pair selection at σ=5.0\sigma=5.0 for N=10N=10 , K=10K=10 , and 5050 seeds. The XX -axis shows the threshold level ( α\alpha ), which is the proportion of true skill-based outcomes, and (1−α)(1-\alpha) is the proportion of coin toss outcomes. The distribution of the final normalised τ\tau value of 50 seeds for a fixed budget ( N×KN\times K ) is shown in each box plot at each α\alpha level. Results for the noise levels σ=1.25,5.0,\sigma=1.25,5.0, and 10.010.0 for entropy and KG-SSR pair selections are given in Appendix E .
   τ≤0.10\tau\leq 0.10τ≤0.05\tau\leq 0.05
σ\sigmaSelectionFloor τ\tauα∗\alpha^{*}Tolerated random levelα∗\alpha^{*}Tolerated random level
1.251.25entropy0.000.000.50.550%50\%0.60.640%40\%
KG-SSR0.000.000.50.550%50\%0.70.730%30\%
5.05.0entropy0.040.040.70.730%30\%0.70.730%30\%
KG-SSR0.040.040.80.820%20\%0.90.910%10\%
10.010.0entropy0.090.090.80.820%20\%−-−-
Table 1: Summary of the BBTM robustness results across noise levels and selection strategies. At α=0\alpha=0 (all outcomes random), the median τ≈0.53\tau\approx 0.53 for all configurations. This is no better than ranking the items at random. The floor is the median τ\tau at α=1\alpha=1 (all outcomes skill-based), giving the best accuracy at each σ\sigma . The threshold α∗\alpha^{*} is the smallest proportion of skill-based outcomes required to reach the accuracy targets of τ≤0.10\tau\leq 0.10 and τ≤0.05\tau\leq 0.05 . The tolerated random level (1−α∗)(1-\alpha^{*}) is the proportion of genuine judgements that can be replaced by random outcomes while still achieving this accuracy. All values are medians over 5050 seeds at a fixed budget of N×K=100N\times K=100 comparisons ( N=10N=10 , K=10K=10 ).
Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Model convergence plot for σ=1.25\sigma=1.25 .
Figure 10: Model convergence plot for σ=10.0\sigma=10.0 .
Figure 11: Performance comparison heatmap for σ=0.0001\sigma=0.0001 .
Figure 12: Performance comparison heatmap for σ=1.25\sigma=1.25 .
Figure 13: Performance comparison heatmap for σ=2.5\sigma=2.5 .
Figure 14: Performance comparison heatmap for σ=5.0\sigma=5.0 .
Figure 15: Performance comparison heatmap for σ=10.0\sigma=10.0 .
Figure 16: Performance comparison heatmap for σ=0.0001\sigma=0.0001 .
Figure 17: Performance comparison heatmap for σ=1.25\sigma=1.25 .
Figure 18: Performance comparison heatmap for σ=2.5\sigma=2.5 .
Figure 19: Performance comparison heatmap for σ=5.0\sigma=5.0 .
Figure 20: Performance comparison heatmap for σ=10.0\sigma=10.0 .
Figure 21: PSR behaviour for entropy selection ( N=10N=10 , K=10K=10 , σ=5.0\sigma=5.0 ).
Figure 22: PSR behaviour for KG-SSR selection ( N=10N=10 , K=10K=10 , σ=5.0\sigma=5.0 ).
Figure 23: Robustness plots for N=10N=10 , K=10K=10 , σ=1.25\sigma=1.25
Figure 24: Robustness plots for N=10N=10 , K=10K=10 , σ=5.0\sigma=5.0
Figure 25: Robustness plots for N=10N=10 , K=10K=10 , σ=10.0\sigma=10.0
Figure 26: Pair coverage for σ=0.0001\sigma=0.0001 .
Figure 27: Convergence for σ=0.0001\sigma=0.0001 .
Figure 28: Pair coverage for σ=1.25\sigma=1.25 .
Figure 29: Convergence for σ=1.25\sigma=1.25 .
Figure 30: Pair coverage for σ=2.5\sigma=2.5 .
Figure 31: Convergence for σ=2.5\sigma=2.5 .
Figure 32: Pair coverage for σ=10.0\sigma=10.0 .
Figure 33: Convergence for σ=10.0\sigma=10.0 .
More in this topic

Pairwise Ranking

Pairwise ranking infers ordered preferences from comparisons between items, systems, or responses.

See all 62 papers

Explore similar work

CardsList
May 14, 2026

Active Learners as Efficient PRP Rerankers

Pairwise Ranking Prompting (PRP) elicits pairwise preference judgments from an LLM, which are then aggregated into a ranking, usually via classical sorting algorithms. However, judgments are noisy, order-sensitive, and sometimes intransitive, so sorting assumptions do not match the setting. Because sorting aims to recover a full permutation, truncating it to meet a call budget does not produce a dependable top-K. We thus reframe PRP reranking as active learning from noisy pairwise comparisons and show that active rankers are drop-in replacements that improve NDCG@10 per call in the call-constrained regime. Our noise-robust framework also introduces a randomized-direction oracle that uses a single LLM call per pair. This approach converts systematic position bias into zero-mean noise, enabling unbiased aggregate ranking without the cost of bidirectional calls.

Jeremías Figueiredo Paschmann, Juan Kaplan, Francisco Nattero +3

ELIAS Lab, Departamento de Ingeniería, Universidad de San Andrés

Pairwise Preference LearningPairwise ComparisonPairwise Preference EvaluationLearning to RankActive LearningLLM Reranking+3
AskPDF
Jul 2, 2026

Ask the Right Comparison:Bias-Aware Bayesian Active Top-kk Ranking with LLM Judges

Large language models (LLMs) are increasingly used as cheap, scalable judges that compare candidate outputs pairwise -- to rank responses, select models, or triage papers. Yet LLM judges are both noisy and systematically biased: they favor verbose or well-formatted answers and exhibit position effects, so simply aggregating their votes recovers a ranking of presentation, not of true quality. We study the practical goal of identifying the \topk{} items under a fixed comparison budget, and make two contributions. First, we cast judging as Bayesian inference over latent quality with explicit, judge-specific bias covariates (verbosity, position), regularized by a shrinkage prior so that the data decide which biases a given judge actually exhibits. Second, we introduce a \topk-aware active acquisition rule that chooses the next comparison to maximally reduce uncertainty about \topk{} membership, rather than about the full ranking. On a controlled benchmark with known ground-truth quality, judged by sixteen real LLMs spanning open and proprietary families (Llama, Qwen, Phi-4, GPT-4o-mini/5.1/5.5, Gemini, DeepSeek, and Claude Haiku/Sonnet/Opus), naive aggregation plateaus at a wrong \topk{} on biased judges regardless of budget, while our bias-aware model recovers it; \topk-aware acquisition reaches this ceiling with far fewer comparisons than round-robin or a global-uncertainty (D-optimal) rule. Bias is real but heterogeneous and capability-dependent: cheap and mid-tier judges carry a strong verbosity bias that our model corrects (lifting recall from \sim$$0.5--0.60.6 to 0.840.84--1.01.0), whereas the frontier judges we tested show little bias and already rank accurately, so bias-aware modeling changes little there.

Jian Xu, Delu Zeng, John Paisley +1

RIKEN iTHEMS · RIKEN AIP · South China University of Technology +1

Pairwise ComparisonLLM-as-a-JudgeLearning to RankLLM RerankingPairwise RankingLanguage Model Bias Evaluation+2
AskPDF
Aug 3, 2026

Isotonic Bradley-Terry Model for Paired Comparison Data

In this paper, we study prediction problems for paired comparison data, for example, predicting the win probability between two unmatched players and ranking all the players according to the order of their strengths by using win probability data between two matched players. Paired comparison data are typically analyzed using Bradley-Terry and Thurstone-Mosteller models. These models predict the win probability by transforming the difference between learned rate parameters, which represent players';strengths, with a pre-specified inverse link function, and employ the order of learned rate parameters for player ranking. However, these models may suffer from model misspecification owing to the selection of a fixed inverse link function. Therefore, in this study, we propose to learn the rate parameters by a (sub-)gradient method and the inverse link function by an isotonic regression technique alternately. The proposed model guarantees monotonic improvement in training error, and is likely to yield an exact tie when the available data is insufficient to establish a strict ranking. We also verified that the proposed model could improve the win probability prediction and ranking performance through numerical experiments with synthetic data and real-world data of football Premier League, baseball MLB, and tennis ATP tour.

Ryoya Yamasaki

Hitotsubashi Institute for Advanced Study, Hitotsubashi University, 2-1 Naka, Kunitachi, 186-8601, Tokyo, Japan.

Pairwise Preference LearningPairwise ComparisonPairwise Preference EvaluationPairwise RankingBradley–Terry ModelIsotonic Regression
AskPDF