cs.LGNov 19, 2026

Bradley-Terry Training Can Distort Outcome Probabilities of Initially Accurate Language World Models

Authors: Xinze Shi, Litian Zhang, Zhenjie Su, Zelong Yu, Yuting Zhang, Qihua Feng, Zifeng Kang, Chaozhuo Li, 

Organizations: Beijing University of Posts and Telecommunications · Beihang University · Beijing Institute of Technology · Microsoft · 

Abstract

Comparative feedback, in which a judge says which of two candidate outcomes is closer to what happened, is a practical way to train language world models that assign probabilities to an action's outcomes. We show that, fitted with a Bradley-Terry objective, it need not teach the true outcome probabilities and can distort accurate ones. On a simulated robot push with enumerated outcome distributions, this objective takes a model that is accurate on unseen layouts (hazard error 2.4 percentage points, pp) and moves 33 to 43 pp of its probability onto outcomes that none of a condition's enumerated branches produces. The cause is the objective, not the comparisons: its optimum, computed before training, predicts where the model ends up, and a likelihood that knows the judge's rule learns from the same comparisons without the shift. The shift is robust: it recurs on unseen conditions, under five candidate laws and a three-fold denser enumeration, and about 29 pp remain without the physically impossible events. Its cost is conditional: with those events it inflates hazard probabilities, and a threshold inherited from the pre-feedback model costs about 3 pp of utility per decision on trained and 1 to 2 pp on unseen conditions, through false rejects; without them the utility loss averages below 1 pp over six seeds, though a small regret in action choice remains. Audits of readout, response and decision trace the effect from probabilities to actions.

Figures & tables

Explore similar work

CardsList
  1. Similar Models, Broken Odds: Where Luce's Axiom Fails in Chatbot Arena

    Oct 13, 2026Soovadeep Bakshi, Apurba Bose, Aishwarya Bose +1Discrete Choice ModelingBradley–Terry Model

  2. Scored vs. Generated Readouts in Behavioral Language Models: An Empirical Study of Elicitation Format

    Sep 9, 2026Touchapon Kraisingkorn, Krittin Pachtrachai, Wachiravit ModecruaLLM EvaluationLanguage Model Calibration