Bradley-Terry Training Can Distort Outcome Probabilities of Initially Accurate Language World Models
Authors: Xinze Shi, Litian Zhang, Zhenjie Su, Zelong Yu, Yuting Zhang, Qihua Feng, Zifeng Kang, Chaozhuo Li, Zhengxuan Hu
Organizations: Beijing University of Posts and Telecommunications · Beihang University · Beijing Institute of Technology · Microsoft · Queen Mary, University of London
Comparative feedback, in which a judge says which of two candidate outcomes is closer to what happened, is a practical way to train language world models that assign probabilities to an action's outcomes. We show that, fitted with a Bradley-Terry objective, it need not teach the true outcome probabilities and can distort accurate ones. On a simulated robot push with enumerated outcome distributions, this objective takes a model that is accurate on unseen layouts (hazard error 2.4 percentage points, pp) and moves 33 to 43 pp of its probability onto outcomes that none of a condition's enumerated branches produces. The cause is the objective, not the comparisons: its optimum, computed before training, predicts where the model ends up, and a likelihood that knows the judge's rule learns from the same comparisons without the shift. The shift is robust: it recurs on unseen conditions, under five candidate laws and a three-fold denser enumeration, and about 29 pp remain without the physically impossible events. Its cost is conditional: with those events it inflates hazard probabilities, and a threshold inherited from the pre-feedback model costs about 3 pp of utility per decision on trained and 1 to 2 pp on unseen conditions, through false rejects; without them the utility loss averages below 1 pp over six seeds, though a small regret in action choice remains. Audits of readout, response and decision trace the effect from probabilities to actions.
Figures & tables
Figure 1: Comparative feedback on a robot push with two hazards H . (a) Before feedback, the hazard probability q of a language world model is close to the truth and below a fixed threshold τ : the push is executed. (b) After feedback, 33 to 43 pp of probability ( Φ ) sit on outcomes that never occur in that condition, and safe pushes are held back. Bottom : the usual evaluation against our three audits. Scenes and strips schematic, numbers measured; frames from different rollouts.
Figure 2: The three audits: what each takes as input, holds fixed (snowflake) or updates (flame), and measures (yellow; Eqs. 6 , 7 and 9 ). q is a probability and 1 pp =0.01 ; D∗ , h and ϵop are in pp. Sketches are schematic: the red part of a bar in (a) is the hazard probability, each small bar in (b) is the response to one batch, and each strip in (c) is one condition under the threshold τ chosen for Frozen, with a red line for its regret; the pictures in (a) are simulation frames.
Figure 3: Push task in simulation; frames from four different branches. (a) Initial state. (b) Final frames without a hazard. (c) The 12 events of a rollout: final region, contact and drop. (d, e) The two hazards 0.2 s before, at (red outline) and 0.5 s after their first recorded frame, with the contact force and the cube’s lowest point relative to the table top. The model sees only text: layout, stated gap and push.
Table 1: Main results at N=64 . (a) Φ in pp on the 153 fractional trained conditions; other metrics against p∗ on all 213 (KL, entropy in nats; p∗ has entropy 0.526; TV: total variation; Mode: share of conditions with p∗ ’s most likely event; ECE: expected calibration error over events z or of q ; MAE: mean absolute error of q ); utility in pp per decision at λ=3 (inherited τ=0.23 ); false rejects (FR) and allows (FA) in %; *: additional truth information. (b) In pp (KL in nats); a row changes one property of the main round (12 events, 80 branches, trained pool, w=0.2 in r=(1−w)pθ0+w/12 ). Bold (a): Event BT worst of all readouts (columns with arrows). Gray (b): the 95% interval includes zero or favors Event BT. Icons as in Fig. 2 . † Pooled over six seeds (KL, Brier: range). ‡ t interval over seed means includes zero.
Figure 4: Push study: one starting model through the three audits. (a) Hazard MAE on blind layouts, model minus baseline; negative values favor the model, whose own MAE is 2.4 pp (gray: layouts that vary the clearance to the table edge). (b) Phantom mass on the 153 fractional trained conditions and on the unseen pool; N=∞ is training on expected feedback. (c) Total variation (TV) from the Event BT endpoint to forecasts fixed before training. (d) Utility per decision on the 213 trained conditions; the two bars of a budget are the two feedback seeds (red: Event BT worse; green: better). Whiskers: 95% intervals; open markers: second feedback seed; *: additional truth information.
Figure 5: Push study, one trained condition ( N=64 ): event probabilities and, right, phantom mass Φ . Tiles as in Fig. 3 c, with how many of the 80 branches end in the event (crossed: impossible); numbers: Event BT where p∗=0 .
Figure 6: Push study, robustness at N=64 : Event BT minus Full Channel with 95% intervals, in the main round, on the eight feasible events and on a 252-branch reference (the two bars of a setting are the two feedback seeds; blue: Φ of each optimum b∗ ). Utility is under the inherited threshold. Red: Event BT worse; green: better.
Chatbot Arena gives each language model one Bradley-Terry score, which fixes how often any model should beat any other. Choice theory predicts where this fails: near-copies are compared on the few details in which they differ, so votes between them are more decisive than their scores imply, exactly for the comparison a lab cares about, a new version against the old one. On 1.8 million Arena votes, votes between versions of one model are 1.37 times as decisive as Bradley-Terry predicts, between sizes of one generation 1.26 times and between models of one lab 1.12 times; a release a year later points the same way, and the effect survives tie, prompt, topic and timing checks. How alike two models' answers are worded and formatted tracks the effect. Plain scores leave the odds between relatives about 9% too close to even on held-out votes; a correction with four to six parameters roughly halves this and predicts held-out votes better, mostly by modeling ties, while the leaderboard order barely moves; frozen and carried to new releases, its gains in head-to-head odds are too small for our test to establish. With language-model judges and three-answer menus, a near-copy takes votes mostly from its twin when choices are pooled over prompts (log-odds shift +0.10, 95% interval 0.05 to 0.16), less so within a prompt.
Language models fine-tuned on customer behavior can predict outcomes and generate explanations, but these readouts are often treated as interchangeable. Holding model checkpoint and prompt content fixed, we compare probabilities obtained by scoring answer tokens with predictions generated after a written rationale. Across 13 model-domain cells covering four retail tasks in three markets, including two using fully public data and checkpoints, the scored readout ranks outcomes more accurately in 12 of 13 cells (two-sided sign test, p approximately 0.003), by 1.5 to 14.5 points in area under the receiver operating characteristic curve (AUC). Paired bootstrap confidence intervals exclude zero in every newly measured cell. The gap varies with task-specific supervision and mismatch between training and serving formats, ranging from -2.2 points for an untuned base model to +13.7 for rationale-format supervision. Analysis of approximately 9,000 rationales identifies two correlates: reduced reliance on the dominant predictive feature and convergence on stock formulations. Probability saturation does not track the gap. A third readout, eliciting a probability before any verdict, improves calibration (Brier score from 0.47 to 0.15) while ranking within noise of scoring, but only for outcome rates represented in training; it is worse than scoring when the scored head is already calibrated. We interpret these differences through the objectives matched by each readout, identify training choices that narrow the gap, and propose retaining generated rationales while sourcing ranking from the scored head.
Reinforcement learning (RL) can only teach a language model what its reward can recognize. Answer checkers are exact but need a reference answer; learned reward models and LLM judges work everywhere, but their scores are not probabilities. We study RLCDR (Reinforcement Learning from Calibrated Decision Rewards). An LLM converted into a decision model answers a typed question about a response ("is the final answer correct?", "which step is the first error?") with a probability distribution, read in one forward pass, debiased over option orders and calibrated with about a hundred labels. Training Qwen3-1.7B with GRPO on MATH, this reference-free reward raises MATH500 accuracy from 72.2% to 76.8% (+4.65 points, p < 1e-4), matching training on the answer key (76.9%), while a random reward gains only +0.65. The gain grows with difficulty, from 44.8% to 51.7% on the hardest level. On the policy's own changing samples the reward stays calibrated: its expected calibration error is half that of a softmax judge on the same 27B backbone (0.074 vs. 0.141), and it shows no sign of reward hacking. Without step training, the same model finds the first wrong step better than a dedicated 7B process reward model (ProcessBench F1 0.772 vs. 0.722, 20/20 splits). We also map where the approach stops being enough: trained only on problems the base model mostly fails, the reference-free reward still improves the policy (+1.9 MATH500 points, p = 0.03) but trails the answer key, because its probabilities become overconfident on that distribution. We state six hypotheses in advance and test each with a control designed to falsify it.