Bradley-Terry Training Can Distort Outcome Probabilities of Initially Accurate Language World Models
Authors: Xinze Shi, Litian Zhang, Zhenjie Su, Zelong Yu, Yuting Zhang, Qihua Feng, Zifeng Kang, Chaozhuo Li, Zhengxuan Hu
Organizations: Beijing University of Posts and Telecommunications · Beihang University · Beijing Institute of Technology · Microsoft · Queen Mary, University of London
Comparative feedback, in which a judge says which of two candidate outcomes is closer to what happened, is a practical way to train language world models that assign probabilities to an action's outcomes. We show that, fitted with a Bradley-Terry objective, it need not teach the true outcome probabilities and can distort accurate ones. On a simulated robot push with enumerated outcome distributions, this objective takes a model that is accurate on unseen layouts (hazard error 2.4 percentage points, pp) and moves 33 to 43 pp of its probability onto outcomes that none of a condition's enumerated branches produces. The cause is the objective, not the comparisons: its optimum, computed before training, predicts where the model ends up, and a likelihood that knows the judge's rule learns from the same comparisons without the shift. The shift is robust: it recurs on unseen conditions, under five candidate laws and a three-fold denser enumeration, and about 29 pp remain without the physically impossible events. Its cost is conditional: with those events it inflates hazard probabilities, and a threshold inherited from the pre-feedback model costs about 3 pp of utility per decision on trained and 1 to 2 pp on unseen conditions, through false rejects; without them the utility loss averages below 1 pp over six seeds, though a small regret in action choice remains. Audits of readout, response and decision trace the effect from probabilities to actions.
Figures & tables
Figure 1: Comparative feedback on a robot push with two hazards H . (a) Before feedback, the hazard probability q of a language world model is close to the truth and below a fixed threshold τ : the push is executed. (b) After feedback, 33 to 43 pp of probability ( Φ ) sit on outcomes that never occur in that condition, and safe pushes are held back. Bottom : the usual evaluation against our three audits. Scenes and strips schematic, numbers measured; frames from different rollouts.
Figure 2: The three audits: what each takes as input, holds fixed (snowflake) or updates (flame), and measures (yellow; Eqs. 6 , 7 and 9 ). q is a probability and 1 pp =0.01 ; D∗ , h and ϵop are in pp. Sketches are schematic: the red part of a bar in (a) is the hazard probability, each small bar in (b) is the response to one batch, and each strip in (c) is one condition under the threshold τ chosen for Frozen, with a red line for its regret; the pictures in (a) are simulation frames.
Figure 3: Push task in simulation; frames from four different branches. (a) Initial state. (b) Final frames without a hazard. (c) The 12 events of a rollout: final region, contact and drop. (d, e) The two hazards 0.2 s before, at (red outline) and 0.5 s after their first recorded frame, with the contact force and the cube’s lowest point relative to the table top. The model sees only text: layout, stated gap and push.
Table 1: Main results at N=64 . (a) Φ in pp on the 153 fractional trained conditions; other metrics against p∗ on all 213 (KL, entropy in nats; p∗ has entropy 0.526; TV: total variation; Mode: share of conditions with p∗ ’s most likely event; ECE: expected calibration error over events z or of q ; MAE: mean absolute error of q ); utility in pp per decision at λ=3 (inherited τ=0.23 ); false rejects (FR) and allows (FA) in %; *: additional truth information. (b) In pp (KL in nats); a row changes one property of the main round (12 events, 80 branches, trained pool, w=0.2 in r=(1−w)pθ0+w/12 ). Bold (a): Event BT worst of all readouts (columns with arrows). Gray (b): the 95% interval includes zero or favors Event BT. Icons as in Fig. 2 . † Pooled over six seeds (KL, Brier: range). ‡ t interval over seed means includes zero.
Figure 4: Push study: one starting model through the three audits. (a) Hazard MAE on blind layouts, model minus baseline; negative values favor the model, whose own MAE is 2.4 pp (gray: layouts that vary the clearance to the table edge). (b) Phantom mass on the 153 fractional trained conditions and on the unseen pool; N=∞ is training on expected feedback. (c) Total variation (TV) from the Event BT endpoint to forecasts fixed before training. (d) Utility per decision on the 213 trained conditions; the two bars of a budget are the two feedback seeds (red: Event BT worse; green: better). Whiskers: 95% intervals; open markers: second feedback seed; *: additional truth information.
Figure 5: Push study, one trained condition ( N=64 ): event probabilities and, right, phantom mass Φ . Tiles as in Fig. 3 c, with how many of the 80 branches end in the event (crossed: impossible); numbers: Event BT where p∗=0 .
Figure 6: Push study, robustness at N=64 : Event BT minus Full Channel with 95% intervals, in the main round, on the eight feasible events and on a 252-branch reference (the two bars of a setting are the two feedback seeds; blue: Φ of each optimum b∗ ). Utility is under the inherited threshold. Red: Event BT worse; green: better.