Evaluations of LLM forecasters increasingly measure how forecasts improve as richer evidence is supplied. We identify two confounds in this update-slope metric. First, outcome-aware evidence construction substantially inflates measured responsiveness: on the same models and questions, slopes are roughly four times larger than with resolution-blinded evidence ladders. Second, update slope is a change score with a prior-dependent structural ceiling, so raw comparisons can reflect differences in starting calibration rather than updating behavior. The ceiling defines only a feasible region; whether slope and prior calibration are empirically coupled depends on headroom utilization and varies across our Manifold, ForecastBench, and ACLED pools. Despite this variability, conditioning on prior calibration roughly triples the association between intermediate updating and final Brier score under both blinded and outcome-aware construction. Question identity also explains substantially more variance than model identity across the evaluated pools. We recommend building evidence ladders blind to outcomes, conditioning on prior calibration, and reporting headroom-normalized measures alongside raw slopes.
Figures & tables
Fig. 1: Aggregate dose response under the two ladder constructions (same models and questions). Starting points nearly coincide at L0, where no ladder content is shown; the outcome-aware ladder then produces roughly 3.5× the aggregate movement of the blinded ladder ( 4× in headroom-utilization terms).
outcome-aware
blinded
Model
aˉm,0
βˉm
um
βˉm
um
Haiku 4.5
0.577
0.0489
0.289
0.0030
0.018
Opus 4.7
0.510
0.0583
0.298
0.0246
0.126
Sonnet 4.6
0.523
0.0787
0.412
0.0218
0.108
GPT-4o
0.533
0.0655
0.351
0.0101
0.053
GPT-5
0.500
0.0690
0.345
0.0253
0.126
TABLE I: Per-model prior, slope, and utilization um=βˉm/(K3Hm) under both constructions ( aˉm,0 from the outcome-aware run; blinded um uses the blinded run’s nearly identical priors).
Metric
Kendall τ
p
Raw slope βˉm
+0.143
0.773
Hake normalized gain
+0.333
0.381
DRRmβ (slope residual)
+0.429
0.239
Murphy resolution gain
+0.619
0.069
DRRmρ (resolution residual)
+0.714
0.0302
TABLE II: Kendall τ between candidate metrics and the Brier@L3 skill ranking (best =1 convention; full-precision scores). Descriptive at n=7 .