Evaluations of LLM forecasters increasingly measure how forecasts improve as richer evidence is supplied. We identify two confounds in this update-slope metric. First, outcome-aware evidence construction substantially inflates measured responsiveness: on the same models and questions, slopes are roughly four times larger than with resolution-blinded evidence ladders. Second, update slope is a change score with a prior-dependent structural ceiling, so raw comparisons can reflect differences in starting calibration rather than updating behavior. The ceiling defines only a feasible region; whether slope and prior calibration are empirically coupled depends on headroom utilization and varies across our Manifold, ForecastBench, and ACLED pools. Despite this variability, conditioning on prior calibration roughly triples the association between intermediate updating and final Brier score under both blinded and outcome-aware construction. Question identity also explains substantially more variance than model identity across the evaluated pools. We recommend building evidence ladders blind to outcomes, conditioning on prior calibration, and reporting headroom-normalized measures alongside raw slopes.
Figures & tables
Fig. 1: Aggregate dose response under the two ladder constructions (same models and questions). Starting points nearly coincide at L0, where no ladder content is shown; the outcome-aware ladder then produces roughly 3.5× the aggregate movement of the blinded ladder ( 4× in headroom-utilization terms).
outcome-aware
blinded
Model
aˉm,0
βˉm
um
βˉm
um
Haiku 4.5
0.577
0.0489
0.289
0.0030
0.018
Opus 4.7
0.510
0.0583
0.298
0.0246
0.126
Sonnet 4.6
0.523
0.0787
0.412
0.0218
0.108
GPT-4o
0.533
0.0655
0.351
0.0101
0.053
GPT-5
0.500
0.0690
0.345
0.0253
0.126
TABLE I: Per-model prior, slope, and utilization um=βˉm/(K3Hm) under both constructions ( aˉm,0 from the outcome-aware run; blinded um uses the blinded run’s nearly identical priors).
Metric
Kendall τ
p
Raw slope βˉm
+0.143
0.773
Hake normalized gain
+0.333
0.381
DRRmβ (slope residual)
+0.429
0.239
Murphy resolution gain
+0.619
0.069
DRRmρ (resolution residual)
+0.714
0.0302
TABLE II: Kendall τ between candidate metrics and the Brier@L3 skill ranking (best =1 convention; full-precision scores). Descriptive at n=7 .
Large language models fine-tuned for forecasting can be accurate yet poorly calibrated, and their chain-of-thought (CoT) reasoning may not faithfully reflect the evidence behind a forecast. We ask whether internal representations offer a more direct window into both. Working with Eternis-Forecaster 8B on OpenForesight, we train representation-pooling probes on intermediate activations and find they achieve substantially better calibration; a result that also holds for GLM-4.7-Flash and GLM-4.5-Air. We then assess CoT faithfulness through evidence ablation and diversionary injection: removing an influential source in the prompt often changes the model's forecast while leaving the reasoning trace untouched. The same probes function as lie detectors: their activations track behavioral shifts far better than the reasoning trace does, and they also predict the direction of change in 84% of cases, including when the CoT conceals the perturbation's influence. Finally, forced answering reveals that forecasts are largely fixed before reasoning begins: a single pre-reasoning pass recovers the committed answer and confidence, and routing questions by the spread of this pre-set answer distribution saves 30-47% of generated tokens, with no loss of accuracy. Together, these results establish probing internal representations as a practical tool for calibrating, auditing, and triaging language model forecasters and reasoning models more broadly.
Large language models (LLMs) are increasingly explored for clinical reasoning, but whether they appropriately revise judgments as patient evidence evolves remains unclear. We evaluated longitudinal belief updating using matched intensive-care trajectories from electronic health records. Across diverse LLMs, conditioning on a preceding judgment more often increased than reduced prediction error when estimates changed, replicated for a second endpoint. Controlled interventions revealed two failure modes. First, with preceding assessment fixed, models responded more strongly to worsening than matched improving respiratory evidence; this asymmetry persisted after headroom normalization at moderate and strong evidence levels. Second, with current evidence fixed, increasing prior risk from 10% to 90% shifted estimates by 26.2 percentage points, demonstrating causal influence of prior model beliefs. Prompting did not restore reliable updating. Evidence-Validated Longitudinal Update (EVLU) identified fewer, more reliable revisions, revealing a reliability-coverage trade-off. These findings establish longitudinal belief updating as a distinct dimension of LLM reliability.
Min Zeng, Rui Zhang
Division of Computational Health Sciences, Department of Surgery, University of Minnesota, Minneapolis, 55455, MN, USA.
We study whether model upgrades improve probability estimates for prediction-market questions. Our Resolved Market Forecasting (RMF) benchmark contains 3,000 resolved binary questions across nine domains, on which we evaluate six Claude and Qwen model variants using a question-only, zero-shot protocol. We assess Brier scores relative to an empirical base-rate predictor, examine their Murphy decomposition, and compare models through paired differences, with results stratified by event category and timing relative to training cutoffs. In the reported post-cutoff stratum, the four Claude models achieve Brier scores of 0.183-0.192, improving on the base-rate reference by 0.024-0.033. Qwen 32B does not significantly outperform that reference, although its paired Brier is 0.024 lower than that of the 7B checkpoint. The evaluated Claude version and tier upgrades yield no significant improvement. Within-model differences across event categories exceed the observed differences among Claude variants. These results show why forecasting scores should be interpreted alongside simple probability baselines and question composition: under this protocol, newer versions or higher model tiers do not consistently produce more accurate probabilities.