cs.LGNov 6, 2026

Update Slope Is Bounded by Prior Calibration: A Structural Bound and Its Contingent Empirical Signature

Authors: Ben Jenkins, Mihaela Cardei

Organizations: Dept. of Electrical Engineering and Computer Science, Florida Atlantic University, Boca Raton, FL, USA

Abstract

Evaluations of LLM forecasters increasingly measure how forecasts improve as richer evidence is supplied. We identify two confounds in this update-slope metric. First, outcome-aware evidence construction substantially inflates measured responsiveness: on the same models and questions, slopes are roughly four times larger than with resolution-blinded evidence ladders. Second, update slope is a change score with a prior-dependent structural ceiling, so raw comparisons can reflect differences in starting calibration rather than updating behavior. The ceiling defines only a feasible region; whether slope and prior calibration are empirically coupled depends on headroom utilization and varies across our Manifold, ForecastBench, and ACLED pools. Despite this variability, conditioning on prior calibration roughly triples the association between intermediate updating and final Brier score under both blinded and outcome-aware construction. Question identity also explains substantially more variance than model identity across the evaluated pools. We recommend building evidence ladders blind to outcomes, conditioning on prior calibration, and reporting headroom-normalized measures alongside raw slopes.

Figures & tables

Explore similar work

CardsList
  1. Large language models exhibit unreliable updating of clinical judgment as patient evidence evolves

    Oct 2, 2026Min Zeng, Rui ZhangBelief Updating in LLMsLLM Reliability

  2. Can LLMs Predict the Future? A Brier Score Analysis of Prediction Markets

    Sep 26, 2026Yuanbo Li, Zekun Li, Xiaoyan congProbabilistic ForecastingLLM Evaluation