cs.LGSep 11, 2026

How Wrong Can a Good Predictor Be? Diverging Updates with Vanishing Predictive KL

Authors: Qifu WenShuaijun LiuZihan ZhouXi ZengNingxin Su

Abstract

Accurate posterior prediction need not require accurate approximation of Bayesian updates. We prove that an unbounded gap between the update maps can coexist with vanishing predictive KL for every fixed finite K2K\ge2 in a stationary symmetric Gaussian HMM. Exact Bayesian mixing and an explicit deterministic radial filter act on the same K1K-1 belief coordinates. As q0+q\to0^+, their separation in centered logits in the worst case grows at least linearly in the natural confidence scale LK(q)L_K(q), while their categorical DKL(exactradial)D_{\mathrm{KL}}(\mathrm{exact}\|\mathrm{radial}) vanishes at the same explicit witness. Along stationary HMM trajectories, the expected terminal KL between filtered posteriors also converges to zero at H(q)=log(q)/c+1H(q)=\lceil-\log(q)/c\rceil+1. Typical blocks without switches drive both filters into a common confidence cone, where softmax curvature suppresses their disagreement; a single Gaussian maximal event controls adaptive noise. A sweep with equally spaced Gaussians over K{2,4,8}K\in\{2,4,8\} illustrates the opposing trends, and binary controls at long horizons compare saturating and nonsaturating recurrences. The result isolates two missing links between internal update gaps and predictive cost: the contribution of separating states to expected loss and decoder sensitivity. Thus even an unbounded internal update gap does not by itself certify predictive failure. The construction is fixed in KK and does not provide a universal criterion for when compression is harmless or characterize when internal gaps must incur task loss.

Explore similar work

CardsList
  1. LLMs are Bayesian, In Expectation, Not in Realization

    Jul 15, 2025Leon Chlon, Fatima Sheaib, Zein Khamis +3PosteriorIn-Context Learning