Many sequential decision-making procedures, such as active learning, implicitly assume that a model's current predictions agree with those it expects to make after updating on new data, a property we call coherence. Most models in use do not update by exact conditioning and are not guaranteed to be coherent, which can lead to worse decisions. Yet there is currently no general way to measure incoherence or to tell when it matters. We develop tools to measure incoherence and characterise when it affects decisions. We introduce the incoherence ratio, an MMD-based measure of how far any updating procedure departs from coherence, computed only from the procedure's own predictions and updates, and accompany it with an anytime-valid test of coherence. In Bayesian experimental design, acquisition objectives that are equivalent for coherent models choose different data under every approximate model we study, and the incoherence ratio tracks how far incoherence distorts them. We also introduce action coherence, which captures only the incoherence that changes a decision. What a resulting disagreement costs, however, is determined primarily by predictive accuracy. Sequential decisions depend on how a model updates as well as on what it predicts, so coherence, like accuracy, should be checked rather than assumed.
Figures & tables
Figure 1: Objective splitting under mean-field VI for Bayesian linear regression. (a) The mean-field model’s current predictive PY (dashed) and its expected future predictives QYA and QYB after acquiring candidates A and B . The inset magnifies the peak, where neither expected future predictive matches the current one, so the model is incoherent for both candidates. (b) The three forms of EPIG against the candidate design a . Under exact Bayes they coincide. Under mean-field VI, \EPIGH prefers B whereas EMI prefers A , and \EPIG\KL (dashed) stays within 0.002 nats of EMI.
Figure 2: Sparse GPs with fixed (blue) or re-selected (green) inducing points, and an incTNP. (a, b) Ψ against KL from the exact GP before and after the update. Vertical lines join procedures with the same number of inducing points, and bands show ±2 standard errors. (c) Fraction of tasks rejected at α=0.05 . The exact GP is never rejected.
Figure 3: Acquisition with mean-field Bayesian linear regression. Every curve is the test log loss of the mean-field model on the data it acquired. (a) Test log loss by acquisition step. (b) Excess log loss over acquisition with the exact predictive, averaged over steps, against Ψ . Points are labelled by the share ρ of the pool over which the acquisition predictive is made coherent, with ρ=0% the mean-field model with \EPIGH . EMI changes only the objective, so it is plotted at the mean-field model’s Ψ . Bands and error bars show ±2 standard errors.
Figure 4: Acquisition on the neural testbed for eight approximate models. Exact Bayes is omitted, as its objectives never disagree. (a) Distortion of the acquisition objective against Ψ . (b) Cost of a disagreement, the excess loss of \EPIGH ’s choice over EMI’s, against the model’s KL from the truth before the update. Curves are local linear fits, markers are model means, and bands show the 10th–90th percentiles in (a) and ±2 standard errors in (b). BBB is Bayes by backprop.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Model
M
Ψ
KL before
KL after
Rejected
Exact GP
–
0
0
0
0.00
incTNP
–
0.0130±0.0008
3.1×10−3
2.9×10−3
0.99
Fixed
10
0.124±0.006
0.47
0.66
0.99
12
0.093±0.005
0.18
0.35
0.99
14
0.066±0.005
4.3×10−2
0.17
0.95
16
0.040±0.004
4.5×10−3
7.0×10−2
0.83
Appendix
Table 1: Results of the sparse GP experiment over 200 tasks. Ψ is the mean over tasks with one standard error over environments, in closed form for the GP-based models and estimated from 1000 rounds for the incTNP. KL is the geometric mean over tasks, in nats per evaluation point, before and after the update, with values below 10−10 clamped before averaging. Rejected is the share of tasks rejected within 1000 rounds at level 0.05.
Model
Closed form
R=10
R=100
R=1000
Exact GP
0
−0.005±0.090
0.000±0.008
0.0000±0.0007
incTNP
–
0.017±0.093
0.014±0.010
0.013±0.002
Fixed, M=10
0.124
0.127±0.141
0.123±0.019
0.124±0.005
Fixed, M=24
0.012
0.016±0.111
0.012±0.011
0.012±0.002
Re-selected, M=20
0.0022
0.001±0.093
0.003±0.008
0.0022±0.0008
Appendix
Table 2: Estimates of Ψ after R rounds, as the mean over tasks ± the root mean square of the per-task leave-one-round-out standard errors. The estimates for different R are prefixes of the same rounds.
Dataset
OpenML id
Rows
Inputs
Classes
Training
Test
Pool
Block
Regression
concrete
44959
1030
8
–
515
100
415
128
california
44977
20640
8
–
1000
100
2000
250
yacht
42370
308
6
–
154
46
108
38
Classification
australian
40981
690
42
2
345
100
245
86
Appendix
Table 3: Datasets and the sizes of each part of a split. Inputs counts features after one-hot encoding. Sources: concrete [ Yeh, 1998a , Yeh, 1998b ] , california [ Pace and Barry, 1997 ] , yacht [ Gerritsma et al., 1981 ] , australian [ Quinlan, 1987 ] , cleveland [ Janosi et al., 1989 , Detrano et al., 1989 ] , german [ Hofmann, 1994 ] , vehicle [ Mowforth and Shepherd, 1987 ] and wdbc [ Wolberg et al., 1993 ] , all fetched from OpenML [ Vanschoren et al., 2013 ] .
Regression
Model
Ψ
Range
Rank
Rej.
GP
0.000
0.000 – 0.000
3.5
0.00
Sparse GP
0.007
0.004 – 0.018
9
0.13
GP, refitted
0.002
0.001 – 0.002
3.5
0.03
BART
0.710
0.632 – 0.863
7
1.00
XBART
0.378
0.271 – 0.787
5
1.00
Appendix
Table 4: Summary over datasets. Ψ is the median over datasets of the model’s mean over splits, with its range. Rank is the median over datasets of the model’s rank by test log loss among the models listed for its task (1 is best, ties averaged). Rej. is the share of the 30 (regression) or 50 (classification) splits rejected within 500 rounds. TabPFN, never rejected on its one split within 128 rounds, is shown as – since this is not comparable.
Model
concrete
california
yacht
GP ( ×10−4 )
−0.6±1.4
2.3±2.2
2.2±2.4
Sparse GP
0.007±0.001
0.004±0.000
0.018±0.004
GP, refitted
0.002±0.001
0.001±0.000
0.002±0.001
BART
0.632±0.035
0.710±0.051
0.863±0.073
XBART
0.378±0.034
0.271±0.018
0.787±0.075
Kernel ridge
0.013±0.002
0.008±0.001
0.119±0.012
Appendix
Table 5: Ψ by dataset: the mean over ten splits ± their standard deviation divided by 10 , or for TabPFN its one split ± the leave-one-round-out standard error.
Model
concrete
california
yacht
GP
0.283
0.738
−1.803
Sparse GP
0.457
0.889
−1.749
GP, refitted
0.283
0.738
−1.803
BART
0.151
0.773
−1.540
XBART
0.089
0.738
−0.128
Kernel ridge
0.412
0.762
−0.231
Appendix
Table 6: Test log loss in nats per output, on the standardised outputs for regression: the mean over ten splits, or TabPFN’s one split. The lowest on each dataset before rounding is in bold.
Model
concrete
california
yacht
Sparse GP
0.007 / 0.008 / 0.005 / 0.003
0.004 / 0.004 / 0.002 / 0.002
0.018 / 0.020 / 0.015 / 0.001
GP, refitted
0.002 / 0.002 / 0.002 / 0.002
0.001 / 0.001 / 0.001 / 0.001
0.002 / 0.002 / 0.001 / 0.001
BART
0.632 / 0.641 / 0.617 / 0.610
0.710 / 0.710 / 0.708 / 0.708
0.863 / 0.941 / 0.783 / 0.674
XBART
0.378 / 0.389 / 0.364 / 0.357
0.271 / 0.277 / 0.261 / 0.261
0.787 / 0.860 / 0.699 / 0.540
Kernel ridge
0.013 / 0.015 / 0.008 / 0.005
0.008 / 0.010 / 0.005 / 0.005
0.119 / 0.118 / 0.112 / 0.020
Nearest neighbours
0.461 / 0.468 / 0.452 / 0.441
0.470 / 0.473 / 0.463 / 0.463
0.701 / 0.704 / 0.703 / 0.683
Appendix
Table 7: Ψ with the kernel lengthscale at the noise scale / half it / twice it / one (the training outputs’ standard deviation). The GP, with ∣Ψ∣<5×10−4 at every lengthscale, is omitted.
Acquisition
Excess loss (nats)
Ψ ( ×10−3 )
Mean-field, \EPIGH ( ρ=0 )
0.270±0.011
1.05±0.05
Coherentised, ρ=1%
0.184±0.009
0.80±0.04
Coherentised, ρ=2.5%
0.133±0.008
0.55±0.03
Coherentised, ρ=5%
0.054±0.005
0.32±0.02
Coherentised, ρ=10%
0.014±0.003
0.13±0.01
Coherentised, ρ=100%
0.002±0.002
0
Appendix
Table 8: Results of the Bayesian linear regression experiment over 100 environments. Excess loss is the mean-field learner’s log loss less that after acquiring with the exact predictive, averaged over the 100 steps, with one standard error over environments. Ψ is measured before acquisition for an update on the whole pool.
Setting
Mean-field \EPIGH
Mean-field EMI
Random
ρ=5%
ρ=100%
Baseline
0.270±0.011
0.008±0.002
0.156±0.008
0.054±0.005
0.002±0.002
Low noise
1.966±0.070
0.920±0.027
0.929±0.037
0.033±0.007
0.009±0.004
High noise
0.065±0.004
0.003±0.001
0.042±0.003
0.017±0.003
0.001±0.002
Distant targets
0.385±0.019
0.006±0.002
0.270±0.013
0.017±0.003
−0.001±0.002
Misspecified
0.537±0.018
0.016±0.003
0.454±0.016
0.204±0.009
0.008±0.002
Appendix
Table 9: Excess loss as in Table 8 , with one standard error over environments, under changes to the baseline. Low and high noise set the noise variance to 0.1 and 3, against 0.8 in the baseline.
Disagreement
Model
Ψ
∥Δ∥
\EPIGH
\EPIG\KL
Cost
KL
Exact Bayes
0
0
0.00
0.00
0
0.171±0.003
IVON
0.0058
0.041
0.28
0.01
0.0084±0.0018
0.366±0.007
Laplace, KFAC
0.0067
0.059
0.40
0.01
0.0101±0.0020
0.310±0.007
Bayes by backprop
0.0102
0.094
0.80
0.10
0.0245±0.0027
0.362±0.007
incTNP
0.0103
0.097
0.26
0.02
0.0009±0.0005
0.168±0.003
Appendix
Table 10: Results on the neural testbed. Ψ and ∥Δ∥ are medians over environments. Disagreement is the share of decisions where each form of EPIG differs from EMI. Cost is averaged over all decisions, with agreements counting as zero, so the cost of a disagreement is Cost divided by the \EPIGH disagreement. KL measures accuracy before the update. Both are in nats with one standard error.
A large body of research measures model coherence based on output variance without adequately considering competing causes. We identify two such causes, ambiguity and indifference, and we introduce a set of 175 questions where contradicting answers cannot easily be explained by either. We then measure incoherence in terms of contradictions when resampling answers to the same question. In contrast to other methods our metric has high specificity, and only ranks models as incoherent when the issues are glaring. Even so, we find narrow finetunes score poorly. Inspecting inconsistencies flagged by our method, we find that model organisms from the literature display severe issues such as identity conflation, introspection failures and rationalizations. These findings suggest that the pathologies induced by narrow finetuning may limit what these models can tell us about coherent misaligned behaviour.
Robert Graham, Yariv Barsheshat, Phil Blandfort +1
Reliable decision-making requires more than accurate prediction: a model must preserve its beliefs, apply the relevant utilities, and recognize when the information needed to justify an action is missing. We study whether language models can learn this decision procedure from supervised fine-tuning and generalize it across domains and differing natural-language expressions of the decision challenge. Across 20 datasets, we explore challenges of belief instability and decision-making errors by first eliciting probabilities of outcomes and then varying only the utilities and the framing of the decision problems, while holding the evidence fixed. We train models to preserve elicited beliefs while selecting the action that maximizes expected utility, and evaluate transfer to unseen application domains, held-out framings, and different classes of payoff structures. We further introduce incomplete-information settings in which required utilities are withheld and replaced with irrelevant text, testing whether models can distinguish missing decision-relevant information from merely additional context. We find that targeted fine-tuning substantially improves coherent decision-making and that in many situations, learning transfers across domains and framings to situations unobserved during training. Further, models trained for decidability learn to identify when action cannot be justified based on missing information. Finally, we show the value of a routed system that considers separately the recognition of decision completeness and utility-sensitive decision execution.
A decision model can give probabilities that sum to one for every question yet disagree with itself when the same decision is broken into smaller steps. We study this form of probabilistic coherence in Jev and the English Laya checkpoint, using 2,500 matched examples per system across TREC, CLINC150, and MASSIVE. Across 72,000 classification questions, we compare direct fine-label predictions with broad-category probabilities and predictions reconstructed through those categories. Both systems show substantial disagreement: mean category-level total variation ranges from 0.219 to 0.349 for Jev and from 0.424 to 0.689 for Laya, on a scale where zero means exact agreement. The consequences differ sharply. On CLINC150, reconstruction reduces Jev's accuracy by 22.9 percentage points (paired 95% bootstrap interval: [-24.9, -20.9]) and improves Laya's by 21.3 points ([18.0, 24.5]). The same directions hold across all three datasets, with all six unadjusted accuracy-change intervals excluding zero. Improved accuracy can also accompany less reliable confidence: on MASSIVE, Laya gains 9.2 accuracy points while its expected calibration error rises from 0.046 to 0.124. Error analysis identifies both broad-category mistakes and within-category confusions. These findings show why decision systems need joint evaluation of accuracy, confidence calibration, and probability coherence in the workflow used by an application.
Saman Sarker Joy
Faculty of Computer Science and Information Technology Universiti Malaya Kuala Lumpur, Malaysia