Many sequential decision-making procedures, such as active learning, implicitly assume that a model's current predictions agree with those it expects to make after updating on new data, a property we call coherence. Most models in use do not update by exact conditioning and are not guaranteed to be coherent, which can lead to worse decisions. Yet there is currently no general way to measure incoherence or to tell when it matters. We develop tools to measure incoherence and characterise when it affects decisions. We introduce the incoherence ratio, an MMD-based measure of how far any updating procedure departs from coherence, computed only from the procedure's own predictions and updates, and accompany it with an anytime-valid test of coherence. In Bayesian experimental design, acquisition objectives that are equivalent for coherent models choose different data under every approximate model we study, and the incoherence ratio tracks how far incoherence distorts them. We also introduce action coherence, which captures only the incoherence that changes a decision. What a resulting disagreement costs, however, is determined primarily by predictive accuracy. Sequential decisions depend on how a model updates as well as on what it predicts, so coherence, like accuracy, should be checked rather than assumed.
Figures & tables
Figure 1: Objective splitting under mean-field VI for Bayesian linear regression. (a) The mean-field model’s current predictive PY (dashed) and its expected future predictives QYA and QYB after acquiring candidates A and B . The inset magnifies the peak, where neither expected future predictive matches the current one, so the model is incoherent for both candidates. (b) The three forms of EPIG against the candidate design a . Under exact Bayes they coincide. Under mean-field VI, \EPIGH prefers B whereas EMI prefers A , and \EPIG\KL (dashed) stays within 0.002 nats of EMI.
Figure 2: Sparse GPs with fixed (blue) or re-selected (green) inducing points, and an incTNP. (a, b) Ψ against KL from the exact GP before and after the update. Vertical lines join procedures with the same number of inducing points, and bands show ±2 standard errors. (c) Fraction of tasks rejected at α=0.05 . The exact GP is never rejected.
Figure 3: Acquisition with mean-field Bayesian linear regression. Every curve is the test log loss of the mean-field model on the data it acquired. (a) Test log loss by acquisition step. (b) Excess log loss over acquisition with the exact predictive, averaged over steps, against Ψ . Points are labelled by the share ρ of the pool over which the acquisition predictive is made coherent, with ρ=0% the mean-field model with \EPIGH . EMI changes only the objective, so it is plotted at the mean-field model’s Ψ . Bands and error bars show ±2 standard errors.
Figure 4: Acquisition on the neural testbed for eight approximate models. Exact Bayes is omitted, as its objectives never disagree. (a) Distortion of the acquisition objective against Ψ . (b) Cost of a disagreement, the excess loss of \EPIGH ’s choice over EMI’s, against the model’s KL from the truth before the update. Curves are local linear fits, markers are model means, and bands show the 10th–90th percentiles in (a) and ±2 standard errors in (b). BBB is Bayes by backprop.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Model
M
Ψ
KL before
KL after
Rejected
Exact GP
–
0
0
0
0.00
incTNP
–
0.0130±0.0008
3.1×10−3
2.9×10−3
0.99
Fixed
10
0.124±0.006
0.47
0.66
0.99
12
0.093±0.005
0.18
0.35
0.99
14
0.066±0.005
4.3×10−2
0.17
0.95
16
0.040±0.004
4.5×10−3
7.0×10−2
0.83
Appendix
Table 1: Results of the sparse GP experiment over 200 tasks. Ψ is the mean over tasks with one standard error over environments, in closed form for the GP-based models and estimated from 1000 rounds for the incTNP. KL is the geometric mean over tasks, in nats per evaluation point, before and after the update, with values below 10−10 clamped before averaging. Rejected is the share of tasks rejected within 1000 rounds at level 0.05.
Model
Closed form
R=10
R=100
R=1000
Exact GP
0
−0.005±0.090
0.000±0.008
0.0000±0.0007
incTNP
–
0.017±0.093
0.014±0.010
0.013±0.002
Fixed, M=10
0.124
0.127±0.141
0.123±0.019
0.124±0.005
Fixed, M=24
0.012
0.016±0.111
0.012±0.011
0.012±0.002
Re-selected, M=20
0.0022
0.001±0.093
0.003±0.008
0.0022±0.0008
Appendix
Table 2: Estimates of Ψ after R rounds, as the mean over tasks ± the root mean square of the per-task leave-one-round-out standard errors. The estimates for different R are prefixes of the same rounds.
Dataset
OpenML id
Rows
Inputs
Classes
Training
Test
Pool
Block
Regression
concrete
44959
1030
8
–
515
100
415
128
california
44977
20640
8
–
1000
100
2000
250
yacht
42370
308
6
–
154
46
108
38
Classification
australian
40981
690
42
2
345
100
245
86
Appendix
Table 3: Datasets and the sizes of each part of a split. Inputs counts features after one-hot encoding. Sources: concrete [ Yeh, 1998a , Yeh, 1998b ] , california [ Pace and Barry, 1997 ] , yacht [ Gerritsma et al., 1981 ] , australian [ Quinlan, 1987 ] , cleveland [ Janosi et al., 1989 , Detrano et al., 1989 ] , german [ Hofmann, 1994 ] , vehicle [ Mowforth and Shepherd, 1987 ] and wdbc [ Wolberg et al., 1993 ] , all fetched from OpenML [ Vanschoren et al., 2013 ] .
Regression
Model
Ψ
Range
Rank
Rej.
GP
0.000
0.000 – 0.000
3.5
0.00
Sparse GP
0.007
0.004 – 0.018
9
0.13
GP, refitted
0.002
0.001 – 0.002
3.5
0.03
BART
0.710
0.632 – 0.863
7
1.00
XBART
0.378
0.271 – 0.787
5
1.00
Appendix
Table 4: Summary over datasets. Ψ is the median over datasets of the model’s mean over splits, with its range. Rank is the median over datasets of the model’s rank by test log loss among the models listed for its task (1 is best, ties averaged). Rej. is the share of the 30 (regression) or 50 (classification) splits rejected within 500 rounds. TabPFN, never rejected on its one split within 128 rounds, is shown as – since this is not comparable.
Model
concrete
california
yacht
GP ( ×10−4 )
−0.6±1.4
2.3±2.2
2.2±2.4
Sparse GP
0.007±0.001
0.004±0.000
0.018±0.004
GP, refitted
0.002±0.001
0.001±0.000
0.002±0.001
BART
0.632±0.035
0.710±0.051
0.863±0.073
XBART
0.378±0.034
0.271±0.018
0.787±0.075
Kernel ridge
0.013±0.002
0.008±0.001
0.119±0.012
Appendix
Table 5: Ψ by dataset: the mean over ten splits ± their standard deviation divided by 10 , or for TabPFN its one split ± the leave-one-round-out standard error.
Model
concrete
california
yacht
GP
0.283
0.738
−1.803
Sparse GP
0.457
0.889
−1.749
GP, refitted
0.283
0.738
−1.803
BART
0.151
0.773
−1.540
XBART
0.089
0.738
−0.128
Kernel ridge
0.412
0.762
−0.231
Appendix
Table 6: Test log loss in nats per output, on the standardised outputs for regression: the mean over ten splits, or TabPFN’s one split. The lowest on each dataset before rounding is in bold.
Model
concrete
california
yacht
Sparse GP
0.007 / 0.008 / 0.005 / 0.003
0.004 / 0.004 / 0.002 / 0.002
0.018 / 0.020 / 0.015 / 0.001
GP, refitted
0.002 / 0.002 / 0.002 / 0.002
0.001 / 0.001 / 0.001 / 0.001
0.002 / 0.002 / 0.001 / 0.001
BART
0.632 / 0.641 / 0.617 / 0.610
0.710 / 0.710 / 0.708 / 0.708
0.863 / 0.941 / 0.783 / 0.674
XBART
0.378 / 0.389 / 0.364 / 0.357
0.271 / 0.277 / 0.261 / 0.261
0.787 / 0.860 / 0.699 / 0.540
Kernel ridge
0.013 / 0.015 / 0.008 / 0.005
0.008 / 0.010 / 0.005 / 0.005
0.119 / 0.118 / 0.112 / 0.020
Nearest neighbours
0.461 / 0.468 / 0.452 / 0.441
0.470 / 0.473 / 0.463 / 0.463
0.701 / 0.704 / 0.703 / 0.683
Appendix
Table 7: Ψ with the kernel lengthscale at the noise scale / half it / twice it / one (the training outputs’ standard deviation). The GP, with ∣Ψ∣<5×10−4 at every lengthscale, is omitted.
Acquisition
Excess loss (nats)
Ψ ( ×10−3 )
Mean-field, \EPIGH ( ρ=0 )
0.270±0.011
1.05±0.05
Coherentised, ρ=1%
0.184±0.009
0.80±0.04
Coherentised, ρ=2.5%
0.133±0.008
0.55±0.03
Coherentised, ρ=5%
0.054±0.005
0.32±0.02
Coherentised, ρ=10%
0.014±0.003
0.13±0.01
Coherentised, ρ=100%
0.002±0.002
0
Appendix
Table 8: Results of the Bayesian linear regression experiment over 100 environments. Excess loss is the mean-field learner’s log loss less that after acquiring with the exact predictive, averaged over the 100 steps, with one standard error over environments. Ψ is measured before acquisition for an update on the whole pool.
Setting
Mean-field \EPIGH
Mean-field EMI
Random
ρ=5%
ρ=100%
Baseline
0.270±0.011
0.008±0.002
0.156±0.008
0.054±0.005
0.002±0.002
Low noise
1.966±0.070
0.920±0.027
0.929±0.037
0.033±0.007
0.009±0.004
High noise
0.065±0.004
0.003±0.001
0.042±0.003
0.017±0.003
0.001±0.002
Distant targets
0.385±0.019
0.006±0.002
0.270±0.013
0.017±0.003
−0.001±0.002
Misspecified
0.537±0.018
0.016±0.003
0.454±0.016
0.204±0.009
0.008±0.002
Appendix
Table 9: Excess loss as in Table 8 , with one standard error over environments, under changes to the baseline. Low and high noise set the noise variance to 0.1 and 3, against 0.8 in the baseline.
Disagreement
Model
Ψ
∥Δ∥
\EPIGH
\EPIG\KL
Cost
KL
Exact Bayes
0
0
0.00
0.00
0
0.171±0.003
IVON
0.0058
0.041
0.28
0.01
0.0084±0.0018
0.366±0.007
Laplace, KFAC
0.0067
0.059
0.40
0.01
0.0101±0.0020
0.310±0.007
Bayes by backprop
0.0102
0.094
0.80
0.10
0.0245±0.0027
0.362±0.007
incTNP
0.0103
0.097
0.26
0.02
0.0009±0.0005
0.168±0.003
Appendix
Table 10: Results on the neural testbed. Ψ and ∥Δ∥ are medians over environments. Disagreement is the share of decisions where each form of EPIG differs from EMI. Cost is averaged over all decisions, with agreements counting as zero, so the cost of a disagreement is Cost divided by the \EPIGH disagreement. KL measures accuracy before the update. Both are in nats with one standard error.