Sharp and Stable Label-Free Calibration Certificates from Logical Coherence

Takahiro Kato

cs.LGOct 15, 2026

Sharp and Stable Label-Free Calibration Certificates from Logical Coherence

Authors: Takahiro Kato

Organizations: NP-hard, Inc.

Abstract

Query-conditioned probabilistic predictors answer logically related questions one at a time, and labels to check their calibration are often missing. For each restriction on what a bettor's stakes may use, a minimax theorem identifies, for finitely supported laws of the answers, the sharp label-free lower bound on the corresponding calibration residual. On bundles of different events its mean, marginal and joint stages can separate strictly, and a bundle's bound cannot be attributed to one question without assumptions on the others. The sharp marginal bound is discontinuous in the answers, and no label-free lower bound continuous in the Wasserstein distance stays within a constant factor of it. Lipschitz stakes give a stable bound, sharp for a smooth calibration residual and, for every positive Lipschitz constant, positive exactly when the sharp marginal bound is. On i.i.d. streams, projected online gradient ascent at fixed nodes gives fixed-horizon certificates whose shortfall is bounded by grid, learning and statistical terms, and predictable betting gives anytime-valid ones. In a preregistered study of four predictors on two data sets, all seven confirmatory certificates were positive (the eighth pair did not pass the pilot). A certificate shows only that calibration is at least this bad; zero means "not refuted".

Figures & tables

Figure 1: Four-point separation (Theorem 7 ). (a) The bundle

(F_{A},F_{\neg A})

takes four values with probability

1/4

each; the coherent set

\mathcal{C}

is the segment

F_{A}+F_{\neg A}=1

. Rows 1–2 alone are Counterexample 2 : marginally calibrated under a compatible label law, yet pointwise incoherent. (b) With equal weights, the sharp label-free lower bounds on the mean, marginal and joint calibration residuals are

0<1/16<3/16

Row	$F_{A}$	$F_{\neg A}$	$P(A\mid F)$	$u_{A}(F_{A})$	$u_{\neg A}(F_{\neg A})$	$d_{u}$
1	$0$	$\tfrac{1}{2}$	$0$	$-1$	$0$	$0$
2	$1$	$\tfrac{1}{2}$	$1$	$1$	$0$	$0$
3	$0$	$\tfrac{3}{4}$	$0$	$-1$	$-1$	$\tfrac{1}{8}$
4	$1$	$\tfrac{1}{4}$	$1$	$1$	$1$	$\tfrac{1}{8}$

Table 1: An attaining label law and optimal stakes for Figure 1 : under this law

\Theta_{w}=1/16

, and these stakes earn

\mathbb{E}[d_{u}]=1/16

, so the sharpness of the marginal bound can be checked without an LP solver.

Figure 2: Sharpness versus stability on the two-point example

F^{\varepsilon}=(0,\tfrac{1}{2}-\varepsilon),(1,\tfrac{1}{2}+\varepsilon)

(Proposition 9 , Corollary 11 ). (a) The exact marginal bound jumps from

0

to above

\tfrac{1}{4}

as soon as

\varepsilon>0

, and equals the joint bound for

\varepsilon>0

; the stabilised bound

L_{H}=(\tfrac{1}{4}+\tfrac{\varepsilon}{2})\min\{1,H\varepsilon\}

is continuous (lines: closed form; dots: Lipschitz-constrained LP). (b) Rounding the outputs to

0.01

sends

L_{\mathrm{marg}}

back to

0

for

\varepsilon<0.005

, while

L_{H}

moves by at most

(1+H)W_{1,w}

(Theorem 12 ).

Condition	Perturbation	$W_{1,w}$	$(1+H)W_{1,w}$	$\|\Delta L_{\mathrm{marg}}\|$	$\|\Delta L_{H}\|$
four-point	jitter 0.001	0.0008	0.0045	0.0000	0.0002
four-point	jitter 0.01	0.0075	0.0450	0.0000	0.0024
four-point	jitter 0.05	0.0375	0.2250	0.0000	0.0156
four-point	quantize 0.01	0.0000	0.0000	0.0000	0.0000
four-point	quantize 0.1	0.0125	0.0750	0.0000	0.0000
four-point	clip 0.01	0.0050	0.0300	0.0050	0.0050

Table 2: Perturbations of the synthetic conditions against the bound of Theorem 12 (

H=5

|\Delta L_{H}|\leq(1+H)W_{1,w}

always holds, while

|\Delta L_{\mathrm{marg}}|

has no such guarantee.

Figure 3: Where the fixed-

T

certificate loses ground (Lemma 14 , Proposition 15 , Theorem 16 ). (a) A predictor answers “

A

?” and “not

A

?” with the same probability

x\in\{0.05,\dots,0.95\}

, uniformly; its true label law is the one attaining

L_{H}

. With

H=6

and a uniform grid of

G=6

nodes,

L_{H}-b_{T}

splits into the grid gap

L_{H}-L_{H,G}

, the learning gap

L_{H,G}-\rho_{T}

and the statistical gap

\rho_{T}-b_{T}

, where

\rho_{T}

is the average population value of the stakes used and

b_{T}=T^{-1}\sum_{t\leq T}d_{t}-a_{T}

; curves are medians over 10 seeds with 10–90% bands, and the reported bound is

\max\{0,b_{T}\}

. The worst-case guarantee of Theorem 16 is vacuous here (

H\Delta/2>L_{H}

) although the bound approaches

L_{H,G}

. (b) The partial-identification gap

\Gamma_{H}-L_{H}

, a different decomposition: zero under the attaining label law, positive under a true label law with a 20% base rate of

A

Condition	$H$	$G$	$T$	$L_{H}$	$L_{H,G}$	$\rho_{T}$	$b_{T}$	learning	statistics
same answer	6	6	100	0.2400	0.2290	0.0120	-0.2597	0.2170	0.2717
same answer	6	6	1,000	0.2400	0.2290	0.1306	0.0447	0.0984	0.0859
same answer	6	6	10,000	0.2400	0.2290	0.2180	0.1917	0.0110	0.0263
four-point (Fig. 1)	8	5	100	0.0625	0.0625	0.0020	-0.2697	0.0605	0.2717
four-point (Fig. 1)	8	5	1,000	0.0625	0.0625	0.0220	-0.0644	0.0405	0.0864
four-point (Fig. 1)	8	5	10,000	0.0625	0.0625	0.0578	0.0304	0.0047	0.0275

Table 3: The decomposition of Figure 3 at three horizons (medians over 10 seeds). On the four-point example of Figure 1 (

H=8

G=5

L_{H,G}=L_{H}=L_{\mathrm{marg}}=1/16

and the sequential method approaches it.

Figure 4: Anytime-valid lower bounds (Proposition 17 ) on the preregistered confirmatory streams, at

\alpha=0.05/7

per certificate. Each panel is one system on one data set, over the budget

t

up to its

T_{\max}

(so the

x

scales differ), with its own

y

scale. The three ways of learning the stakes share the stream (the same items in the same arrival order), the bundle, the betting rule and the total budget. The

\mathcal{U}_{H,G}

bounds are on

L_{H,G}

; the

\mathcal{U}_{0}

stakes are constants, a mean-only audit (they target

L_{0}

). The split method learns on the first

\lfloor T_{\max}/2\rfloor

items without scoring them, so its bound is

0

there. The

\mathcal{U}_{H,G}

-sequential bound is the confirmatory certificate; its value at

T_{\max}

is in each panel’s title. Every bound is on the bundle total, not on any one question. Jev on DBpedia-14 did not meet the preregistered pilot criterion for collecting confirmation data (“not collected”); its pilot anytime bound, on the 200 exploration items, was

0

AG News ( $H=2$ , $G=6$ )
		anytime			fixed- $T$
System	$T_{\max}$	$\mathcal{U}_{H,G}$ seq.	$\mathcal{U}_{H,G}$ split	$\mathcal{U}_{0}$ seq.	$\mathcal{U}_{H,G}$ seq.	$\mathcal{U}_{H,G}$ split
Jev	10,000	0.029	0.034	0.031	0	0
Eikos	10,000	0.095	0.102	0.098	0.0641	0.0587
Qwen3.5	10,000	0.185	0.191	0.186	0.1543	0.1473
DeBERTa	10,000	0.070	0.074	0.061	0.0386	0.0308

Table 4: The certificates of Figure 4 at

T_{\max}

, all at the same

\alpha

: the confirmatory anytime bound (bold) and the secondary results. Every anytime bound, the confirmatory one included, is on a grid of

10^{-3}

; the fixed-

T

bounds (Theorem 16 ) are rounded to four decimals, and

0

means the bound was clipped at zero. Under the assumptions of Proposition 17 (the items of a stream i.i.d. in arrival order, predictable stakes), each positive confirmatory bound refutes that its system is marginally calibrated (

\Theta_{w}=0

) on that bundle, the seven together at family-wise level

0.05

; a zero is never read as evidence of calibration.

Work	Quantity	Observed	Optimised over	Guarantee
This paper	least information-restricted calibration residual over the compatible label laws (mean, marginal, joint, smooth)	law of the bundle and logic; no labels	stakes restricted in information and Lipschitz constant, settled in the worst world	sharpness (finite support for the general minimax); strict separation on an example; no attribution to a given question; discontinuity and no uniform constant factor; $(1+H)$ -stability; zero set kept for $H>0$ (population); for i.i.d. items and predictable stakes, fixed- $T$ validity, approach for the stakes of Proposition 15 , and anytime validity without a rate
De Bona and Finger (2015) ; Schervish et al. (2003)	incoherence: distance to coherent assessments, sure loss	assessments and logic; no labels	bounded stakes, or coherent assessments	identities by LP duality; postulates; no link to calibration
Andrews and Sarkar (2026)	worst-case sure profit per item	predictions and logic; no labels	stakes bounded in $\ell_{1}$ on the whole bundle	identity with a supremum-norm distance; bootstrap intervals; no link to calibration
Paleka et al. (2025)	arbitrage in a logarithmic scoring market	forecasts and relations; no labels	arbitrageur’s prices	values per checked tuple; correlation with Brier scores, empirical
Arieli et al. (2021)	feasibility of joint posteriors under a common prior	joint law of posteriors on one binary state; no state	trades in $[-1,1]$ on one’s own posterior	feasible under a prior in $(0,1)$ if and only if no trade profits, apart from the laws $F\equiv 0^{m}$ and $F\equiv 1^{m}$
Kong et al. (2026)	least calibration error of two forecasters over the compatible label laws	joint law of two forecasts of one label; no labels	zero-mean bounded trades on one’s own forecast	bracket within a factor of two; zero set; population only

Table 5: The closest works from four viewpoints. In the table, “labels” means that realised outcomes are observed; “logic” means that relations among different events are used.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Power design on a synthetic bundle shaped like the main one (

K=4

: a choice query, atoms and complements, with equal weights for the three roles). The choice query is calibrated; the yes/no answers are overconfident by

\delta

and yes-biased by

\beta

. (a) The empirical learning gap

L_{H,G}-\rho_{T}

of projected OGD (median over 5 seeds),

H=2

, against

G/(2\eta T)+\eta/2

(dotted), the bound that Proposition 15 gives on its expectation, not on each seed, for the fixed step

\eta=\sqrt{G/T_{\max}}

(Appendix C ); it meets

\sqrt{G/T}

only at

T=T_{\max}=10^{4}

. (b) Anytime-valid lower bounds (Proposition 17 ,

\alpha=0.05

, the mixture of

\lambda_{k}=2^{-k}

k=1,\dots,8

) of the three ways of learning the stakes on the same streams: constants learnt online (

\mathcal{U}_{0}

\mathcal{U}_{H,G}

learnt on the first half of the budget and held fixed, and

\mathcal{U}_{H,G}

learnt online; medians over 10 seeds with 10–90% bands. The budget counts the learning items too. With only 64 support points the split method’s batch fit is nearly exact.

$n$	one-sided $\sqrt{2\log(1/\alpha)/n}$	Theorem 16 $a_{T}$	planned
200	0.1731	0.1921	–
500	0.1095	0.1215	0.110
1,000	0.0774	0.0859	0.077
2,000	0.0547	0.0607	–
5,000	0.0346	0.0384	0.035
10,000	0.0245	0.0272	0.025

Appendix

Table 6: Power design. Top: Hoeffding widths at

\alpha=0.05

; “planned” is what the study plan assumed. Middle: grid and learning gaps at

T=10^{4}

(medians over 5 seeds;

\delta=0.4

\beta=0.03

) against

H\Delta/2

, the worst case of the grid gap (Lemma 14 ), and

\sqrt{G/T}

, the bound on the expected learning gap (Proposition 15 , Appendix C ). Bottom: the budget at which the bound first turns positive in the median seed (

N_{\mathrm{any}}

N_{\mathrm{fixed}}

; about half the seeds detect) and in at least 90% of the seeds (

N_{\mathrm{any}}^{90\%}

), read off runs with

T_{\max}=10^{4}

(

H=2

G=6

, 10 seeds).

condition	procedure	role	$H$	$G$	mode	$\hat{p}$	pass line
Counterexample 2	$\mathcal{U}_{H,G}$ seq.	valid	6	6	fixed-T	0.000	0.040
Counterexample 2	$\mathcal{U}_{H,G}$ seq.	valid	6	6	anytime	0.000	0.071
Counterexample 2	$\mathcal{U}_{0}$ seq.	valid	6	6	fixed-T	0.000	0.040
Counterexample 2	$\mathcal{U}_{0}$ seq.	valid	6	6	anytime	0.000	0.071
Counterexample 2	$\mathcal{U}_{H,G}$ split	valid	6	6	fixed-T	0.000	0.040
Counterexample 2	$\mathcal{U}_{H,G}$ split	valid	6	6	anytime	0.004	0.071

Appendix

Table 7: Coverage by simulation (

R=1{,}000

replicates of

T=1{,}000

items,

\alpha=0.05

\hat{p}

is the share of replicates in which the checked event fails: the fixed-

T

bound above its target (level

\alpha/2

), Theorem 16 ’s whole chain failing (level

\alpha

), or the anytime bound above its target at some

t\leq T

(level

\alpha

). A valid procedure passes when

\hat{p}

is at most the pass line; a negative control (clipping each item at zero, choosing the stake after seeing the item, the uncorrected empirical LP) must exceed it. “same answer” is the predictor of Figure 3 (a) under its attaining label law, on which the population-optimal stake (“fixed optimum”) has expected profit exactly

L_{H,G}

; “yes to both” answers close to yes to both

A

and

\neg A

, so that at

H=1

and

G=11

the whole chain of Theorem 16 can be checked (“fixed-T chain”); “fixed stake” is one stake fixed in advance whose profit has mean exactly

0

Figure 6: A blind spot of every label-free audit. One stream of bundles from the predictor of Figure 3 (a), which gives “

A

?” and “not

A

?” the same answer; the law of

Y

given

F

changes twice while

P_{F}

stays put. The certificate’s trajectory is the same in every regime, while the true

\Theta_{w}

falls and then rises; a labelled audit (plug-in

\Theta_{w}

over the last 1,000 items) follows it. Medians over 10 seeds with 10–90% bands.

$Y\mid F$	$L_{H}$	$L_{H,G}$	true $\Gamma_{H}$	true $\Theta_{w}$
attaining $L_{H}$	0.240	0.229	0.240	0.310
fair coin	0.240	0.229	0.240	0.250
5% base rate	0.240	0.229	0.450	0.450

Appendix

Table 8: Top: the regimes of Figure 6 . Bottom: a calibrated, coherent predictor of “

A

?” and “

B

?” on the labels

\{A,B,C\}

, audited under the true structure, under a structure that forgets

C

(“claimed”), and with the doubtful relation dropped (“widened”). A wrong relation yields a positive certificate although

\Theta_{w}=0

; widening only lowers the bound.

AG News
		$\widehat{\Theta}_{w}$ (binned ECE)					negative controls		reference
System	certificate	15 eq.-w.	10 eq.-w.	20 eq.-w.	15 eq.-m.	Brier	$\hat{L}_{0}$	$\hat{L}_{\mathrm{marg}}$	$\hat{L}_{\mathrm{joint}}$
Jev	0.029	0.059	0.059	0.060	0.059	0.061	0.038	0.040	0.053
Eikos	0.095	0.128	0.127	0.128	0.129	0.096	0.106	0.106	0.113
Qwen3.5	0.185	0.217	0.216	0.217	0.217	0.190	0.194	0.201	0.232
DeBERTa	0.070	0.121	0.121	0.122	0.121	0.126	0.069	0.103	0.103

Appendix

Table 9: Labelled descriptive metrics and negative controls on the confirmatory streams of Figure 4 ; none of them is a certificate. “certificate” is the confirmatory anytime lower bound on

L_{H,G}

(Table 4 ).

\widehat{\Theta}_{w}

estimates

\Theta_{w}=\sum_{i}w_{i}\,\mathrm{ECE}_{1}(F_{i})

with the items’ labels: per coordinate, the binned ECE (15 equal-width bins, in bold; 10 and 20 equal-width and 15 equal-mass bins for sensitivity), aggregated with the bundle’s fixed weights. Brier is

\sum_{i}w_{i}

times the mean squared error of coordinate

i

. A binned ECE estimates

\Theta_{w}

; it is not

\Theta_{w}

. The estimation conditions were not preregistered. The negative controls are

L_{0}

and

L_{\mathrm{marg}}

computed on the empirical distribution of the stream without correction: they overstate, are never valid bounds and are not compared with the certificate.

\hat{L}_{\mathrm{joint}}

, the empirical mean of

\mathrm{dist}^{w}_{1}(F,\mathcal{C})

, is shown for reference; it concerns the joint calibration error, not

\Theta_{w}

	anytime at $T_{\max}$					fixed- $T$		first positive $t$
	sequential		split			split		split
System	$\mathcal{U}_{H,G}$	$\mathcal{U}_{0}$	$\mathcal{U}_{H,G}$	$\mathcal{U}_{0}$	difference	$\mathcal{U}_{H,G}$	$\mathcal{U}_{0}$	$\mathcal{U}_{H,G}$	$\mathcal{U}_{0}$
AG News
Jev	0.029	0.031	0.034	0.034	0	0	0	5,381	5,389
Eikos	0.095	0.098	0.102	0.102	0	0.0587	0.0587	5,150	5,150
Qwen3.5	0.185	0.186	0.191	0.190	+0.001	0.1473	0.1463	5,076	5,076

Appendix

Table 10: Stake class against learning method on the confirmatory streams of Figure 4 , every bound at

\alpha=0.05/7

. The two split arms learn on the first

\lfloor T_{\max}/2\rfloor

items and score the rest with the learnt stake; they differ only in

H

(the frozen value for

\mathcal{U}_{H,G}

0

for

\mathcal{U}_{0}

), with the same bundle, weights, nodes, betting rule and arrival order. “difference” is the

\mathcal{U}_{H,G}

split bound minus the

\mathcal{U}_{0}

split bound; “first positive

t

” is the budget at which a split arm’s anytime bound first turned positive. The sequential columns and the

\mathcal{U}_{H,G}

split arm repeat Table 4 ; recomputed by the same code as the

\mathcal{U}_{0}

split arm, the

\mathcal{U}_{H,G}

split arm was reproduced exactly. The

\mathcal{U}_{0}

split arm is post hoc and not preregistered.

		anytime bound		plug-in $L_{H,G}$		Choice accuracy		shift
System	Control	main	control	main	control	main	control	all	Choice	$(1+H)\,\cdot$ all	noise floor
AG News ( $H=2$ , $G=6$ )
Jev	option names	0.000	0.000	0.0397	0.0398	0.845	0.855	0.0060	0.0017	0.0181	0.0158
Eikos	option names	0.006	0.007	0.1063	0.1063	0.823	0.840	0.0037	0.0037	0.0111	0
Jev	option order	0.000	0.000	0.0397	0.0394	0.845	0.823	0.0093	0.0052	0.0280	0.0158
Eikos	option order	0.006	0.006	0.1063	0.1063	0.823	0.835	0.0046	0.0046	0.0137	0

Appendix

Table 11: Preregistered controls on the 200 exploration items of each data set, each against the main condition it changes in one factor. “anytime bound” is the lower bound on

L_{H,G}

computed as in the pilot, at

\alpha=0.05

and the selected

H

and

G

, on a grid of

10^{-3}

; “plug-in

L_{H,G}

” is the uncorrected empirical value, never a bound; “Choice accuracy” is the accuracy of the choice query, computed with the items’ labels, ties for the largest probability counted fractionally. “shift” is the item-matched weighted

\ell_{1}

difference of the answers, over all components and over the choice query’s alone (at the bundle’s weights);

(1+H)

times the shift over all components bounds how far apart the plug-in values of

L_{H}

and

L_{H,G}

under the two conditions can be. The noise floor is

(1+H)

times the largest

W_{1,w}

between the empirical laws of two of the system’s five main-condition calls; it is

0

for the local systems, whose calls return identical answers. A

0

is exactly zero;

0.0000

is a value rounded to zero. None of these values is a confirmatory certificate. The main conditions’ anytime bounds and plug-in values and the noise floors are the pilot’s; the accuracy of the choice query, under both conditions, and the shifts were computed with the comparison.

Data set	Repository at revision	pool	$K$	item text	seed	exploration	confirmation
AG News	fancyzhx/ag_news@eb185aa	120,000	4	{text}	20261001	200	10,000
DBpedia-14	fancyzhx/dbpedia_14@9abd46c	560,000	14	{title}\n{content}	20261001	200	10,000

Appendix

Table 12: The data sets and the exploration/confirmation split. “item text” is how a row becomes the text the predictors see, unnormalised ( \n is a line break); the pool is the whole train split, in file order, and the seed fixes its permutation.

Role	Question	Components	Weight each
AG News ( $K=4$ , $m=12$ )
Choice	What is the topic of this news article?	4	$1/12$
Atom	Is the topic of this news article $\langle$ label $\rangle$ ?	4	$1/12$
Complement	Is the topic of this news article something other than $\langle$ label $\rangle$ ?	4	$1/12$
Every question ends with: Every article has exactly one of these topics: $\langle$ labels $\rangle$ .
For example: Is the topic of this news article world news? Every article has exactly one of these topics: 'world news', 'sports', 'business' and 'science and technology'.

Appendix

Table 13: The main bundles.

\langle

label

\rangle

is a label’s description (Table 14 );

\langle

labels

\rangle

lists descriptions, each in straight single quotes, separated by commas and with “and” before the last one, or “or” in a union, which lists its member labels. The choice query offers every label as an option, named by a neutral letter in the main bundle and described by the label’s description; how the options are shown is the predictor’s readout (Appendix I.3 ). The weights sum to

1

over the

m

components.

		Option name
Label	Description	main	reversed	named	Union
AG News
World	world news	A	D	World
Sports	sports	B	C	Sports
Business	business	C	B	Business
Sci/Tech	science and technology	D	A	Sci/Tech

Appendix

Table 14: The labels in the order of the label space, the description that fills

\langle

label

\rangle

, the name of each label’s option in the choice query of the main bundle and of the two option controls, and, on DBpedia-14, the union a label belongs to, numbered as listed below the labels. AG News has no union.

Data set	$H$	$G$	Eikos	Qwen3.5	mean	on the grid	selected
AG News	1	6	0.004	0.113	0.02925	0.029
	1	11	0.000	0.104	0.026	0.026
	2	6	0.006	0.117	0.03075	0.031	$\checkmark$
	2	11	0.000	0.111	0.02775	0.028
	4	6	0.006	0.118	0.031	0.031
	4	11	0.000	0.113	0.02825	0.028

Appendix

Table 15: Selection of

H

and

G

on the pilot. Each system’s anytime lower bound (Proposition 17 ,

\alpha=0.05

) of

\mathcal{U}_{H,G}

learnt sequentially,

\eta=\sqrt{G/200}

, on the 200 exploration items of its main condition, for every candidate pair, with the mean over the four systems, exact and on the grid of

10^{-3}

(halves rounded up). The selected pair has the largest mean on the grid, a tie going to the smaller

H

, then to the smaller

G

. A larger

H

bounds a different

\Gamma_{H}

, so a larger mean does not mean a better configuration.

				noise floor
Data set	System	answered	s/item	$\max W_{1,w}$	$(1+H)\,\cdot$	identical	$\max\lvert\Delta\rvert$	bound	plug-in	threshold	step	$T_{\max}$
AG News ( $H=2$ , $G=6$ )
	Jev	200/200	0.21	0.0053	0.0158	0.05	0.2100	0.000	0.0397	0.013	(3)	10,000
	Eikos	200/200	9.97	0	0	1.00	0	0.006	0.1063	0.013	(2)	10,000
	Qwen3.5	200/200	5.07	0	0	1.00	0	0.117	0.2002	0.013	(2)	10,000
	DeBERTa	200/200	0.66	0	0	1.00	0	0.000	0.0819	0.013	(3)	10,000

Appendix

Table 16: Gate B on the pilot, at the selected

(H,G)

. “s/item” is the mean time of asking an item in the first call under the main condition (repeat index

0

), model loading excluded. The noise floor is

(1+H)

times the largest

W_{1,w}

between the empirical laws of two of the system’s five main-condition calls (Theorem 12 and Lemma 14 : it bounds how far the empirical

L_{H}

and

L_{H,G}

moved between the calls); “identical” is the share of items answered identically in all five, and

\max\lvert\Delta\rvert

the largest difference of one component of one item between two calls. “bound” is the anytime bound of Table 15 ; “plug-in” is the uncorrected empirical

L_{H,G}

, an effect size and never a bound; the threshold is that of the system’s

T_{\max}

. “step” is the Gate B step that decided; steps (2) and (3) send a system on to the main collection, and step (4) does not. A

0

is exactly zero.

AG News ( $H=2$ )
	$W_{1,w}$ between repeats $r$ and $s$
System	0–1	0–2	0–3	0–4	1–2	1–3	1–4	2–3	2–4	3–4	max	$(1+H)\,\cdot$ max
Jev	0.0048	0.0049	0.0051	0.0048	0.0048	0.0049	0.0053	0.0049	0.0051	0.0052	0.0053	0.0158
Eikos	0	0	0	0	0	0	0	0	0	0	0	0
Qwen3.5	0	0	0	0	0	0	0	0	0	0	0	0
DeBERTa	0	0	0	0	0	0	0	0	0	0	0	0

Appendix

Table 17: The noise floor’s distances:

W_{1,w}

between the empirical laws of every pair of a system’s five calls under the main condition (repeat indices

0

4

) on the 200 exploration items, as Part A lists them among the reported values. “max” and “

(1+H)\,\cdot

max” are the pilot’s noise floor of Table 16 , the value Gate B used. The pilot recorded only the maximum; all ten pairs were computed again after the main collection (Appendix J ). A

0

is exactly zero.

Data set	System	$T_{\max}$	settled	stream	failed	defects	interrupted	safe stops	s/item	sessions
AG News	Jev	10,000	10,000	10,000	0	0	0	2	0.39	10-05 to 10-06
	Eikos	10,000	10,000	10,000	0	0	0	0	8.52	10-07 to 10-09
	Qwen3.5	10,000	10,000	10,000	0	0	0	0	5.06	10-07 to 10-07
	DeBERTa	10,000	10,000	10,000	0	0	0	0	0.75	10-06 to 10-06
DBpedia-14	Eikos	2,000	2,000	2,000	0	0	0	0	40.45	10-10 to 10-11
	Qwen3.5	5,000	5,000	5,000	0	0	0	0	32.07	10-09 to 10-10

Appendix

Table 18: The main collection. “settled” counts the items whose outcome was logged and “stream” those before the first failure or defect, which the certificates use; “interrupted” counts items settled from the records after the process had stopped; a safe stop (here, an item whose answer went missing) leaves the item unsettled, to be asked again when the collection resumes. “s/item” is the mean time of asking an item; “sessions” gives the month and day (2026, UTC) on which the collection’s first session started and its last one ended. Jev on DBpedia-14 reached step (4) of Gate B (Table 16 ) and was not collected.

Part frozen	Tag at commit	Round	collections	settled
A (pre-pilot)	v0.12.0 at 4921775	pilot	52	10,400
OSF registration https://osf.io/dj3ut
SHA-256 of PROTOCOL.md at the tag: 90c316be6ab55501e6d8af79da9f7102b59e689726b8b5919563b203e568b3ab
Round opened 2026-10-01 19:07; records 2026-10-01 19:07 to 2026-10-04 23:13; Gate B computed 2026-10-05 00:24 (UTC)
B (pre-confirmation)	v0.16.0 at cfd9a27	main collection	7	57,000
OSF registration https://osf.io/d5jr6

Appendix

Table 19: The two registered rounds. Each part of the protocol was frozen at the commit its tag names and registered on OSF; the round ran from that commit. “collections” counts the collections the round ran and “settled” the items they settled, an item counted once per collection (the pilot asked each of its 400 exploration items in the 26 collections of its data set). Times are to the minute.

Module	Figures and tables	render reads	computed
fig1	Figure 1 , Table 1	results/fig1/	synthetic, from the code ( compute )
fig2	Figure 2 , Table 2	results/fig2/	synthetic, from the code ( compute )
fig3	Figure 3 , Table 3	results/fig3/	synthetic, from the code ( compute )
fig4	Figure 4 , Table 4	results/confirm/v0.16.0/	confirm certify , once, from the response ledger
power	Figure 5 , Table 6	results/power/	synthetic, from the code ( compute )
coverage	Table 7	results/coverage/	synthetic, from the code ( compute )

Appendix

Table 20: How the generated figures and tables are produced. The modules belong to the package coherence_calibration.experiments . A module’s render step reads the committed files listed, and “computed” says how those were computed: by the module’s own compute step where it says so, otherwise once, by the step named, which ran only from its round’s tag. Files under results/ are committed; the response ledger is not.

Probability Calibration

Probability calibration measures and improves whether predicted probabilities match observed outcome frequencies, supporting reliable decisions under uncertainty.

See all 134 papers

Oct 13, 2026

Recalibrating Many Questions with Few Labels: Minimax Rates and Identification Limits

Many related prediction problems, which we call questions, come with few labels each. What information can be shared across them, and what is actually necessary for good decisions? We answer both for recalibration, motivated by decision models such as Jev (TypeSafe AI, 2026) that return a probability for any yes/no question a developer writes. A predictor reports a probability

P

with output law

μ

m

questions, each with

n

labels and a nondecreasing map

g_j(p)=\mathbb{P}(Y=1\mid P=p)

; the compound risk is the excess Brier score over the true maps, averaged over questions. Over maps within sup-distance

τ\le 1

and total variation

V\le 2

of one nondecreasing shape (fixed effects), the minimax compound risk is

(mn)^{-2/3}+\min\{τ^2,1/n\}+\min\{τ^2,V^2,(V/n)^{2/3}\}

up to universal constants, for every non-atomic

μ

. A pooled isotonic fit with a per-question correction sized once, on held-out labels of all questions, attains it without knowing

τ

V

μ

(

n\ge 2

m\ge 3

); tuning each question on its own labels loses a factor of order

\min\{n,\log(m^2/n)\}

when the maps differ by constants and

m^2/n

is large. When the maps are drawn from an unknown prior (random effects), other questions need not identify the Bayes rule. The loss, the identification regret, is at most

2^{-n}

once the law of a question's data is known, extending a bound of Skibinsky (1968) for coins, and in the worst case at least

β_\star^{-n}/(72\sqrt{n})

at every finite

m

, where

β_\star\approx 3.583

. Beyond a few labels per question, what cannot be learned is nearly unnecessary. For constant maps (an uninformative predictor) and constant-map priors, the exponential base is exactly

β_\star

, not the 4 suggested by the moment range; for monotone maps it lies in

[2,β_\star]

Xiaoyu Li, Chiwun Yang

University of New South Wales. · City University of Hong Kong.

Probability Calibration Minimax Estimation Multi-Task Learning

Ask PDF

Jun 30, 2026

Verifiable Rewards for Calibrated Probabilistic Forecasting

Reinforcement learning with verifiable rewards can in principle train calibrated probabilistic forecasters, since a proper scoring rule such as the Brier score is computed from outcomes alone and is minimized in expectation by the true probability. In practice it degrades calibration, and existing remedies address epistemic uncertainty, where a model's confidence accompanies a verifiably correct or incorrect answer. We study aleatoric forecasting, where the forecast itself is the output and the label is one stochastic outcome, taking NFL in-game win probability as a testbed with the betting market as a reference. Rewarding the realized per-play outcome fails, because the single outcome is a noisy target and the policy gradient corrupts the chain of thought. We introduce a verifiable, label-free reward, a state-conditioned empirical win rate estimated from past outcomes, that removes the label noise, and we keep the gradient off the reasoning, by direct prediction or a gradient mask, so it cannot be corrupted. Trained with this reward alone, without human labels or supervised fine-tuning, a 7B model reaches the calibration of the betting market by direct prediction and is better calibrated than a zero-shot frontier model. That frontier model and a tabular estimator reach the same Brier score as this model, identifying the market's small remaining edge as live in-game information beyond their shared inputs. Masking the gradient, rather than dropping the chain of thought, preserves reasoning from which the forecast follows, which ordinary chain-of-thought training corrupts.

Sadanand Singh, Allam Reddy, Manan Chopra

Cascade Research

Probability Calibration RL for Language Model Reasoning Probabilistic Forecasting Reinforcement Learning with Verifiable Rewards Sports Outcome Prediction Reward Design for RL

Ask PDF

Aug 21, 2026

Truthful Calibration Measures for Sequential Prediction

Calibration requires probabilistic reports to be conditionally unbiased and reliably interpretable as probabilities. A calibration measure assigns numerical error to miscalibrated reports. Haghtalab et al. (2024) proposed an approximately truthful calibration measure for online prediction, leaving open whether exact truthfulness is compatible with completeness and soundness. We resolve this question negatively for sequential binary prediction: exact truthfulness is incompatible with completeness and soundness, even for independent outcomes. We then show that this impossibility is specific to exact truthfulness. On the positive side, we give two general reductions that transform a base calibration measure satisfying a uniform completeness condition into additively and multiplicatively approximately truthful measures, respectively. Applying the multiplicative reduction, for every

0<\varepsilon<1

we construct a sound and complete calibration measure that is

1+\exp\left(-Ω(T^{1-\varepsilon})\right)

-multiplicatively truthful. This improves the approximate truthfulness guarantee of Haghtalab et al. (2024). We also prove a general transfer theorem showing that the uniform completeness of one calibration measure transfers to another when they are quantitatively related. Using this result, we show that every consistent calibration measure (Blasiok et al., 2023) satisfies uniform completeness and can serve as the base calibration measure to achieve

1 + o(1)

multiplicative truthfulness.

Anagha Gokul, Jason Hartline, Lunjia Hu +2

Northeastern University. Khoury College of Computer Sciences. · Northwestern University. Department of Computer Science. · Microsoft Research. Microsoft Research New England.

Online Calibration Sequence Prediction Probability Calibration Model Calibration

Ask PDF

Abstract

Probability Calibration

Explore similar work

Recalibrating Many Questions with Few Labels: Minimax Rates and Identification Limits

Verifiable Rewards for Calibrated Probabilistic Forecasting

Truthful Calibration Measures for Sequential Prediction