Confidence Is Not a Promise: A Reference Audit of Typed Decision Models

Pedro Carvalho Brom

cs.LGOct 26, 2026

Confidence Is Not a Promise: A Reference Audit of Typed Decision Models

Authors: Pedro Carvalho Brom

Organizations: Department of Mathematics, Federal Institute of Brasília, Brasília, Brazil

Abstract

Typed decision models return a probability distribution over options supplied by the caller, and software tends to act on that number as if it were a guarantee. We measure what it licenses, comparing the open-weight Laya, in five checkpoints, with Jev 1.13, available only through an API. A probability licenses a statement about groups of decisions, not about one decision, and a gate that promises an accuracy among the cases it decides alone keeps that promise only with a lower bound checked on data like its traffic. Across 190 exchangeable evaluations, a threshold chosen by point estimate breaks the promise in 147, Learn then Test keeps it in all of them, and a Beta lower-bound rule decides more items on average than Learn then Test in 128 and fewer in none, without its guarantee. Calibrated in English, Jev keeps a promise of 95% in 8 of 51 locales. Ordinary typing noise breaks a promise calibrated on clean text, and, in exploratory runs, training with noise raises the noisy accuracy with no loss on clean text. In four further domains, a fine-tuned Laya needs from 250 to 8,000 labels to be non-inferior to Jev, and does not get there on LIAR2. Under drift, the preregistered claim for a Bayesian retraining loop is not supported: the loop avoids needless retraining where the change does not degrade the model, at a cost equivalent to recalibration alone, and re-qualifying the threshold in every window keeps the promise more often where it does. On the monthly series of M3, neither decider predicts the direction of the next value better than a constant answer. Confidence separates right from wrong answers with an AUROC from 0.80 to 0.96, but it does not select faithful explanations, and exact coalition games of single decisions show what an explanation can still say through an API that rounds to two decimals. We release the frozen answers with a datasheet, a promise check and the code.

Figures & tables

Claim	Status	Evidence	Where
A point-estimate threshold breaks the promise in 147 of 190 evaluations, Learn then Test in none	Exploratory	two-sided Clopper–Pearson interval per evaluation, 2,000 to 10,000 draws	Section 6.3
The ascending Beta rule decides more items on average than Learn then Test in 128 of 190	Exploratory	means over draws, and on the static front no gain remains after Holm over 24 comparisons	Section 6.3
Calibrated in English, Jev keeps a promise of 95% in 8 of 51 locales	Registered	per-locale violation rate on report items	Section 8
Typing noise breaks a promise calibrated on clean text, and training with noise helps	Registered	paired accuracy, 95% bootstrap intervals, margin of 2 points	Section 7
A fine-tuned Laya needs 250 to 8,000 labels to be non-inferior to Jev, and no tested count suffices on LIAR2	Registered	paired non-inferiority over a grid of label counts	Section 9
The retraining loop does not meet the preregistered criterion	Confirmatory	paired cost against the best comparator in three drifts	Section 10

Table 1: The claims of the abstract, their status and their evidence. Confirmatory: fixed in a preregistration. Registered: analysis written before the runs and exploratory. Exploratory: decided after seeing data or not registered. Descriptive: a measurement without a test.

Name in the text	In tables	Base	Fine-tuning
Jev 1.13 (and 1.14)	Jev	not published	by the vendor
Laya multilingual	Laya, released	mmBERT-base	none
Laya English	Laya English, root	ModernBERT-large	none
Laya typed-decisions	Laya typed-dec.	ModernBERT-large	by the developer of Laya
Laya multilingual, fine-tuned	Laya-ft	mmBERT-base	MASSIVE pt-PT, 11,274 items
Laya typed-decisions, fine-tuned	Laya typed-dec., ft	ModernBERT-large	MASSIVE pt-PT, 11,274 items

Table 2: Deciders and the names used for them. Base: the pre-trained encoder of each Laya checkpoint. Laya-ft is short for the fine-tuned multilingual checkpoint, and the battery and the time-series front refit it on their own data with the drift recipe.

Symbol	Meaning
$\alpha$	target of the promise, the accuracy among accepted items (0.95 or 0.99 on MASSIVE)
$\theta(\tau)$ , $\phi(\tau)$	accuracy among accepted items and coverage at threshold $\tau$
$q$	credibility level of the Beta lower bound, its $q$ -quantile (0.05 unless stated)
$\delta$	level of Learn then Test, the probability over the calibration sample that its threshold breaks the promise (0.05)
tolerance	largest acceptable violation rate of a rule against the plug-in population (0.05)
$\varepsilon$	margin of non-inferiority and equivalence ( Section 4.7 ), 2 points for accuracy

Table 3: Symbols and levels. Several of them take the values 0.05 or 95%, with different roles.

Lang.	System	Accuracy [95% CI]	Macro-F1	$\Delta$ vs Jev, pp [95% CI]	$p$
pt	Jev 1.13	0.947 [0.931, 0.959]	0.945
	Laya multilingual	0.731 [0.703, 0.758]	0.709	$-21.6$ [ $-24.4$ , $-18.8$ ]	$<10^{-40}$
	Laya English	0.736 [0.708, 0.762]	0.690	$-21.1$ [ $-23.9$ , $-18.3$ ]	$<10^{-40}$
	Laya typed-decisions	0.739 [0.711, 0.765]	0.687	$-20.8$ [ $-23.6$ , $-18.0$ ]	$<10^{-40}$
	Laya multilingual, FT	0.973 [0.961, 0.981]	0.953	$+2.6$ [ $+1.1$ , $+4.1$ ]	$8.6\times 10^{-4}$
	Laya typed-decisions, FT	0.955 [0.940, 0.966]	0.928	$+0.8$ [ $-0.8$ , $+2.4$ ]	0.40

Table 4: Accuracy on the 1,000 reported utterances, macro-F1 over intents, and the paired difference to Jev 1.13 (exact McNemar). FT: fine-tuned on MASSIVE pt-PT. Best per language in bold.

Figure 1: Accuracy of the fine-tuned Laya multilingual checkpoint against the number of labelled pt-PT utterances (three seeds per size, mean in the line), with Jev 1.13 without labels as reference.

System	Accuracy	Labels	Training	Inference	US$ per 1,000	State leaves
Jev 1.13	0.947	0	none	API, not measured	0.016	yes
Laya multilingual	0.731	0	none	12.7 ms, CPU	local	no
Laya multilingual, 4,000 labels	0.947	4,000	17 s, one GPU	as above	local	no
Laya multilingual, fine-tuned	0.973	11,274	75 to 105 s, one GPU	11.5 ms, CPU	local	no

Table 5: What each route costs, MASSIVE pt-PT, reported half. Price per 1,000 decisions from the mean of 384 input tokens over 1,200 calls, at US$0.042 per million input tokens with output tokens free, as stated by the vendor in October 2026. Inference is CPU time per item for Laya. Jev’s latency was not measured, and a local CPU time is not comparable with an API round trip. Training on one RTX 5090 with three seeds. The full-partition times were measured while the GPU was shared with other runs. In bold, the best of each column where an order exists: the highest accuracy, the fewest labels and the state that does not leave the machine.

Lang.	System	$T$	ECE (as ret.)	ECE ( $T$ )	Brier	NLL	AUROC [95% CI]	ECE scen. med./worst
pt	Jev 1.13	–	0.020	–	0.082	0.264	0.905 [0.862, 0.943]	0.039 / 0.077
	Laya multilingual	1.77	0.110	0.045	0.370	0.764	0.825 [0.798, 0.852]	0.142 / 0.312
	Laya English	1.10	0.051	0.039	0.374	0.764	0.802 [0.773, 0.833]	0.127 / 0.236
	Laya typed-decisions	0.55	0.158	0.041	0.358	0.706	0.818 [0.790, 0.846]	0.115 / 0.226
	Laya multilingual, FT	1.28	0.018	0.012	0.045	0.095	0.928 [0.890, 0.962]	0.024 / 0.083
	Laya typed-dec., FT	0.83	0.021	0.013	0.064	0.130	0.958 [0.936, 0.977]	0.037 / 0.057

Table 6: Calibration and discrimination on the reported half. ECE with 15 bins as returned and after temperature, Brier score and NLL after temperature, AUROC of confidence as a detector of a correct answer with bootstrap 95% interval, and ECE per MASSIVE scenario (median and worst over scenarios with at least 30 items). Best per language and column in bold (lowest, except AUROC).

Figure 2: Top: reliability diagrams (10 bins with at least 5 items), the diagonal is perfect calibration. Bottom: ROC of confidence as a detector of a correct answer. Arrows point from worse to better.

Figure 3: Cost per item against the gate threshold for a cost ratio of 10:1 (a wrong automatic action costs ten reviews), MASSIVE pt-PT. The dashed line is the Bayes threshold

\tau^{*}=0.9

Target	Lang.	System	Point	Beta, ascending	Beta, fixed seq.	LTT	SGR
99%	pt	Jev 1.13	0.386 (0.394) / 848	0.014 (0.017) / 400	0.014 (0.016) / 338	0.014 (0.016) / 338	0.002 (0.003) / 156
99%	pt	Laya multilingual, FT	0.408 (0.416) / 953	0.020 (0.023) / 879	0.019 (0.022) / 851	0.019 (0.022) / 851	0.002 (0.003) / 776
99%	pt	Laya typed-dec., FT	0.504 (0.513) / 911	0.031 (0.034) / 831	0.019 (0.022) / 818	0.019 (0.021) / 817	0.001 (0.001) / 739
99%	en	Jev 1.13	0.487 (0.495) / 880	0.021 (0.024) / 448	0.017 (0.019) / 447	0.017 (0.019) / 447	0.004 (0.005) / 128
99%	en	Laya typed-decisions	0.534 (0.542) / 631	0.034 (0.037) / 344	0.012 (0.014) / 216	0.012 (0.014) / 214	0.000 (0.000) / 1
99%	en	Laya English	0.533 (0.541) / 503	0.026 (0.028) / 88	0.012 (0.013) / 56	0.011 (0.013) / 56	0.002 (0.003) / 1

Table 7: The gate against the plug-in population,

n_{\text{cal}}=1{,}000

, 10,000 calibration draws. Each cell: violation rate (one-sided 95% upper bound) / decisions per 1,000 items. Beta rules at

q=0.05

, Learn then Test and SGR at

\delta=0.05

. In bold, per case, the rule with the most decisions among those whose upper bound is at most 0.05.

Figure 4: Violation rate of the promise of 99% against the plug-in population, for the five rules of Table 7 , with the one-sided 95% upper bound as error bar. The dashed line is the tolerance of 0.05, and the shaded band below it is where the promise is kept. The annotations mark the two cases in which the ascending Beta rule decides more items than Learn then Test with a 95% interval above zero that includes the item-level uncertainty ( Section 4.7 ).

			Beta, ascending $-$ LTT		Beta, fixed seq. $-$ LTT
Target	Lang.	System	decisions	violation	decisions	violation
99%	pt	Jev 1.13	$+61.8$ (46.5)	$+0.001$ (0.002)	$+0.2$ (0.6)	$+0.000$ (0.000)
99%	pt	Laya multilingual, FT	$+28.4$ (11.1)	$+0.001$ (0.003)	$+0.4$ (0.2)	$+0.000$ (0.000)
99%	pt	Laya typed-dec., FT	$+13.9$ (8.0)	$+0.012$ (0.004)	$+0.3$ (0.2)	$+0.001$ (0.000)
99%	en	Jev 1.13	$+1.3$ (4.1)	$+0.004$ (0.003)	$+0.4$ (0.8)	$+0.000$ (0.000)
99%	en	Laya typed-decisions	$+129.9$ (52.2)	$+0.022$ (0.006)	$+1.3$ (1.5)	$+0.001$ (0.000)

Table 8: Paired differences against Learn then Test on the same calibration draws, plug-in population,

n_{\text{cal}}=1{,}000

: decisions per 1,000 items and violation rate, with the item-level standard deviation from 200 bootstrap populations in parentheses. In bold, coverage differences larger than twice their item-level standard deviation.

Figure 5: What is decided alone against what is right, MASSIVE pt-PT, target 99%, split-half evaluation. The line is every threshold on the reported half, and each point is one of 1,000 splits. Splits in which a rule qualifies no threshold decide nothing and are not shown: 608 of 1,000 for the Beta rule with Jev, none for the fine-tuned Laya. Shaded bands: the coverage (5th to 95th percentile) of the Beta-rule splits that kept the promise on the reported half (green) and of those that broke it (red). For the fine-tuned Laya the rule keeps the promise between coverages of about 0.74 and 0.93 and breaks it above, where the curve flattens near the target, the region that Proposition 4 and the measured disagreement of the scans point to.

		Kept			Beta		Mean violation			Decisions per 1,000
Decider	Cases	point	Beta	LTT	undet.	broken	point	Beta	LTT	point	Beta	LTT
Jev 1.13	18	5	18	18	0	0	0.290	0.015	0.012	659	494	478
Laya as released	82	5	78	82	4	0	0.477	0.027	0.010	277	162	89
Laya fine-tuned	90	32	87	90	2	1	0.321	0.024	0.017	737	686	652

Table 9: The three rules over the 190 exchangeable evaluations of the article, by kind of decider, each case with its tightest measurement (10,000 draws where 2,000 left it undetermined). Kept: upper bound of the violation rate at most 0.05. Undetermined and broken: the Beta cases with the interval straddling 0.05 or above it. Mean violation rate and mean decisions per 1,000 over the cases. In bold, per row, the rule that keeps the promise most often, the lowest mean violation and, between the two bounded rules, the one that decides more.

		Accuracy				Violation at 95%
System	Lang.	clean	typo 10%	mixed 10%	mixed 20%	clean	mixed 10%
Jev 1.13	pt	0.950	0.939	0.939	0.934	0.00	0.15
Laya multilingual	pt	0.732	0.686	0.646	0.636	0.18	0.98
Laya multilingual, fine-tuned	pt	0.973	0.944	0.939	0.930	0.00	1.00
Jev 1.13	en	0.961	0.947	0.948	0.943	0.00	0.75
Laya multilingual	en	0.816	0.752	0.744	0.726	0.01	0.18

Table 10: Natural noise, report items. Accuracy on clean text and under typos at 10% of the words and the mixed profile at 10% and 20%, and the violation rate of the promise of 95% by the ascending Beta rule, with thresholds chosen on 1,000 calibration samples of clean items and applied to the clean and to the mixed-10% report items. In bold, per language, the highest accuracy and the lowest violation rate of each column.

Figure 6: Greedy typo attack on Laya, 200 report items per language. Left: share of decisions not yet changed after each number of typos, with 95% intervals, for the released and the fine-tuned multilingual checkpoint. Fine-tuning delays the fall but does not prevent it. Right: for the fine-tuned checkpoint, the confidence of each clean decision, in log-odds, grouped by the number of typos that changed it. The groups overlap across the whole range, so a confidence of 0.99 or 0.9999 (dashed lines) does not tell which decisions will survive. Jev was not attacked, only measured under random noise.

Lang.	Clean utterance	Attacked utterance	Clean decision	Decision after the typos
pt	existe um tosquiador só de gatos na cidade	exiate um tosquiador só de gatos na cidade	general_quirky (0.9993)	play_game (0.92)
en	open temperature	open temprature	weather_query (0.9993)	play_music (0.33)
pt	eliminar uma lista	eiminar uma lisha	lists_remove (0.99997)	iot_hue_lightup (0.90)

Table 11: Typos that change a decision of the fine-tuned Laya. The examples are not chosen by hand: in each language, the item of median clean confidence among those changed by one typo, and in Portuguese also among those changed by two, as fixed in the implementation notes before they were seen. The gold intent is the clean decision in all three.

Figure 7: The promise of 95% across the 51 locales of MASSIVE other than en-US, report items. Left: violation rate of the Beta rule calibrated on the en-US calibration items, against the accuracy lost relative to en-US on the same items. The green band is where the promise travels (violation at most 0.05). Jev keeps it only where the loss stays below about one point. Right: locales where the promise holds (upper bound of the violation rate at most 0.05), for Jev calibrated in en-US and for Laya calibrated on the calibration items of each locale, with the median decisions per 1,000 over the bars. Calibrating in the locale recovers the promise for Laya with the bounded rules, at the price of fewer decisions.

		Accuracy			Labels for			Promise kept			Decisions		Typos
Domain	Opt.	Jev	Laya	Laya-ft	NI	sup.	ECE Jev	point	Beta	LTT	Beta	LTT
MASSIVE pt (ref.)	5	0.947	0.732	0.960	4,000	none	0.020	7	8	10	996	996	0 of 3
News Category	5	0.817	0.585	0.831	8,000	none	0.076	0	6	9	517	331	0 of 3
Financial tweets	3	0.728	0.642	0.926	250	250	0.171	3	9	10	842	532	0 of 3
LIAR2	6	0.319	0.282	0.324	none	none	0.162	10	10	10	0	0	no decisions
Fake.Br (pt)	2	0.716	0.567	0.981	250	250	0.087	7	10	10	999	999	3 of 3

Table 12: The battery, report items. Columns, from the left: accuracy of Jev, of the released Laya and of the largest fine-tune (mean of three seeds), the labels from which the fine-tune is non-inferior (margin of 2 points) and superior to Jev, Jev’s ECE as returned, the cases in which each rule keeps the promise against the plug-in population (of 10, targets 0.90 and 0.95 for five deciders counting each seed) with the tolerance of Section 6.3 , the median decisions per 1,000 of the fine-tunes at 0.95, and the seeds in which the Beta promise of the fine-tune at 0.95, calibrated on clean text, survives typos at 10%. The first row is MASSIVE pt-PT measured with the same code and grid, with Jev’s earlier answers reused. In LIAR2 no rule accepts any item, so every promise is kept by abstention. In bold, per row, the highest accuracy, the rule that keeps the promise most often and the rule that decides most, where they differ.

Figure 8: Label curves of the battery. Accuracy of the fine-tuned Laya on the report items against the labelled items used (three seeds and their mean), with Jev without labels and the released Laya as reference lines. Gold where the fine-tune is more accurate than Jev with a 95% interval above zero, green where it is non-inferior with the margin of 2 points, red where it is neither.

	Cost per item				Windows below $\alpha$			Retrains
Arm	D1	D2	D3	control	D1	D2	D3	D1	D3	control
Author’s loop	0.764	0.402	0.825	0.433	0.86	0.00	0.93	0.9	1.2	0.0
Periodic ( $K=3{,}000$ )	0.720	0.466	0.806	0.482	0.83	0.26	0.62	3.0	3.0	3.0
DDM	0.843	0.448	0.838	0.500	0.91	0.08	0.65	2.2	2.1	2.0
ADWIN	0.795	0.427	0.883	0.475	0.88	0.03	0.96	0.9	0.0	0.0
Page–Hinkley	0.815	0.428	0.843	0.476	0.93	0.05	0.77	1.2	1.0	0.2

Table 13: Drift experiment, main run: mean over the 60 streams of each drift (30 replicates, abrupt and gradual) and the 30 control streams. Cost per item at

C_{e}/C_{r}=10

, share of post-change windows of 500 items whose true selective accuracy is below

\alpha=0.95

, and retrains per stream. In bold, the lowest cost and the fewest windows below

\alpha

per column. Retrains are not ranked, because under a drift that degrades the model fewer retrains is not better.

Method	Against	D1	D2	D3	control
Author’s loop	Periodic	–	C	–	C
Author’s loop	DDM	C	C	–	C
Author’s loop	ADWIN	–	NI	NI	–
Author’s loop	Page–Hinkley	T	T	–	T
Author’s loop	Recalibration only	–	E	–	E
Author’s loop	LTT per window	–	T	–	T

Table 14: Efficiency at equal promise in the drift experiment (exploratory, 64 comparisons, Section 4.7 ). Outcome: share of windows below

\alpha

(post-change windows, all windows on the control stream), margin 0.02. Resources: labels per 1,000 items, retrains and GPU seconds, paired over 60 streams per drift and 30 control streams. C: clean efficiency gain. T: trade-off, one resource saved and another resource or the error rate larger. E: equivalent, no resource saved. NI: non-inferior, no resource saved. –: non-inferiority not shown, because the method is worse or the interval is too wide. Clean gains, the best outcome of a comparison, in bold.

	S1, noise cycle		S2, language cycle		D1		D3
Arm	Cost	Below $\alpha$	Cost	Below $\alpha$	Cost	Below $\alpha$	Cost	Below $\alpha$
Recalibration only	0.614	0.51	0.407	0.00	0.742	0.57	0.781	0.38
Recalibration, $\lambda=0.80$	0.646	0.52	0.420	0.00	0.728	0.26	0.762	0.22
LTT per window	0.493	0.01	0.420	0.00	0.668	0.37	0.773	0.21
Author’s loop	0.578	0.45	0.408	0.00	0.764	0.86	0.825	0.93
Periodic	0.578	0.44	0.456	0.22	0.720	0.83	0.806	0.62

Table 15: The dynamic adapter on streams with a pattern and under the degrading drifts: mean cost per item and share of windows below

\alpha=0.95

(all windows on S1 and S2, post-change windows on D1 and D3), over 30 streams per cycle and 60 per drift. The seasonal variants were run on S1 and S2 only. In bold, the lowest value per column.

Decider	Accuracy	Macro-F1	Rise	Minus ETS	Top 100
ETS	0.673 $[0.643,0.701]$	0.673	0.50	–	0.88
Jev	0.540 $[0.514,0.568]$	0.510	0.23	$-0.133$ $[-0.168,-0.097]$	0.76
Laya as released	0.476 $[0.449,0.503]$	0.324	0.00	$-0.197$ $[-0.239,-0.154]$	0.56
Laya fine-tuned	0.522 $[0.495,0.551]$	0.343	1.00	$-0.151$ $[-0.187,-0.116]$	0.60

Table 16: Deciders on the direction of the next value, 150 M3 series at their last 12 origins (1,800 decisions, 52.2% of rises). Accuracy and macro-F1 with 95% intervals from a bootstrap over series, share of decisions that predict a rise, accuracy minus that of ETS on the same decisions, and accuracy of the 100 most confident decisions. Laya fine-tuned averages the probabilities of three seeds. In bold, the highest accuracy.

	Target 0.70		Target 0.80
Gate	Below target	Per 1,000	Below target	Per 1,000
Point, calibrated once	0.42	795	0.29	350
Learn then Test, calibrated once	0.33	763	0.17	311
Point, sliding	0.50	828	0.42	398
Beta rule, sliding	0.42	792	0.38	372
Learn then Test, sliding	0.42	790	0.17	184

Table 17: Gates on exponential smoothing over the 1,428 M3 series and the 24 test origins: share of origins whose accuracy among acted decisions is below the target, and decisions acted per 1,000. Sliding gates are calibrated on the 12 preceding origins. Per-bin gates accept a confidence bin when its predicted accuracy reaches the target. In bold, the lowest share below the target per column.

Figure 9: Deletion curves by occlusion, mean over the 1,000 reported utterances: words removed most important first (solid) against random order (dotted), with areas in parentheses. A faster fall means a more faithful explanation.

Lang.	Checkpoint	AUD, MoRF	AUD, random	gap	Spearman $(p_{0},\text{AUD})$
pt	Laya multilingual, FT	0.631	0.790	0.159	$+0.25$
pt	Laya multilingual	0.490	0.675	0.185	$-0.38$
en	Laya multilingual	0.514	0.693	0.179	$-0.13$
en	Laya English	0.555	0.723	0.169	$+0.09$

Table 18: Area under the deletion curve on the 1,000 reported utterances, words removed most important first (MoRF) and in random order, and the Spearman correlation between the probability of the decision

p_{0}

and its MoRF area. The gap is random minus MoRF area: a larger gap means the ranking points to the words the decision rests on, beyond what any removal does. The largest gap per language is in bold.

Question	Measured (criterion)	Useful when
Are words and pairs enough?	median order-two share 0.86 to 0.93 (passes, $\geq 0.6$ )	always, for inputs of up to eight units
Intervals for the effect of a word under rounding	coverage of $W_{i}$ 0.55 to 0.59 and of $E_{i}$ 0.64 to 0.72 , KernelSHAP 0.59 to 0.63 (fails, $L\geq 0.85$ )	not available through a two-decimal API, where rounding bounds alone certify a relevant word in 60% to 70% of Jev decisions and the most relevant one in 17% to 21%, and not needed with unrounded probabilities and short inputs
Can redundant evidence be detected?	F1 0.300 for the Bayesian model, 0.901 for the Shapley interaction index , 0.889 for HEDGE (fails for the Bayesian model)	with an interaction index or HEDGE, also through the API
Does confidence signal a faithful explanation?	positive slope in 4 of 4 Laya cases, small, Jev 1 of 4, planted intents at ranks 2, 3, 6 of 56 (passes)	no, faithfulness has to be measured per decision
Does the explanation depend on the options?	approximate invariance in 3 of 4 Laya and 8 of 8 Jev comparisons, 68% to 79% of items change decision or runner-up	only for the option set it was computed on
Which ranking deletes fastest?	exact Shapley and HEDGE first, occlusion last, in Laya and in Jev (Kendall 0.6 to 1.0)	exact Shapley or HEDGE, not occlusion

Table 19: Explanation front on the report items: what was asked, what was measured against the preregistered criterion, and when the answer is useful to a caller. Coverage is the share of exact values inside the 90% interval. Pt-ft is the fine-tuned multilingual checkpoint rounded to two decimals as a stand-in for Jev. Where a row compares methods, the best one is in bold. HEDGE is the hierarchical explanation method of Chen et al. [2020] . G1 to G5 are the criteria of the preregistration.

Control	What to measure	Measured in
Validation	a fixed schema for the state, a token count against the context budget, the language, and the format in which the content is sent, which is part of the input	Sections 7 and 8
Signals	a second, independent decider and the McNemar table of disagreements	Section 6
Logging	state, question, options, model version and probabilities of every call	the released package
Drift	a fixed probe set re-run over time, with accuracy, ECE and McNemar between windows, and the accuracy of each confidence bin tracked online	Section 10
Anomaly	shifts in option frequencies, cases with high $c_{\max}$ but low $c_{H}$ , and the decision on the empty state	Sections 6.2 and 11
Escalation	a gate with a lower-bound rule, Learn then Test where a guarantee is required	Section 6.3

Table 20: Seven operational controls turned into measurements, and where the article measures each.

Situation	What was measured	What to do	Where
Promising an accuracy among accepted decisions	the point estimate breaks the promise in 147 of 190 evaluations, Learn then Test keeps it in all, the ascending Beta rule in 183 with more decisions	Learn then Test where a guarantee is required, the ascending Beta rule where more automatic decisions matter more, never the point estimate	Section 6.3
Few calibration labels	a threshold needs at least $\lceil\ln q/\ln\alpha\rceil-1$ accepted items to qualify, 58 at 95%	collect at least that many accepted items, or let the gate abstain	Proposition 5
Noisier inputs than the calibration items	a few points of accuracy lost on typed text break the promise	calibrate on items that look like the traffic, noise included, and train with noise	Section 7
Several languages	calibrated in English, Jev keeps the promise in 8 of 51 locales, with a slack of 1.2 points	calibrate in each language, and read the slack as the loss the promise can absorb	Section 8
A new domain	labels for non-inferiority to Jev range from 250 to 8,000, Jev’s ECE from 0.076 to 0.171	measure the label curve and the calibration in the domain, and test what a fine-tune reads	Section 9
No rule decides anything	the selective ceiling lies below the target	add informative inputs to the state, ask a coarser question or answer with sets	Proposition 6

Table 21: What the measurements of the article say to a caller of a typed decider.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: When fine-tuning Laya pays off, for any priced API. Two conditions must hold. The fine-tune must be non-inferior to Jev in accuracy, which the paired test with a margin of 2 points establishes from 4,000 labels on (solid line). Below it, in the red band, no price of a label makes the fine-tune worthwhile. And its labels must cost less than the API spend they replace,

L\,c_{\ell}\leq B

with

B=N\,p

the cost of the

N

decisions served at a price

p

per decision. For a fixed

B

this boundary is a hyperbola, so the number of labels and the price of a label trade off non-linearly. The fine-tune pays off in green, between the solid line and a hyperbola, and the green deepens where it pays off already at a smaller API spend. The right axis gives, at the label counts of the test grid only, the measured difference in accuracy between the fine-tuned Laya and Jev, averaged over three seeds. The curves are labelled with the API spend avoided and, in parentheses, the number of Jev decisions it buys at the measured price. The dashed line marks the 11,274 labels at which Laya becomes more accurate than Jev, and in gold, above it and below a hyperbola, Laya beats Jev on both counts, more accurate and cheaper. Superiority was measured at 11,274 labels, the whole training pool, and the gold band above that count assumes that more labels do not lower the accuracy, which was not tested. Each hyperbola is a lower bound on the volume needed, because local inference is not free, and it does not price that accuracy gain, which would enlarge the region wherever an error has a cost.

		95%			99%			99.5%
Lang.	System	point	Beta	LTT	point	Beta	LTT	point	Beta	LTT
pt	Jev 1.13	0.50 / 995	0.10 / 976	0.10 / 976	0.50 / 859	0.03 / 314	0.03 / 251	0.46 / 536	0.02 / 15	0.02 / 15
pt	Laya multilingual	0.55 / 443	0.10 / 311	0.07 / 212	0.64 / 142	none	none	0.36 / 58	none	none
pt	Laya English	0.52 / 327	0.09 / 182	0.02 / 89	0.60 / 105	none	none	0.47 / 82	none	none
pt	Laya typed-decisions	0.52 / 397	0.09 / 279	0.05 / 226	0.47 / 160	none	none	0.49 / 139	none	none
pt	Laya multilingual, FT	0.00 / 999	0.00 / 999	0.00 / 999	0.49 / 955	0.07 / 883	0.06 / 846	0.48 / 898	0.04 / 709	0.04 / 709

Appendix

Table 22: Broken promises per split on the report half and decisions per 1,000, for the point rule, the ascending Beta rule (

q=0.05

) and Learn then Test (

\delta=0.05

), at targets of 95%, 99% and 99.5%, 1,000 splits. “none”: no qualifying threshold in any split, which keeps the promise by abstaining. In bold, per system and target, the rule with the fewest broken promises and, among ties, the most decisions.

		Accuracy			Violation, calibrated in en-US			Violation, in the locale
Locale	Script	Jev	Laya	Laya-ft	Jev	Laya	Laya-ft	Laya	Laya-ft
af-ZA	Latin	0.934	0.595	0.832	0.86	1.00	1.00	0.51	0.40
am-ET	Ethiopic	0.905	0.344	0.571	1.00	1.00	1.00	0.00	0.15
ar-SA	Arabic	0.917	0.637	0.878	0.98	1.00	1.00	0.01	0.08
az-AZ	Latin	0.946	0.624	0.844	0.73	1.00	1.00	0.16	0.25
bn-BD	Bengali	0.938	0.555	0.788	0.83	1.00	1.00	0.03	0.19

Appendix

Table 23: Languages without fine-tuning (and Laya fine-tuned in Portuguese), report items. Script: the Unicode script of most letters in the locale’s utterances. Accuracy and violation rate of the Beta rule at 95% (2,000 draws) calibrated in en-US or in the locale. Laya-ft is the multilingual checkpoint fine-tuned on pt-PT. In bold, per row, the highest accuracy and the lowest violation rate of each group, where they differ.

			Promise at 0.90			Promise at 0.95			Beta at 0.95 under		Empty
Decider	Acc.	ECE	point	Beta	LTT	point	Beta	LTT	typos	mixed	state
MASSIVE pt-PT (reference) , 5 options, fine-tunes on 8,000 labels
Jev 1.13	0.947	0.020	0.00/999	0.00/999	0.00/999	0.41/993	0.03/975	0.03/974	0.48/968	0.16/967	0.281
Laya	0.732	0.043	0.48/635	0.05/561	0.04/525	0.52/438	0.05/321	0.03/229	0.54/319	0.98/303	0.175
Laya-ft, seed 0	0.959	0.014	0.00/999	0.00/999	0.00/999	0.00/999	0.00/994	0.00/993	0.95/983	0.97/983	0.395
Laya-ft, seed 1	0.963	0.019	0.00/999	0.00/999	0.00/999	0.00/999	0.00/997	0.00/997	1.00/995	0.99/992	0.408

Appendix

Table 24: Report cards of the battery, report items. Accuracy, ECE (Jev as returned, Laya after temperature on the calibration half), violation rate over decisions per 1,000 of the point rule, the ascending Beta rule and Learn then Test at 0.90 and 0.95, the Beta promise at 0.95 calibrated on clean text and checked under typos and under the mixed profile at 10%, and accuracy with an empty state. In bold, within each block, the highest accuracy, the lowest ECE and, in each promise column, the decider that keeps the promise with the most decisions.

LLM Reliability

Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.

See all 789 papers

Sep 26, 2026

Typed Decision Models: An Early Evidence Audit and Evaluation Checklist

Typed decision models (TDMs) return probability distributions over caller-defined options without generating text. TypeSafe released Jev, a commercial typed decision model, on 15 September 2026, and a small body of evaluation and replication work appeared within days. We review 28 papers posted between 19 and 24 September and relate their findings to earlier work on label-probability classification, constrained decoding, reranking, calibration, and model cascades. In this early literature, the typed readout itself has not shown an independent accuracy advantage over comparable label-probability readouts. Jev's clearest gains are in latency and cost, while accuracy gaps remain on harder tasks. In practical deployments, confidence is often used to decide when to defer to a stronger model or a human. We use recurring weaknesses in these studies to derive a 14-item evaluation checklist for future TDM work. Because the evidence covers only the first nine days after the release of one hosted model, the review should be read as an early evidence map rather than a settled assessment of the model class.

Lijuan Tang, Yuemeng Zheng

Northeastern University

LLM Evaluation

Ask PDF

Oct 1, 2026

Labels Override Definitions in Jev-Style Typed Decision Models

A typed decision model answers a fixed question about an input by returning a probability for each of several caller-defined options. Each option carries a short label and a written definition, which is where a developer states the rule the model should apply. Jev introduced this interface for routing, moderation and triage, open implementations followed, and the same operation occurs whenever a language model is used as a classifier by scoring label strings. We study the open implementations, whose weights we can inspect and patch, and ask whether the probability follows the definitions or the labels. A preference for the label we call option-label bias. Across four open-weight typed decision models, three ways of reading an answer from a Qwen2.5 backbone, eleven classification tasks and PolicyBench, a synthetic routing suite we introduce in which the rule appears only in the definitions, the answer is mostly the labels. Deleting every definition leaves accuracy unchanged (laya-td: 0.8559 against 0.8487), although those definitions support 0.7971 on their own, and renaming the options to A and B raises accuracy by +0.1511 [+0.1377, +0.1646]. One system, von, is unaffected, and the two code bases differ in one expression: laya writes each option as "{label}: {definition}", while von writes only the definition. Changing that expression in both directions, with no weight changed, makes all three laya checkpoints exactly invariant (+0.0000 [+0.0000, +0.0000]) and creates the effect in von, whose accuracy falls from 0.8511 to 0.2281 when a label contradicts its definition. Earlier work attributed this failure to the constrained decision head these models use in place of a text decoder; our results locate it in the prompt rendering. We give a two-call test that tells a practitioner which case applies to their model, and measure what four mitigations are worth.

Seyedarmin Azizi, Erfan Baghaei Potraghloo, Massoud Pedram

University of Southern California (USC), USA

Language Model Bias Evaluation Prompt Sensitivity

Ask PDF

Sep 27, 2026

Laya as a Typed Probabilistic Assessor: An Independent Reproduction and a Preregistered Study of Calibration and Selective Escalation

The shipped Laya Typed-Decisions checkpoint, a 421M-parameter ModernBERT-large assessor that answers typed choice/noul/score questions over workflow state, is uniformly under-confident. The signed confidence-accuracy gap is

-0.214

, every occupied reliability bin's accuracy exceeds its confidence, and that sign uniformity collapses every binned ECE variant to the same value,

0.214

. The card frames the risk as over-confidence; the measured direction is the opposite, and the direction decides which way a confidence-gated cascade fails. A single disjointly fitted temperature (

T=0.469

, sharpening) removes most of the miscalibration (held-out ECE

0.204

0.037

) and outperforms the shipped per-option-count table. The frozen selection rule instead chose isotonic regression, which overfit and failed its held-out NLL contrast on both tracks, so hypothesis H2 is not supported. Re-running the released checkpoint on its full official test split reproduces the card's headline accuracy (

0.767

vs.

0.766

). The retrospective E1 reproduction preceded the analysis freeze; E2-E8 were prospectively preregistered, and 20 of 22 executed confirmatory tests reject under Benjamini-Hochberg FDR at

q=0.05

(two descoped). The frozen gate beats random escalation but misses its 10% accepted-set error target on both tracks, an exploratory out-of-distribution probe finds no zero-shot transfer (accuracy

0.617

), and every score measures agreement with a synthetic teacher whose self-agreement ceiling (

0.735

) the specialist exceeds. Per-decision predictions, run manifests, and the frozen preregistration are in the ancillary files. The author has no affiliation with the model's publisher, the dataset's publisher, or TypeSafe.

Gowthamkumar Nandakishore

LLM Evaluation Selective Prediction LLM Auditing Language Model Calibration LLM Uncertainty Estimation Confidence Estimation

Ask PDF

Abstract

LLM Reliability

Explore similar work

Typed Decision Models: An Early Evidence Audit and Evaluation Checklist

Labels Override Definitions in Jev-Style Typed Decision Models

Laya as a Typed Probabilistic Assessor: An Independent Reproduction and a Preregistered Study of Calibration and Selective Escalation