Laya as a Typed Probabilistic Assessor: An Independent Reproduction and a Preregistered Study of Calibration and Selective Escalation

Gowthamkumar Nandakishore

cs.AISep 27, 2026

Laya as a Typed Probabilistic Assessor: An Independent Reproduction and a Preregistered Study of Calibration and Selective Escalation

Authors: Gowthamkumar Nandakishore

Abstract

The shipped Laya Typed-Decisions checkpoint, a 421M-parameter ModernBERT-large assessor that answers typed choice/noul/score questions over workflow state, is uniformly under-confident. The signed confidence-accuracy gap is $-0.214$ , every occupied reliability bin's accuracy exceeds its confidence, and that sign uniformity collapses every binned ECE variant to the same value, $0.214$ . The card frames the risk as over-confidence; the measured direction is the opposite, and the direction decides which way a confidence-gated cascade fails. A single disjointly fitted temperature ( $T=0.469$ , sharpening) removes most of the miscalibration (held-out ECE $0.204$ to $0.037$ ) and outperforms the shipped per-option-count table. The frozen selection rule instead chose isotonic regression, which overfit and failed its held-out NLL contrast on both tracks, so hypothesis H2 is not supported. Re-running the released checkpoint on its full official test split reproduces the card's headline accuracy ( $0.767$ vs. $0.766$ ). The retrospective E1 reproduction preceded the analysis freeze; E2-E8 were prospectively preregistered, and 20 of 22 executed confirmatory tests reject under Benjamini-Hochberg FDR at $q=0.05$ (two descoped). The frozen gate beats random escalation but misses its 10% accepted-set error target on both tracks, an exploratory out-of-distribution probe finds no zero-shot transfer (accuracy $0.617$ ), and every score measures agreement with a synthetic teacher whose self-agreement ceiling ( $0.735$ ) the specialist exceeds. Per-decision predictions, run manifests, and the frozen preregistration are in the ancillary files. The author has no affiliation with the model's publisher, the dataset's publisher, or TypeSafe.

Figures & tables

Metric	Card ( reported )	Ours ( measured )	$\Delta$	95% CI (BCa, measured )
Accuracy (all, $n{=}2000$ )	0.766	0.7670	$+0.0010$	$[0.7455,0.7870]$
Soft accuracy (ch+no, $n{=}1200$ )	0.471	0.5086	$+0.0376$	$[0.4987,0.5196]$
Brier vs. teacher (ch+no)	0.062	0.0658	$+0.0038$	$[0.0610,0.0716]$
ECE (card conv., 10 bins)	0.213	0.2142	$+0.0012$	$[0.1954,0.2328]$
Score MAE ( score , $n{=}800$ )	0.242	0.2424	$+0.0004$	$[0.2272,0.2580]$

Table 1: E1 headline reproduction (T1). reported = transcribed from the model card (accessed 2026-09-26); measured = this work’s logged run (manifest 20260926T120855Z ).

\Delta

= measured

-

reported. Intervals: case-level cluster bootstrap (400 clusters,

n_{\mathrm{boot}}=10{,}000

, BCa). “ch+no” = choice + noul scope, which is the card’s own scope for those metrics (§ 3.5 ).

Slice	Card ( reported )	measured	$\Delta$	95% CI (BCa, measured )	$n$
agent_trace_observability	0.730	0.7300	$0.0000$	$[0.6800,\ 0.7740]$	500
customer_service	0.764	0.7660	$+0.0020$	$[0.7206,\ 0.8058]$	500
invoice_processing	0.804	0.8060	$+0.0020$	$[0.7660,\ 0.8380]$	500
security_incidents	0.766	0.7660	$0.0000$	$[0.7260,\ 0.8000]$	500
noul accuracy	0.857	0.8583	$+0.0013$	n/a	600
choice accuracy	0.733	0.7333	$+0.0003$	n/a	600

Table 2: E1 per-workflow and per-primitive reproduction. Same provenance and interval conventions as Table 1 . Note that choice ECE (

0.255

) is worse than the aggregate ECE (

0.214

); the aggregate flatters the model’s most error-prone primitive.

Group	Model	Acc. ( measured )	NLL	ECE 15	95% BCa CI (acc.)
specialist	Laya Typed-Decisions (released)	0.7670	0.7069	0.2142	$[0.7455,0.7870]$
	Laya Track B replication (ours)	0.7710	0.6271	0.1509	$[0.7495,0.7905]$
	ModernBERT-base specialist (ours)	0.5425	0.9736	0.0728	$[0.5145,0.5685]$
generalist, zero-shot	laya base	0.3630	1.3311	0.1731	$[0.3385,0.3875]$
generalist, zero-shot	laya-multilingual	0.3495	1.8616	0.3159	$[0.3270,0.3717]$
trivial (fit on train)	majority per question	0.4835	14.27	0.5165	$[0.4585,0.5070]$

Table 3: Comparator results on the full official test split (T3). All measured rows are this work’s logged runs, 2,000 decisions each, accuracy CIs from case-level cluster bootstrap (BCa,

n_{\mathrm{boot}}{=}10{,}000

). NLL uses the one-hot gold argmax; the majority baseline’s NLL is degenerate by construction (near-one-hot predictions). Specialists and zero-shot generalists are not comparable rows (dataset-card discipline). The teacher ceiling is recomputed from the shipped label_agreement fields, which record agreement but not the three raw samples; the strict all-agree rate (

0.594

, measured ) is a floor, and assuming every disagreement is a 2–1 split gives

0.865

( measured ); the card’s

0.735

( reported ) lies inside this bracket and could not be recomputed exactly without the raw samples.

	calib (fit split)		eval (held out, single frozen pass)
Method	NLL	ECE 15	NLL [95% BCa]	ECE 15 [95% BCa]	Acc.	AURC
identity (shipped)	0.7111	0.2250	0.7027 $[0.676,0.731]$	0.2035 $[0.174,0.230]$	0.762	0.0996
single temperature	0.5195	0.0228	0.5364 $[0.487,0.598]$	0.0372 $[0.020,0.043]$	0.762	0.0938
per-primitive	0.5170	0.0226	0.5352 $[0.485,0.597]$	0.0398 $[0.014,0.047]$	0.762	0.0923
per-option-count	0.5170	0.0226	0.5352 $[0.485,0.597]$	0.0398 $[0.014,0.047]$	0.762	0.0923
vector ( choice )	0.5071	0.0239	0.5346 $[0.484,0.598]$	0.0314 $[0.015,0.035]$	0.764	0.0938

Table 4: Calibration methods, Track A (T4). All measured . Fitted on calib only; eval touched exactly once with all methods frozen. Accuracy is invariant under (1)–(4) by construction (temperature preserves argmax); isotonic and vector scaling can change it and do, trivially (

+0.002

). Bold marks the frozen selection rule’s choice (lowest calib NLL), which has the worst held-out NLL in the table, worse than the uncalibrated identity. Fitted parameters: single

T=0.469

; per-primitive

T=(0.539,0.431,0.412)

for ( choice , score , noul ); the per-option-count method collapses to per-primitive because the benchmark occupies only one bucket per primitive (all choice at

k\leq 5

). Per-method Brier and error-detection AUROC with intervals are in anc/metrics/e5_e6_eval.json .

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

ID	Contrast (vs. identity unless noted)	$\Delta$	$p$	BH outcome
A2/B2	eval NLL / ECE, single temp.	$-0.166$ / $-0.166$	${\leq}2{\times}10^{-4}$ / ${\leq}2{\times}10^{-4}$	reject / reject
A3/B3	eval NLL / ECE, per-primitive	$-0.167$ / $-0.164$	${\leq}2{\times}10^{-4}$ / ${\leq}2{\times}10^{-4}$	reject / reject
A4/B4	eval NLL / ECE, per-option-count	$-0.167$ / $-0.164$	${\leq}2{\times}10^{-4}$ / ${\leq}2{\times}10^{-4}$	reject / reject
A5/B5	eval NLL / ECE, vector	$-0.168$ / $-0.172$	${\leq}2{\times}10^{-4}$ / ${\leq}2{\times}10^{-4}$	reject / reject
A6/B6	eval NLL / ECE, isotonic (winner)	$+0.071$ / $-0.153$	$0.446$ / ${\leq}2{\times}10^{-4}$	fail / reject
C2/D2	Track B NLL / ECE, single temp.	$-0.097$ / $-0.129$	${\leq}2{\times}10^{-4}$ / ${\leq}2{\times}10^{-4}$	reject / reject

Appendix

Table 5: Confirmatory family outcomes ( measured ; BH-FDR

q{=}0.05

m{=}24

). Paired case-cluster bootstrap deltas;

p\leq 2{\times}10^{-4}

(two-sided) and

p\leq 10^{-4}

(one-sided, E1t/E2t) are add-one-corrected bounds,

(b{+}1)/(B{+}1)

with

b{=}0

exceedances in

B{=}10{,}000

resamples (the resolution limit). Hypothesis verdicts as operationalized: H1 supported except soft accuracy (§ 4 ); H2 not supported on either track (the frozen selection rule’s winner fails its NLL contrast; A6, C6); H3 partially supported on both tracks (b: better-than-random gating holds, E1t/E2t; a: the

0.10

error target fails out-of-sample); H4 not tested (descoped).

Explore similar work

Sep 27, 2026cs.CL

Do System One Decisions Add Up? A Study of Probabilistic Coherence

A decision model can give probabilities that sum to one for every question yet disagree with itself when the same decision is broken into smaller steps. We study this form of probabilistic coherence in Jev and the English Laya checkpoint, using 2,500 matched examples per system across TREC, CLINC150, and MASSIVE. Across 72,000 classification questions, we compare direct fine-label predictions with broad-category probabilities and predictions reconstructed through those categories. Both systems show substantial disagreement: mean category-level total variation ranges from 0.219 to 0.349 for Jev and from 0.424 to 0.689 for Laya, on a scale where zero means exact agreement. The consequences differ sharply. On CLINC150, reconstruction reduces Jev's accuracy by 22.9 percentage points (paired 95% bootstrap interval: [-24.9, -20.9]) and improves Laya's by 21.3 points ([18.0, 24.5]). The same directions hold across all three datasets, with all six unadjusted accuracy-change intervals excluding zero. Improved accuracy can also accompany less reliable confidence: on MASSIVE, Laya gains 9.2 accuracy points while its expected calibration error rises from 0.046 to 0.124. Error analysis identifies both broad-category mistakes and within-category confusions. These findings show why decision systems need joint evaluation of accuracy, confidence calibration, and probability coherence in the workflow used by an application.

Saman Sarker Joy

Faculty of Computer Science and Information Technology Universiti Malaya Kuala Lumpur, Malaysia

Confidence Calibration Decisions Motion Coherence Joint Search Intermediate Checkpoints

Ask PDF

Jan 12, 2026cs.CL

Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations

Confidence estimation (CE) indicates how reliable the answers of large language models are and impacts user trust and decision-making. Existing evaluations mainly concern the alignment between confidence and correctness, but ignore the variability of language: confidence estimates should remain consistent under semantically equivalent prompts or answer variations, while changing when answer meaning differs, as this may indicate a change in correctness. Therefore, we introduce a novel evaluation framework based on three complementary properties: \textbf{robustness} to prompt perturbations, \textbf{stability} across semantically equivalent answers, and \textbf{sensitivity} to semantically different answers. We show that these metrics are largely independent from existing CE metrics, and that common CE methods often fail on them: while most methods achieve high robustness and stability, they struggle to distinguish semantically different answers, potentially because they do not effectively leverage generation-side information. Overall, our framework exposes overlooked limitations of current CE evaluations and provides guidance for selecting confidence estimators for real-world applications.

Yuxi Xia, Dennis Ulmer, Terra Blevins +3

Faculty of Computer Science, UniVie Doctoral School Computer Science · Faculty of Philological and Cultural Studies, University of Vienna, Austria · ILLC, University of Amsterdam, Netherlands +2

Confidence Estimation

Ask PDF

Jul 9, 2026cs.CL

Two Axes of LLM Abstention: Answer Correctness and Question Answerability

A model should refuse two different things: answers it would get wrong, and questions it should not answer at all, such as unanswerable ones or ones resting on a false premise. The usual recipe thresholds a single confidence score, which cannot tell these apart. Across five instruction-tuned models from three families (2B to 14B), we find they are separate axes. Ordinary answer-confidence tracks whether an answer is right but is nearly blind to whether the question is answerable; a linear probe on hidden states does the reverse. The blind spot does not shrink with scale. It is worst on naturally occurring false-premise questions (CREPE). There, answer-confidence, P(IK), P(True), and even asking the model outright whether a premise is false all stay near chance, while a hidden-state probe reaches 0.69 to 0.77 AUROC: the model represents a problem it will not report. This turns out to be fixable. Instructing a model to check premises backfires, because it then disputes sound and false premises alike (57% false challenges), unable to tell them apart; routing the same instruction with the probe roughly triples challenge precision. We turn the two axes into a calibrated policy that answers only when an answerability score and a correctness score each clear a separately certifies behave differently: the unanswerable-answer rate is controllable at every scale, while the wrong-answer rate is capped by model accuracy, so the guarantee tightens as threshold policy certifies both budgets at 0.75 coverage of correct answers, against 0.31 for a single threshold; at 14B it is the only policy that certifies at all.

Benedikt J. Wagner

City St George’s, University of London

Answer Instruction-Tuned Models Question Decision Thresholds

Ask PDF