Laya as a Typed Probabilistic Assessor: An Independent Reproduction and a Preregistered Study of Calibration and Selective Escalation

Gowthamkumar Nandakishore

cs.AISep 27, 2026

Laya as a Typed Probabilistic Assessor: An Independent Reproduction and a Preregistered Study of Calibration and Selective Escalation

Authors: Gowthamkumar Nandakishore

Abstract

The shipped Laya Typed-Decisions checkpoint, a 421M-parameter ModernBERT-large assessor that answers typed choice/noul/score questions over workflow state, is uniformly under-confident. The signed confidence-accuracy gap is $-0.214$ , every occupied reliability bin's accuracy exceeds its confidence, and that sign uniformity collapses every binned ECE variant to the same value, $0.214$ . The card frames the risk as over-confidence; the measured direction is the opposite, and the direction decides which way a confidence-gated cascade fails. A single disjointly fitted temperature ( $T=0.469$ , sharpening) removes most of the miscalibration (held-out ECE $0.204$ to $0.037$ ) and outperforms the shipped per-option-count table. The frozen selection rule instead chose isotonic regression, which overfit and failed its held-out NLL contrast on both tracks, so hypothesis H2 is not supported. Re-running the released checkpoint on its full official test split reproduces the card's headline accuracy ( $0.767$ vs. $0.766$ ). The retrospective E1 reproduction preceded the analysis freeze; E2-E8 were prospectively preregistered, and 20 of 22 executed confirmatory tests reject under Benjamini-Hochberg FDR at $q=0.05$ (two descoped). The frozen gate beats random escalation but misses its 10% accepted-set error target on both tracks, an exploratory out-of-distribution probe finds no zero-shot transfer (accuracy $0.617$ ), and every score measures agreement with a synthetic teacher whose self-agreement ceiling ( $0.735$ ) the specialist exceeds. Per-decision predictions, run manifests, and the frozen preregistration are in the ancillary files. The author has no affiliation with the model's publisher, the dataset's publisher, or TypeSafe.

Figures & tables

Metric	Card ( reported )	Ours ( measured )	$\Delta$	95% CI (BCa, measured )
Accuracy (all, $n{=}2000$ )	0.766	0.7670	$+0.0010$	$[0.7455,0.7870]$
Soft accuracy (ch+no, $n{=}1200$ )	0.471	0.5086	$+0.0376$	$[0.4987,0.5196]$
Brier vs. teacher (ch+no)	0.062	0.0658	$+0.0038$	$[0.0610,0.0716]$
ECE (card conv., 10 bins)	0.213	0.2142	$+0.0012$	$[0.1954,0.2328]$
Score MAE ( score , $n{=}800$ )	0.242	0.2424	$+0.0004$	$[0.2272,0.2580]$

Table 1: E1 headline reproduction (T1). reported = transcribed from the model card (accessed 2026-09-26); measured = this work’s logged run (manifest 20260926T120855Z ).

\Delta

= measured

-

reported. Intervals: case-level cluster bootstrap (400 clusters,

n_{\mathrm{boot}}=10{,}000

, BCa). “ch+no” = choice + noul scope, which is the card’s own scope for those metrics (§ 3.5 ).

Slice	Card ( reported )	measured	$\Delta$	95% CI (BCa, measured )	$n$
agent_trace_observability	0.730	0.7300	$0.0000$	$[0.6800,\ 0.7740]$	500
customer_service	0.764	0.7660	$+0.0020$	$[0.7206,\ 0.8058]$	500
invoice_processing	0.804	0.8060	$+0.0020$	$[0.7660,\ 0.8380]$	500
security_incidents	0.766	0.7660	$0.0000$	$[0.7260,\ 0.8000]$	500
noul accuracy	0.857	0.8583	$+0.0013$	n/a	600
choice accuracy	0.733	0.7333	$+0.0003$	n/a	600

Table 2: E1 per-workflow and per-primitive reproduction. Same provenance and interval conventions as Table 1 . Note that choice ECE (

0.255

) is worse than the aggregate ECE (

0.214

); the aggregate flatters the model’s most error-prone primitive.

Group	Model	Acc. ( measured )	NLL	ECE 15	95% BCa CI (acc.)
specialist	Laya Typed-Decisions (released)	0.7670	0.7069	0.2142	$[0.7455,0.7870]$
	Laya Track B replication (ours)	0.7710	0.6271	0.1509	$[0.7495,0.7905]$
	ModernBERT-base specialist (ours)	0.5425	0.9736	0.0728	$[0.5145,0.5685]$
generalist, zero-shot	laya base	0.3630	1.3311	0.1731	$[0.3385,0.3875]$
generalist, zero-shot	laya-multilingual	0.3495	1.8616	0.3159	$[0.3270,0.3717]$
trivial (fit on train)	majority per question	0.4835	14.27	0.5165	$[0.4585,0.5070]$

Table 3: Comparator results on the full official test split (T3). All measured rows are this work’s logged runs, 2,000 decisions each, accuracy CIs from case-level cluster bootstrap (BCa,

n_{\mathrm{boot}}{=}10{,}000

). NLL uses the one-hot gold argmax; the majority baseline’s NLL is degenerate by construction (near-one-hot predictions). Specialists and zero-shot generalists are not comparable rows (dataset-card discipline). The teacher ceiling is recomputed from the shipped label_agreement fields, which record agreement but not the three raw samples; the strict all-agree rate (

0.594

, measured ) is a floor, and assuming every disagreement is a 2–1 split gives

0.865

( measured ); the card’s

0.735

( reported ) lies inside this bracket and could not be recomputed exactly without the raw samples.

	calib (fit split)		eval (held out, single frozen pass)
Method	NLL	ECE 15	NLL [95% BCa]	ECE 15 [95% BCa]	Acc.	AURC
identity (shipped)	0.7111	0.2250	0.7027 $[0.676,0.731]$	0.2035 $[0.174,0.230]$	0.762	0.0996
single temperature	0.5195	0.0228	0.5364 $[0.487,0.598]$	0.0372 $[0.020,0.043]$	0.762	0.0938
per-primitive	0.5170	0.0226	0.5352 $[0.485,0.597]$	0.0398 $[0.014,0.047]$	0.762	0.0923
per-option-count	0.5170	0.0226	0.5352 $[0.485,0.597]$	0.0398 $[0.014,0.047]$	0.762	0.0923
vector ( choice )	0.5071	0.0239	0.5346 $[0.484,0.598]$	0.0314 $[0.015,0.035]$	0.764	0.0938

Table 4: Calibration methods, Track A (T4). All measured . Fitted on calib only; eval touched exactly once with all methods frozen. Accuracy is invariant under (1)–(4) by construction (temperature preserves argmax); isotonic and vector scaling can change it and do, trivially (

+0.002

). Bold marks the frozen selection rule’s choice (lowest calib NLL), which has the worst held-out NLL in the table, worse than the uncalibrated identity. Fitted parameters: single

T=0.469

; per-primitive

T=(0.539,0.431,0.412)

for ( choice , score , noul ); the per-option-count method collapses to per-primitive because the benchmark occupies only one bucket per primitive (all choice at

k\leq 5

). Per-method Brier and error-detection AUROC with intervals are in anc/metrics/e5_e6_eval.json .

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

ID	Contrast (vs. identity unless noted)	$\Delta$	$p$	BH outcome
A2/B2	eval NLL / ECE, single temp.	$-0.166$ / $-0.166$	${\leq}2{\times}10^{-4}$ / ${\leq}2{\times}10^{-4}$	reject / reject
A3/B3	eval NLL / ECE, per-primitive	$-0.167$ / $-0.164$	${\leq}2{\times}10^{-4}$ / ${\leq}2{\times}10^{-4}$	reject / reject
A4/B4	eval NLL / ECE, per-option-count	$-0.167$ / $-0.164$	${\leq}2{\times}10^{-4}$ / ${\leq}2{\times}10^{-4}$	reject / reject
A5/B5	eval NLL / ECE, vector	$-0.168$ / $-0.172$	${\leq}2{\times}10^{-4}$ / ${\leq}2{\times}10^{-4}$	reject / reject
A6/B6	eval NLL / ECE, isotonic (winner)	$+0.071$ / $-0.153$	$0.446$ / ${\leq}2{\times}10^{-4}$	fail / reject
C2/D2	Track B NLL / ECE, single temp.	$-0.097$ / $-0.129$	${\leq}2{\times}10^{-4}$ / ${\leq}2{\times}10^{-4}$	reject / reject

Appendix

Table 5: Confirmatory family outcomes ( measured ; BH-FDR

q{=}0.05

m{=}24

). Paired case-cluster bootstrap deltas;

p\leq 2{\times}10^{-4}

(two-sided) and

p\leq 10^{-4}

(one-sided, E1t/E2t) are add-one-corrected bounds,

(b{+}1)/(B{+}1)

with

b{=}0

exceedances in

B{=}10{,}000

resamples (the resolution limit). Hypothesis verdicts as operationalized: H1 supported except soft accuracy (§ 4 ); H2 not supported on either track (the frozen selection rule’s winner fails its NLL contrast; A6, C6); H3 partially supported on both tracks (b: better-than-random gating holds, E1t/E2t; a: the

0.10

error target fails out-of-sample); H4 not tested (descoped).

Explore similar work

Do System One Decisions Add Up? A Study of Probabilistic Coherence
Sep 27, 2026Saman Sarker JoyConfidence Calibration Decisions
Faculty of Computer Science and Information Technology Universiti Malaya Kuala Lumpur, Malaysia
Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations
Jan 12, 2026Yuxi Xia, Dennis Ulmer, Terra Blevins +3Confidence Estimation
Faculty of Computer Science, UniVie Doctoral School Computer Science · Faculty of Philological and Cultural Studies, University of Vienna, Austria · ILLC, University of Amsterdam, Netherlands +2
Two Axes of LLM Abstention: Answer Correctness and Question Answerability
Jul 9, 2026Benedikt J. WagnerAnswer Instruction-Tuned Models
City St George’s, University of London

Sep 27, 2026, cs.CLJ/K move · Enter open · S save

Do System One Decisions Add Up? A Study of Probabilistic Coherence

Saman Sarker Joy

Faculty of Computer Science and Information Technology Universiti Malaya Kuala Lumpur, Malaysia

Confidence Calibration Decisions Motion Coherence Joint Search Intermediate Checkpoints

Open paper AskPDF arXiv

A decision model can give probabilities that sum to one for every question yet disagree with itself when the same decision is broken into smaller steps. We study this form of probabilistic coherence in Jev and the English Laya checkpoint, using 2,500 matched examples per system across TREC, CLINC150, and MASSIVE. Across 72,000 classification questions, we compare direct fine-label predictions with broad-category probabilities and predictions reconstructed through those categories. Both systems show substantial disagreement: mean category-level total variation ranges from 0.219 to 0.349 for Jev and from 0.424 to 0.689 for Laya, on a scale where zero means exact agreement. The consequences differ sharply. On CLINC150, reconstruction reduces Jev's accuracy by 22.9 percentage points (paired 95% bootstrap interval: [-24.9, -20.9]) and improves Laya's by 21.3 points ([18.0, 24.5]). The same directions hold across all three datasets, with all six unadjusted accuracy-change intervals excluding zero. Improved accuracy can also accompany less reliable confidence: on MASSIVE, Laya gains 9.2 accuracy points while its expected calibration error rises from 0.046 to 0.124. Error analysis identifies both broad-category mistakes and within-category confusions. These findings show why decision systems need joint evaluation of accuracy, confidence calibration, and probability coherence in the workflow used by an application.

Abstract

Explore similar work

Do System One Decisions Add Up? A Study of Probabilistic Coherence

Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations

Two Axes of LLM Abstention: Answer Correctness and Question Answerability