Typed Decision Models under Prevalence Shift: Decision Training, Calibration, and Implicit Priors

Abdul Rahman Iqbal; Muhammad Fawad Nasim; Amir Saleem

cs.LGOct 19, 2026

Typed Decision Models under Prevalence Shift: Decision Training, Calibration, and Implicit Priors

Authors: Abdul Rahman Iqbal, Muhammad Fawad Nasim, Amir Saleem

Organizations: Faculty of Computer Science and Information Technology, The Superior University, Lahore, Pakistan · Independent Researcher

Abstract

Typed decision models (TDMs) return a probability over caller-defined options, and Jev is described by its vendor as calibrated. But in deployment class frequencies change, so calibration measured on a fixed test set need not hold where thresholds are applied. Existing evaluations leave two gaps: the gains of vendor decision training have not been separated from those of the backbone under an identical readout, and TDMs have been tested only at the test set's class distribution. We ask whether decision training improves a model over its own backbone, and whether a TDM's calibration and thresholds withstand prevalence (label) shift as well as a fine-tuned classifier's. We compare three TDMs (Jev, Tev1-4B, Nimble-9B), their base models and two trained classifiers on identical decisions from five public datasets under simulated shift, with 35 paired tests fixed before any test was computed. First, decision training is not uniformly beneficial: Nimble-9B, which differs from Qwen3.5-9B only by a LoRA adapter, is better calibrated on intent tasks but 12.0 Macro-F1 points worse on financial sentiment. Second, under moderate shift Jev's expected calibration error (ECE) and threshold risk grow significantly more than a fine-tuned DeBERTa's only on financial sentiment, by 1.7 and 2.5 percentage points; averaged over 2,000 shift scenarios (exploratory), the gap is at most one point on every dataset. Third, EM prior correction with the model's mean development prediction as its prior needs no labels and lowers Jev's ECE under shift by 2.7 and 1.8 points on the two datasets where it is highest, although labelled recalibration lowers it far more. The three TDMs we tested are neither uniformly better than their backbones nor uniquely fragile under shift; for calibration, their main weakness is the miscalibration they already have without shift. We release all inputs, code and outputs.

Figures & tables

Figure 1: Decision training changes a backbone in a task-dependent direction, and label-free prior correction helps where calibration is poor. (a, b) Paired differences between Nimble-9B and its own base model Qwen3.5-9B (same weights and 8-bit precision, differing only by a LoRA adapter) in Macro-F1 (higher favours Nimble) and expected calibration error (ECE; lower favours Nimble), in percentage points. (c) Change in Jev’s ECE under prevalence shift (

\alpha=1

, 50 fixed target priors) when EM prior correction with its implicit prior is applied (lower is better). Bars are 95% paired bootstrap intervals (

B=10{,}000

, unadjusted); filled markers are significant after Holm correction over all 35 primary tests.

Study	TDM	Pair	Trained	Shift	Prior	Tests
Deußer et al. (2026)	✓	–	–	–	–	$\circ$
Rafe & Das (2026)	✓	$\circ$	✓	–	–	✓
Li & Wagle (2026)	✓	$\circ$	–	–	–	$\circ$
Ibrahim & Zaki (2026)	✓	–	–	–	–	–
Estienne (2023)	–	–	–	–	✓	–
Lipton et al. (2018) ; Alexandari et al. (2020) ; Garg et al. (2020)	–	–	✓	✓	–	$\circ$

Table 1: No prior study combines vendor TDMs, matched backbones, trained classifiers and controlled prevalence shift with label-free prior correction. ✓: covered;

\circ

: partly covered (Pair: different readouts for the two models, or the authors’ own fine-tunes rather than a vendor TDM; Tests: intervals or tests without multiplicity control); –: not covered. Columns: TDM = commercial or vendor TDMs evaluated; Pair = TDM compared with its own backbone under the same readout; Trained = task-trained classifier baseline; Shift = controlled class-prior shift; Prior = label-free prior correction for zero-shot models; Tests = paired tests with multiplicity control.

Figure 2: All systems answer identical candidate-set decisions, and shift, recalibration and tests operate on their saved probabilities, so every comparison is paired. (1) Five public datasets with frozen label texts and fixed candidate sets. (2) A hosted TDM, two open TDMs each paired with its own backbone, and two trained classifiers return a probability vector over the candidates. (3) The test set is resampled to simulated target priors ( Eq. 1 ), and five recalibration arms are fitted on the unshifted development set ( Eqs. 2 – 3 ). (4) Metrics ( Eqs. 4 and 5 ) and the 35 confirmatory tests ( Eqs. 6 – 8 ); everything else is descriptive.

Dataset	Domain	Classes	Candidates $K$	Train	Dev	Test
CLINC150	intent (assistant)	150	18–19	14,939	2,994	4,500
HWU64	intent (home assistant)	64	21–22	7,871	1,076	1,076
FPB	financial sentiment	3	3	3,090	775	970
HateXplain	hate / offensive speech	3	3	15,330	1,921	1,924
TREC-6	question type	6	6	4,869	498	500

Table 2: The five datasets span intent, sentiment, moderation and question-type decisions with 3 to 150 classes. CLINC150 and HWU64 use fixed label groups as candidate sets (

K

= 18–22 options); the other three use all labels. Train/dev/test sizes after cleaning.

System	Source	Revision	Precision	Readout
Typed decision models
Jev	Cloudflare API ( typesafe/jev )	jev-1.13.0	API	API probabilities
Tev1-4B	togethercomputer/Tev1-4B-experimental	0b7becf0	bf16	letters, Tev1 format
Nimble-9B	Qwen3.5-9B + LoRA bespokelabs/Bespoke-Nimble-9B	fe33f82b ( original-2676 )	8-bit	letters, Nimble format
Base LLMs
Qwen3.5-4B	Qwen/Qwen3.5-4B	851bf6e8	bf16	as Tev1-4B

Table 3: Each open TDM is paired with its own backbone under identical prompts, readout and precision. Tev1-4B with Qwen3.5-4B (Tev1 prompt format) and Nimble-9B with Qwen3.5-9B (Nimble prompt format, identical 8-bit base weights). Revisions are Hugging Face commit hashes. Readout: “letters” is the softmax (

T=1

) over the logits of the candidate letters at the first answer position; the trained classifiers use a softmax over each example’s candidates; Jev returns probabilities directly.

	CLINC150		HWU64		FPB		HateXplain		TREC-6
System	F1 $\uparrow$	ECE $\downarrow$	F1 $\uparrow$	ECE $\downarrow$	F1 $\uparrow$	ECE $\downarrow$	F1 $\uparrow$	ECE $\downarrow$	F1 $\uparrow$	ECE $\downarrow$
Typed decision models
Jev	0.978	0.007	0.908	0.036	0.752	0.095	0.514	0.279	0.922	0.042
Tev1-4B	0.964	0.008	0.895	0.040	0.735	0.072	0.458	0.214	0.889	0.038
Nimble-9B	0.962	0.008	0.896	0.035	0.629	0.263	0.470	0.336	0.872	0.043
Base LLMs

Table 4: Full-data DeBERTa-v3-large has the highest Macro-F1 point estimate on every dataset, and the calibration of zero-shot systems depends more on the task than on the system. Test-set Macro-F1 and ECE (15 bins, top label), candidate-restricted, no shift. These are point estimates; tested comparisons are in Figure 3 and Table 6 .

Figure 3: Tev1-4B is never significantly worse than its base model, whereas Nimble-9B is better calibrated on intent tasks and 12 Macro-F1 points worse on FPB. Paired differences (TDM

-

base) in percentage points with 95% paired bootstrap intervals (

B=10{,}000

, unadjusted). Filled markers: significant after Holm correction over all 35 primary tests.

Test	Prior set	CLINC150	HWU64	FPB	HateXplain	TREC-6
PA1	frozen (confirmatory)	+0.1	+0.1	+1.7 †	$-$ 0.8	+1.0
	5 other sets: range	[0.0, +0.1]	[+0.1, +0.4]	[+0.3, +1.7]	[+0.2, +0.8]	[+0.8, +1.2]
	CI excludes 0	1/5	0/5	3/5	4/5	5/5
PA2	frozen (confirmatory)	0.0	0.0	+2.5 †	$-$ 0.6	+0.5
	5 other sets: range	[0.0, 0.0]	[0.0, +0.2]	[+0.6, +2.3]	[+0.6, +1.5]	[+0.4, +0.6]
	CI excludes 0	0/5	0/5	5/5	5/5	0/5

Table 5: The EM correction effect (PA3) is stable across shift scenarios, whereas the Jev–DeBERTa difference (PA1, PA2) varies in size, and on HateXplain in sign. Frozen: the confirmatory estimate on the pre-specified set of 50 target priors (

B=10{,}000

;

\dagger

: Holm-adjusted

p<0.05

). Exploratory: range of the estimates over five other sets of 50 priors (

B=1{,}000

each) and the number of sets whose unadjusted 95% interval excludes zero. Points.

Test (points)	CLINC150	HWU64	FPB	HateXplain	TREC-6
PA1: $\Delta$ ECE under shift, Jev $-$ DeBERTa	+0.09 [ $-$ 0.02, +0.20]	+0.11 [ $-$ 0.30, +0.50]	+1.71 † [+0.98, +2.49]	$-$ 0.78 [ $-$ 1.29, $-$ 0.26]	+1.01 [+0.26, +1.85]
PA2: excess risk at $\tau_{80}$ , Jev $-$ DeBERTa	0.00 [ $-$ 0.04, +0.04]	+0.01 [ $-$ 0.21, +0.23]	+2.49 † [+1.83, +3.16]	$-$ 0.55 [ $-$ 0.98, $-$ 0.14]	+0.55 [ $-$ 0.06, +1.30]
PA3: ECE, Jev EM $-$ none	$-$ 0.12 [ $-$ 0.29, +0.03]	+0.12 [ $-$ 0.33, +0.54]	$-$ 2.65 † [ $-$ 3.74, $-$ 1.61]	$-$ 1.75 † [ $-$ 2.71, $-$ 0.82]	$-$ 1.06 [ $-$ 1.96, $-$ 0.28]

Table 6: At moderate shift (

\alpha=1

), Jev and DeBERTa-v3-large are distinguishable only on FPB, and EM with Jev’s implicit prior lowers its shifted ECE on FPB and HateXplain. Estimates in percentage points with 95% bootstrap intervals (unadjusted; class-stratified test bootstrap,

B=10{,}000

, conditional on 50 fixed target priors, common random numbers).

\dagger

and bold: Holm-adjusted

p<0.05

over the 35-test family; an unadjusted interval that excludes zero is not by itself significant.

Figure 4: Under stronger shift, Jev’s calibration error grows more than DeBERTa-v3-large’s on all five datasets, but this is not a zero-shot versus trained contrast: the TF-IDF classifier degrades more than Jev on two of them, and Tev1-4B more than Jev on FPB and TREC-6 (descriptive). Change in ECE relative to a resampled no-shift control, mean over 200 Dirichlet draws per

\alpha

, raw probabilities. Only

\alpha=1

is tested ( Table 6 ); all systems in Table 11 .

Figure 5: Most of the calibration gain under shift comes from calibrating on the development set, not from correcting the prior (descriptive). ECE under shift (

\alpha=1

, mean over 200 draws) for each recalibration arm. Label-free: EM, prior correction with the source prior (implicit for Jev, training prior for DeBERTa-v3-large). Labelled (fitted on the labelled development set): temp., temperature scaling; BCTS+EM, bias-corrected temperature scaling, then EM. Reference: oracle, true target prior (not deployable). All systems in Table 12 .

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Test	Dataset	$\Delta$ (points)	95% CI	$p$	Holm $p$
P1-F1	CLINC150	+2.48	[+1.94, +3.10]	$<$ 0.0002	$<$ 0.007
P1-F1	HWU64	+2.84	[+1.10, +4.78]	0.002	0.040
P1-F1	FPB	+9.99	[+7.35, +12.69]	$<$ 0.0002	$<$ 0.007
P1-F1	HateXplain	+1.69	[ $-$ 0.16, +3.52]	0.077	1.000
P1-F1	TREC-6	+0.65	[ $-$ 2.92, +4.44]	0.724	1.000
P1-ECE	CLINC150	$-$ 0.34	[ $-$ 0.84, +0.21]	0.255	1.000

Appendix

Table 7: Fourteen of the 35 confirmatory tests are significant after Holm correction. Paired differences in percentage points (TDM

-

base for P1/P2; Jev

-

DeBERTa-v3-large for PA1/PA2; EM

-

none for PA3) with unadjusted 95% bootstrap intervals, raw two-sided

p

and Holm-adjusted

p

over the 35 tests.

Dataset	System	Acc $\uparrow$	F1 $\uparrow$	MCC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AURC $\downarrow$	Acc@80 $\uparrow$
CLINC150	Jev	0.978	0.978	0.978	0.007	0.032	0.152	0.004	0.997
	Tev1-4B	0.964	0.964	0.963	0.008	0.055	0.130	0.003	0.997
	Nimble-9B	0.962	0.962	0.961	0.008	0.060	0.140	0.004	0.996
	Nimble-9B (main) ∗	0.969	0.969	0.968	0.010	0.050	0.123	0.002	0.997
	Qwen3.5-4B	0.939	0.939	0.939	0.011	0.089	0.233	0.007	0.994
	Qwen3.5-9B	0.955	0.955	0.955	0.020	0.067	0.176	0.005	0.995

Appendix

Table 8: Accuracy and MCC rank the seven analysed systems almost as Macro-F1 does (Spearman

\rho\geq 0.75

on every dataset; descriptive), whereas NLL, which penalises confident errors, ranks them differently on HWU64 and HateXplain (

\rho=0.29

and

0.32

). Test-set metrics for all systems, no shift. ∗ Exploratory: a later Nimble release whose training data include public TREC questions, which may explain its TREC-6 score; it is not part of any confirmatory test or of the rank correlations. Acc@80: accuracy on the 80% most confident examples.

Dataset	seed 1001	1002	1003	1004	1005	mean $\pm$ SD
CLINC150	0.980	0.981	0.983	0.982	0.981	0.981 $\pm$ 0.001
HWU64	0.939	0.943	0.954	0.943	0.942	0.944 $\pm$ 0.006
FPB	0.870	0.854	0.857	0.865	0.868	0.863 $\pm$ 0.007
HateXplain	0.703	0.700	0.700	0.711	0.700	0.703 $\pm$ 0.005
TREC-6	0.971	0.978	0.974	0.971	0.959	0.971 $\pm$ 0.007

Appendix

Table 9: DeBERTa-v3-large is stable across seeds. Test Macro-F1 of the five full-data seeds; all analyses use seed 1001.

Dataset	$n$	same prediction	same prob. vector	max $\|\Delta p\|$	accuracy (3 repeats)	ECE range (4 runs)
CLINC150	500	99.8%	87.4%	0.12	0.972 / 0.970 / 0.972	0.012–0.015
HWU64	500	99.6%	72.2%	0.20	0.918 / 0.920 / 0.916	0.039–0.049
FPB	500	97.6%	27.4%	0.16	0.738 / 0.738 / 0.736	0.110–0.116
HateXplain	500	96.4%	19.4%	0.29	0.538 / 0.532 / 0.532	0.271–0.282
TREC-6	500	98.2%	63.8%	0.15	0.926 / 0.928 / 0.926	0.030–0.042

Appendix

Table 10: Jev’s predictions are nearly stable across repeated calls, its probabilities are not, and its ECE varies by up to 1.2 points between runs. Four runs (the full run and three repeats) over the same 500 test examples per dataset, all returning jev-1.13.0 ; accuracy shown for the three repeats, ECE range over all four runs.

	CLINC150		HWU64		FPB		HateXplain		TREC-6
System	$\alpha$ =1	0.1	$\alpha$ =1	0.1	$\alpha$ =1	0.1	$\alpha$ =1	0.1	$\alpha$ =1	0.1
Jev	+0.1	+0.5	+0.2	+1.5	+0.2	+0.8	$-$ 1.0	+2.5	+1.4	+2.4
Tev1-4B	+0.1	+0.9	+0.3	+1.9	+1.1	+4.8	$-$ 0.8	+2.0	+1.9	+4.4
Nimble-9B	+0.1	+1.0	+0.4	+2.0	+0.5	+1.0	$-$ 1.2	+2.5	+0.8	+2.6
Qwen3.5-4B	+0.3	+1.3	+0.3	+1.7	+0.8	+3.9	$-$ 0.9	+3.7	+0.8	+2.7
Qwen3.5-9B	+0.1	+0.4	+0.1	+0.8	+0.2	+0.3	$-$ 1.0	+1.9	+0.2	+1.2

Appendix

Table 11: Under the strongest shift, the calibration error grows for every system on four of the five datasets (TF-IDF on FPB is unchanged), and DeBERTa-v3-large is the most stable on three of the five datasets (descriptive). Change in ECE (points) relative to the resampled no-shift control, raw probabilities, mean over 200 Dirichlet draws per

\alpha

Dataset	System	none	temp.	EM	BCTS+EM	oracle
CLINC150	Jev	0.009	0.010	0.008	0.008	0.007
	Tev1-4B	0.011	0.010	0.011	0.010	0.010
	Nimble-9B	0.011	0.009	0.012	0.017	0.010
	Qwen3.5-4B	0.016	0.015	0.015	0.014	0.012
	Qwen3.5-9B	0.022	0.011	0.020	0.014	0.017
	DeBERTa-v3-large	0.014	0.008	0.013	0.012	0.012

Appendix

Table 12: Bias-corrected temperature scaling followed by EM gives the lowest calibration error under shift for most systems, and EM with the implicit prior helps every zero-shot system on FPB and HateXplain (descriptive). ECE under shift (

\alpha=1

, mean over 200 draws) per recalibration arm; bold marks the lowest value per row (point estimates).

Quantity (points)	CLINC150	HWU64	FPB	HateXplain	TREC-6
PA1 ( $\Delta$ ECE, Jev $-$ DeBERTa): 50 fixed priors	+0.1	+0.1	+1.7	$-$ 0.8	+1.0
200 descriptive draws	0.0	+0.1	+0.2	$-$ 0.8	+1.0
PA2 (excess risk, Jev $-$ DeBERTa): 50 fixed priors	0.0	0.0	+2.5	$-$ 0.6	+0.5
200 descriptive draws	0.0	0.0	+0.4	$-$ 0.7	+0.4
PA3 (ECE, Jev EM $-$ none): 50 fixed priors	$-$ 0.1	+0.1	$-$ 2.7	$-$ 1.8	$-$ 1.1
200 descriptive draws	$-$ 0.1	0.0	$-$ 1.8	$-$ 1.7	$-$ 1.1

Appendix

Table 13: The confirmatory PA estimates and the 200-draw descriptive experiment agree in direction on every dataset, but their size depends on the set of target priors. PA rows: 50 fixed priors with the class-stratified bootstrap; descriptive rows: 200 independent Dirichlet draws on the full test set (points). Prior-set sensitivity: Section 5.2 .

Test	Estimate	CLINC150	HWU64	FPB	HateXplain	TREC-6
PA1	frozen 50 priors	+0.1	+0.1	+1.7 †	$-$ 0.8	+1.0
	2,000 priors: mean (SE)	0.0 (0.0)	+0.1 (0.0)	+0.4 (0.1)	0.0 (0.2)	+1.0 (0.0)
	SD across priors	0.4	1.4	5.4	7.4	1.7
PA2	frozen 50 priors	0.0	0.0	+2.5 †	$-$ 0.6	+0.5
	2,000 priors: mean (SE)	0.0 (0.0)	0.0 (0.0)	+0.5 (0.2)	+0.4 (0.2)	+0.5 (0.0)
	SD across priors	0.2	0.9	7.6	9.8	1.3

Appendix

Table 14: Averaged over many target priors, the Jev–DeBERTa difference under moderate shift is small on every dataset, while the EM effect (PA3) stays clearly negative on FPB and HateXplain (exploratory). The PA1–PA3 statistics for 2,000 target priors drawn at

\alpha=1

(seed 777), each evaluated on one resample of the full test set (with a resampled no-shift control for PA1, but without the bootstrap base): mean with its standard error over priors, and the standard deviation across single priors (which includes resampling noise), next to the frozen confirmatory estimate on 50 priors (

\dagger

: Holm-adjusted

p<0.05

). Points.

System	CLINC150	HWU64	FPB	HateXplain	TREC-6
Jev	0.0 / 0.3 / 0.6	0.0 / 1.3 / 2.3	+0.4 / 9.6 / 12.4	$-$ 0.9 / 19.5 / 21.8	+0.7 / 2.8 / 4.1
Tev1-4B	0.0 / 0.3 / 0.6	$-$ 0.1 / 1.7 / 2.9	+1.0 / 17.1 / 21.3	$-$ 0.6 / 12.1 / 16.1	+0.9 / 5.3 / 7.9
Nimble-9B	0.0 / 0.2 / 0.7	+0.1 / 2.4 / 5.6	+2.3 / 28.4 / 33.0	$-$ 1.1 / 17.4 / 20.2	0.0 / 2.9 / 4.9
Qwen3.5-4B	0.0 / 0.3 / 0.7	0.0 / 2.0 / 4.8	+2.1 / 27.3 / 34.7	$-$ 1.0 / 17.2 / 19.2	+0.4 / 3.2 / 4.8
Qwen3.5-9B	0.0 / 0.3 / 0.5	+0.1 / 2.3 / 4.4	+0.5 / 9.3 / 11.2	$-$ 1.0 / 15.7 / 17.6	0.0 / 2.6 / 3.7
DeBERTa-v3-large	0.0 / 0.2 / 0.4	$-$ 0.1 / 0.9 / 2.1	0.0 / 3.4 / 5.9	$-$ 0.2 / 10.2 / 12.6	+0.3 / 2.1 / 3.8

Appendix

Table 15: Averaged over shifts, the excess risk of a fixed confidence threshold is small, but single shifts can raise it by up to 35 points (descriptive). Excess selective risk at 80% development coverage under shift (

\alpha=1

, 200 draws, raw probabilities), in points: mean / 95th percentile / maximum over draws. The mean is a single system’s value; PA2 ( Table 6 ) is the Jev–DeBERTa-v3-large difference over 50 fixed priors.

	CLINC150		HWU64		FPB		HateXplain		TREC-6
System	$\alpha$ =1	0.1	$\alpha$ =1	0.1	$\alpha$ =1	0.1	$\alpha$ =1	0.1	$\alpha$ =1	0.1
Jev	0.0	+0.1	$-$ 0.2	$-$ 0.6	0.0	+0.3	$-$ 2.0	+3.7	+1.7	+1.9
Tev1-4B	$-$ 0.1	+0.1	$-$ 0.1	$-$ 0.6	+0.2	+0.3	$-$ 1.1	+2.0	+1.0	+1.7
Nimble-9B	0.0	+0.1	$-$ 0.1	$-$ 0.7	+0.9	+0.9	$-$ 2.1	+3.9	$-$ 1.0	$-$ 0.6
Qwen3.5-4B	$-$ 0.1	+0.1	$-$ 0.1	$-$ 0.4	+0.9	+1.1	$-$ 1.7	+3.2	+0.5	+0.8
Qwen3.5-9B	0.0	0.0	$-$ 0.1	$-$ 0.6	+0.2	+0.2	$-$ 1.9	+3.6	$-$ 0.4	0.0

Appendix

Table 16: Brier score under shift follows the same pattern as ECE: small changes at

\alpha=1

, larger ones at

\alpha=0.1

, mostly for the zero-shot systems on HateXplain (descriptive). Change in Brier score (

\times 100

) relative to the resampled no-shift control, raw probabilities, mean over 200 draws.

	CLINC150		HWU64		FPB		HateXplain		TREC-6
System	$\alpha$ =1	0.1	$\alpha$ =1	0.1	$\alpha$ =1	0.1	$\alpha$ =1	0.1	$\alpha$ =1	0.1
Jev	$-$ 0.1	0.0	$-$ 2.8	$-$ 7.3	$-$ 0.2	+0.7	$-$ 10.8	+19.7	+3.7	+5.3
Tev1-4B	$-$ 0.2	+0.4	$-$ 0.6	$-$ 1.7	+0.3	+0.5	$-$ 2.5	+4.8	+3.8	+5.0
Nimble-9B	$-$ 0.1	+0.1	$-$ 0.7	$-$ 2.1	+1.5	+1.4	$-$ 5.1	+8.9	$-$ 0.5	$-$ 0.1
Qwen3.5-4B	$-$ 0.2	+0.4	$-$ 0.2	$-$ 1.1	+1.4	+2.0	$-$ 3.4	+6.6	+1.4	+1.8
Qwen3.5-9B	0.0	0.0	$-$ 0.4	$-$ 1.8	+0.3	+0.7	$-$ 4.6	+8.7	$-$ 0.3	+0.8

Appendix

Table 17: NLL under shift changes most for Jev on HateXplain, where its probabilities are least calibrated (descriptive). Change in NLL (

\times 100

, nats; probabilities clipped at

10^{-12}

) relative to the resampled no-shift control, raw probabilities, mean over 200 draws. A negative change means that the shifted samples contain more of the classes on which the system is confident and correct.

	CLINC150		HWU64		FPB		HateXplain		TREC-6
System	no shift	$\alpha$ =1	no shift	$\alpha$ =1	no shift	$\alpha$ =1	no shift	$\alpha$ =1	no shift	$\alpha$ =1
Jev	0.0	$-$ 0.1	+0.6	0.0	+0.1	$-$ 1.8	+0.4	$-$ 1.7	$-$ 0.2	$-$ 1.1
Tev1-4B	+0.1	0.0	+0.5	+0.1	+0.1	$-$ 0.9	+0.3	$-$ 4.6	+0.4	0.0
Nimble-9B	+0.2	+0.1	+0.5	+0.3	0.0	$-$ 2.8	$-$ 0.2	$-$ 1.9	$-$ 0.6	$-$ 0.6
Qwen3.5-4B	+0.1	$-$ 0.1	+0.4	$-$ 0.1	$-$ 1.5	$-$ 3.1	+0.4	$-$ 3.6	$-$ 0.3	$-$ 0.7
Qwen3.5-9B	+0.1	$-$ 0.2	+0.4	$-$ 0.2	+0.3	$-$ 2.0	+0.1	$-$ 1.6	$-$ 0.3	$-$ 0.6

Appendix

Table 18: Without shift, EM changes the calibration error of most systems by at most 0.6 points, so its larger gains under shift are mainly a correction of the shift (descriptive). Exceptions are Qwen3.5-4B on FPB (

-1.5

) and TF-IDF on HWU64 (

-2.5

), where EM also reduces miscalibration that exists without shift, and TF-IDF on TREC-6 (

+0.8

). Change in ECE (points) from EM prior correction (implicit prior for zero-shot systems, training prior for trained classifiers) relative to raw probabilities, on the resampled no-shift control and at

\alpha=1

; mean over 200 draws each.

	Qwen3.5-9B predicted			Nimble-9B predicted
Gold	negative	neutral	positive	negative	neutral	positive
negative (125)	111	12	2	119	3	3
neutral (565)	42	364	159	78	202	285
positive (280)	7	27	246	7	4	269
Total predicted	160	403	407	204	209	557

Appendix

Table 19: On FPB, the Nimble adapter moves neutral sentences to positive and negative (descriptive). Confusion matrices on the 970 FPB test sentences for Qwen3.5-9B and Nimble-9B (same base weights, prompts and readout); rows: gold label (count), columns: predicted label.

System	CLINC150	HWU64	FPB	HateXplain	TREC-6
Jev	0.014	0.052	0.184	0.232	0.060
Tev1-4B	0.020	0.068	0.230	0.233	0.115
Nimble-9B	0.024	0.068	0.368	0.268	0.043
Qwen3.5-4B	0.033	0.084	0.381	0.238	0.052
Qwen3.5-9B	0.023	0.080	0.169	0.229	0.029

Appendix

Table 20: The implicit prior is close to the true class distribution on the intent datasets and far from it on FPB and HateXplain, where the models have a label bias. Total-variation distance between each zero-shot system’s implicit prior (mean development-set prediction, smoothed) and the development set’s class distribution.

System	F1 (all 1076)	F1 (non-overlap 462)	ECE (all)	ECE (non-overlap)
Jev	0.908	0.883	0.036	0.047
Tev1-4B	0.895	0.872	0.040	0.046
Nimble-9B	0.896	0.860	0.035	0.042
Qwen3.5-4B	0.866	0.842	0.058	0.056
Qwen3.5-9B	0.884	0.857	0.078	0.085
DeBERTa-v3-large	0.939	0.904	0.030	0.027

Appendix

Table 21: Removing the HWU64 examples that also occur in MASSIVE lowers every system’s Macro-F1 but leaves the P2 calibration result intact. Test Macro-F1 and ECE on all 1,076 examples and on the 462 non-overlapping ones, and the P1/P2 paired bootstrap on the subset (same

B

and seed as the primary tests; unadjusted).

Item	Status	Where
B1 label-probability baseline from an ordinary LM	yes	Qwen3.5-4B/9B with the same prompts and letter readout ( Section 4 )
B2 cheap generative cascade	not applicable	we study single models, not cascades
B3 trained task-specific classifier	yes	DeBERTa-v3-large and TF-IDF + LR on full training data
T1 model spread; no parity claims on saturated tests	yes	Table 4 ; no equivalence claims
T2 agreement across repeats	yes, for Jev	Table 10 ; local models repeated identically in a 50-example check
T3 option-name invariance	partly	fixed candidate order and label texts for all systems; no renaming study

Appendix

Table 22: The study meets 11 of the 14 items of the TDM evaluation checklist of Tang & Zheng (2026) fully, two partly and one is not applicable.

Language Model Calibration

Language model calibration studies whether models’ confidence estimates match their actual accuracy across tasks, prompts, and deployment settings.

See all 383 papers

Sep 26, 2026

Typed Decision Models: An Early Evidence Audit and Evaluation Checklist

Typed decision models (TDMs) return probability distributions over caller-defined options without generating text. TypeSafe released Jev, a commercial typed decision model, on 15 September 2026, and a small body of evaluation and replication work appeared within days. We review 28 papers posted between 19 and 24 September and relate their findings to earlier work on label-probability classification, constrained decoding, reranking, calibration, and model cascades. In this early literature, the typed readout itself has not shown an independent accuracy advantage over comparable label-probability readouts. Jev's clearest gains are in latency and cost, while accuracy gaps remain on harder tasks. In practical deployments, confidence is often used to decide when to defer to a stronger model or a human. We use recurring weaknesses in these studies to derive a 14-item evaluation checklist for future TDM work. Because the evidence covers only the first nine days after the release of one hosted model, the review should be read as an early evidence map rather than a settled assessment of the model class.

Lijuan Tang, Yuemeng Zheng

Northeastern University

LLM Evaluation

Ask PDF

Oct 25, 2026

SHIFT-JEV: Certified Abstention and Gain-Gated Continual Adaptation for LLM-Based Decision Models under Distribution Shift

Large language models (LLMs) can act as decision models that return a probability distribution over a fixed set of options, but deployed decision models face domain, label-prior, emerging-class and candidate-set shift. We study how such models fail under shift and propose SHIFT-Jev, a monitoring, abstention and continual-adaptation procedure for them. It scores the risk of each decision from uncertainty, format and option-consistency signals, certifies abstention thresholds with Learn-then-Test, monitors the decision stream with drift and label-shift detectors, and accepts a model update only when an expected-gain gate finds a significant held-out improvement without loss on earlier regimes. On a benchmark of ten shifted test environments with Qwen3-0.6B and Qwen3-4B (three seeds each) and Phi-4-mini, fine-tuning with KL anchors and temperature scaling lowers calibration error under shift from 0.289 to 0.087. At a target risk of 10%, a learned uncertainty score with a certified threshold exceeds the target in 3 to 5 of 27 environment-seed tests, against 5 to 15 for a max-probability score with the same certification and 9 to 14 for conformal prediction sets; drift-based routing changes little. A chi-squared test on predicted labels detects 95 to 99% of class-prior shifts that confidence- and embedding-based detectors miss. In continual adaptation, SHIFT-Jev reaches 70.5% accuracy on Qwen3-0.6B, above replay, LwF, EWC and fine-tuning baselines in every seed, whereas on Qwen3-4B its conservative gate leaves it 0.4 to 1.4 points behind them.

Muhammad Atif Saeed, Azka Atiq, Adarsh Kumar +3

Department of AI & Data Science, National University of Computer & Emerging Sciences, Islamabad, Pakistan · Data Engineer, University of Arkansas at Little Rock, Arkansas, USA · Department of Computer Science, National University of Computer & Emerging Sciences, Islamabad, Pakistan +2

Online Language Model Adaptation Selective Prediction Domain Adaptation for LLMs Language Model Calibration Continual Learning for LLMs Label Shift

Ask PDF

Oct 1, 2026

Labels Override Definitions in Jev-Style Typed Decision Models

A typed decision model answers a fixed question about an input by returning a probability for each of several caller-defined options. Each option carries a short label and a written definition, which is where a developer states the rule the model should apply. Jev introduced this interface for routing, moderation and triage, open implementations followed, and the same operation occurs whenever a language model is used as a classifier by scoring label strings. We study the open implementations, whose weights we can inspect and patch, and ask whether the probability follows the definitions or the labels. A preference for the label we call option-label bias. Across four open-weight typed decision models, three ways of reading an answer from a Qwen2.5 backbone, eleven classification tasks and PolicyBench, a synthetic routing suite we introduce in which the rule appears only in the definitions, the answer is mostly the labels. Deleting every definition leaves accuracy unchanged (laya-td: 0.8559 against 0.8487), although those definitions support 0.7971 on their own, and renaming the options to A and B raises accuracy by +0.1511 [+0.1377, +0.1646]. One system, von, is unaffected, and the two code bases differ in one expression: laya writes each option as "{label}: {definition}", while von writes only the definition. Changing that expression in both directions, with no weight changed, makes all three laya checkpoints exactly invariant (+0.0000 [+0.0000, +0.0000]) and creates the effect in von, whose accuracy falls from 0.8511 to 0.2281 when a label contradicts its definition. Earlier work attributed this failure to the constrained decision head these models use in place of a text decoder; our results locate it in the prompt rendering. We give a two-call test that tells a practitioner which case applies to their model, and measure what four mitigations are worth.

Seyedarmin Azizi, Erfan Baghaei Potraghloo, Massoud Pedram

University of Southern California (USC), USA

Language Model Bias Evaluation Prompt Sensitivity

Ask PDF

Abstract

Language Model Calibration

Explore similar work

Typed Decision Models: An Early Evidence Audit and Evaluation Checklist

SHIFT-JEV: Certified Abstention and Gain-Gated Continual Adaptation for LLM-Based Decision Models under Distribution Shift

Labels Override Definitions in Jev-Style Typed Decision Models