Typed Decision Models: An Early Evidence Audit and Evaluation Checklist

Lijuan Tang; Yuemeng Zheng

cs.CLSep 26, 2026

Typed Decision Models: An Early Evidence Audit and Evaluation Checklist

Authors: Lijuan Tang, Yuemeng Zheng

Organizations: Northeastern University

Abstract

Typed decision models (TDMs) return probability distributions over caller-defined options without generating text. TypeSafe released Jev, a commercial typed decision model, on 15 September 2026, and a small body of evaluation and replication work appeared within days. We review 28 papers posted between 19 and 24 September and relate their findings to earlier work on label-probability classification, constrained decoding, reranking, calibration, and model cascades. In this early literature, the typed readout itself has not shown an independent accuracy advantage over comparable label-probability readouts. Jev's clearest gains are in latency and cost, while accuracy gaps remain on harder tasks. In practical deployments, confidence is often used to decide when to defer to a stronger model or a human. We use recurring weaknesses in these studies to derive a 14-item evaluation checklist for future TDM work. Because the evidence covers only the first nine days after the release of one hosted model, the review should be read as an early evidence map rather than a settled assessment of the model class.

Figures & tables

TDM component	Ancestor	What TDMs change
Scoring a label set by LM probability	Cloze/verbalizer classification [ 20 , 57 ] ; likelihood scoring [ 10 ]	Options declared per request rather than fixed verbalizers
Option-label readout	Multiple-choice symbol binding [ 55 ]	Named options may include rubrics rather than letters
Guaranteed valid output	Grammar-constrained decoding [ 22 , 69 ]	Returns a renormalized distribution without token generation
Fine-tuned label-token head	Yes/no rerankers [ 48 , 82 ] ; generative verifiers [ 81 ]	Arbitrary option sets; many questions per call
Runtime labels in small encoders	NLI zero-shot classifiers [ 35 , 75 ] ; GLiClass [ 60 ]	Presented as a general decision interface
Rubric-conditioned judging	Prometheus [ 34 ] ; G-Eval [ 41 ]	Rubric attached to each option

Table 1: Components of a TDM and their closest ancestors.

Model	Backbone	Readout	Training
this-that-model-1.0 [ 12 ]	1.88B hybrid linear-attention decoder	Answer-slot hidden state scored against output-embedding rows of single-token option labels; restricted softmax	Full fine-tune; cross-entropy + Brier
JevLite [ 54 ]	Qwen3-4B, LoRA	Softmax over allowed label-token logits	Cross-entropy + Brier; temperature scaling
Visual Jev [ 76 ]	Qwen3-VL-4B/8B, LoRA	LM-head logits of option letters, renormalized over candidates	Answer SFT; dedicated typed head tried and dropped
SemIf-4B [ 28 ]	Qwen3.5-4B, frozen	Option-letter logits	None
Qwen3-0.6B-RLCD [ 28 ]	Qwen3-0.6B-Base	Letter-token readout	RL for calibrated decisions (third-party replication)
PixelJev [ 86 ]	Qwen3.5-2B/4B, optional LoRA	Option-letter logits at the first answer position, restricted to the candidate slots	Few-shot LoRA; temperature fitted on held-out data

Table 2: Open TDMs with documented readouts. n.r.: not reported in the corpus.

Role	Papers	Where low confidence goes
Step replacement	Edge/6G orchestration [ 36 , 37 ] ; scientific choices [ 16 ] ; contract inference [ 79 ] ; video quality [ 56 ] ; standalone visual and grid-world questions [ 76 , 86 , 12 ] ; numerical decoding [ 74 ] ; mobile GUI actions [ 80 ] ; edge triage [ 14 ]	Planner when the TDM selects a done/blocked option [ 80 ] ; not reported in the other studies
Controller	Tool-using agent [ 71 ] ; memory control [ 29 ] ; game control [ 44 ] ; coding-agent routing [ 1 ]	Strong LLM by confidence [ 1 , 71 ] ; planner by schedule [ 44 ] ; not reported in the other studies
Judge or screener	Preference/factuality judging [ 39 ] ; rubric judging [ 53 ] ; radiology reports [ 27 ] ; scam calls [ 54 ] ; alignment-failure detection [ 24 ] ; security findings [ 18 ]	Stronger judge by confidence [ 39 ] ; LLM judge by confidence, replayed [ 53 ] ; human review of needs-review findings by design, not exercised in the evaluation [ 18 ] ; not reported in the other studies
Measurement instrument	Social-science annotation [ 28 ] ; crash narratives [ 52 ] ; population simulation [ 38 ]	LLM [ 28 ] or human [ 52 ] by confidence; flagship model on fixed sample [ 38 ]
Diagnostic	Option-name polarity [ 61 ] ; context flipping [ 73 ] ; prompt injection [ 70 ]	–
Ecosystem	Public GitHub projects [ 40 ]	–

Table 3: Corpus papers by the role of the TDM and where low-confidence decisions go.

Paper	Comparator / control	Accuracy (TDM vs. comparator)	Latency	Cost	Calib.	LP	Stats
Open and alternative TDMs
Cheng et al. [12]	GPT-5.6	0.844 vs. 0.897 (own bench); 0.941 vs. Jev 0.765 (68 items)	30.9 ms local	local (electricity)	Yes	?	none
Ren et al. [54]	same backbone, label token	AUROC .974 vs. .950 (n.s.)	64.5 ms	n.r.	Yes	Yes	bootstrap, 3 seeds
Yu and Yao [76]	typed-head control (default: LM head)	macro acc. .761 vs. .761	8.9 $\times$ faster ∗	n.r.	appendix	Yes	3 seeds (SD); bootstrap described, CIs not reported
Zhou et al. [86]	same backbone, greedy generation	readout = generation when valid; +1.88 pp (adapted, ScienceQA) only from invalid generations	1.3–1.4 $\times$ faster than gen. (1.03 $\times$ vs. one-token constrained gen.)	n.r.	Yes	Yes	group bootstrap, 3 seeds
Dağlı et al. [14]	internal ablations; random	JevBench 55.4% vs. 31.8% random (v2, self-reported)	$\approx$ 2–9 ms (inconsistent)	estimates only	No	No	CIs on in-house tests only

Table 4: The TDM corpus, part 1: open and alternative TDMs, step replacement and controllers (arXiv, 19–24 Sept. 2026). Calib. : TDM probabilities evaluated for calibration. LP : at least one baseline reads label log-probabilities from an ordinary or same-backbone LM; ?: unclear. n.r.: not reported. ∗ Amortized time per question at 32 questions per image versus serial execution. ∗∗ Median decision latency; the median paired gap falls below 1 ms once repeated requests are cached.

Paper	Comparator / control	Accuracy (TDM vs. comparator)	Latency	Cost	Calib.	LP	Stats
Judges and screeners
Li et al. [39]	GPT-6 Astra	RewardBench $-$ 1.3; JudgeBench $-$ 14.6; RM-Bench hard $-$ 19.8	0.15 vs. 1.89 s	0.36% of fee	Yes	No	cluster bootstrap
Rao and Callison-Burch [53]	three flash-tier LLM judges	differs in 8/27 comparisons (4 ahead, 4 behind; graded up to $-$ 16.4)	0.18–0.20 s	29–325 $\times$ less	Yes	No	bootstrap, Holm
Huang et al. [27]	RadMatch (local)	$\tau_{b}$ .573 vs. .580; sig. errors $-$ .145; 200-pair RadEvalExpert subset .322 vs. .453	434 ms/req. (median, Full)	2.3–2.6 cents/100 pairs	Yes	No	bootstrap; 5-run repeatability on 50 pairs
Guo et al. [24]	in-domain TF-IDF classifier (accuracy); LLM judges (cost)	median AUROC .886 vs. .754	0.31 s	63 $\times$ less at list prices; 6 $\times$ under repricing	Yes	No	bootstrap
dos Santos Barbosa [18]	same harness without TDM	9/13 vs. 10/13 scenarios (1 run each)	26.9 vs. 32.2 min/run	$<$ $5 TypeSafe API	No	No	none ( $n=1$ per arm)

Table 5: The TDM corpus, part 2: judges and screeners, measurement instruments and diagnostics. Calib. : TDM probabilities evaluated for calibration. LP : at least one baseline reads label log-probabilities from an ordinary or same-backbone LM. n.r.: not reported. ∗∗∗ Against list-price scenarios for frontier models, which were not metered.

Baselines
ID	Report	Why it matters / failure observed	Count
B1	A baseline that reads label log-probabilities from an ordinary or same-backbone LM.	Otherwise interface gains cannot be separated from model gains; the same backbone is the most informative choice; where tested, the readout added little accuracy (Finding 1).	5/26; 1 unclear
B2	Where a confidence threshold passes decisions to another model: a cheap generative cascade as a baseline.	On external $\tau^{2}$ -bench a cheap generative cascade was cheaper than the TDM cascade and not less accurate [ 71 ] ; two of six open-weight LLMs in Jev’s price range had higher median macro-F1 [ 28 ] .	1/5
B3	Where labeled data suffice to train one: a task-specific model (encoder, classical classifier, reward model or probe) as a baseline.	Fine-tuned encoders were not significantly worse than an open TDM in ranking and had nominally lower ECE (but higher Brier) and were faster [ 54 ] ; trained quality models stayed well ahead [ 56 ] .	7/13; 3 unclear
Test design
T1	Spread between models on the test; avoid parity claims on saturated tests.	Several parity results come from tests on which strong models are at or near ceiling (Finding 2).	–

Table 6: Evaluation checklist for TDMs. Count : compliant papers over papers to which the item applies, among the 27 papers that evaluate a model (criteria in Appendix B ); papers coded as unclear are excluded from both counts; “–” marks disclosures not reliably scorable from the text.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Paper	arXiv ID	Version	Version date
Li et al. [36]	2609.22753	v1	19 Sep 2026
Li et al. [37]	2609.23136	v1	19 Sep 2026
Cheng et al. [12]	2609.23886	v1	20 Sep 2026
Ren et al. [54]	2609.23959	v1	21 Sep 2026
Jiang et al. [29]	2609.23986	v1	21 Sep 2026
Rafe and Das [52]	2609.24052	v1	21 Sep 2026

Appendix

Table 7: arXiv versions of the 28 corpus papers used in this review.

Paper	B1	B2	B3	T2	T3	R2	C1	C3	C4
Cheng et al. [12]	?	–	✗	✗	✗	✗	✓	–	✗
Ren et al. [54]	✓	–	✓	✓	✗	✓	✓	✓	✓
Yu and Yao [76]	✓	–	✗	✓	✗	✗	✓	✓	✓
Zhou et al. [86]	✓	–	✓	✓	✗	✓	✓	–	✗
Dağlı et al. [14]	✗	–	✓	✓	✗	✗	✗	–	✗
Li et al. [36]	✗	–	?	✓	✗	✗	–	–	✓

Appendix

Table 8: Per-paper checklist coding. ✓: compliant; ✗: applicable but not compliant; ?: unclear; –: not applicable.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Confidence Is Not a Promise: A Reference Audit of Typed Decision Models
Oct 26, 2026Pedro Carvalho BromLLM Reliability LLM Auditing
Department of Mathematics, Federal Institute of Brasília, Brasília, Brazil
Labels Override Definitions in Jev-Style Typed Decision Models
Oct 1, 2026Seyedarmin Azizi, Erfan Baghaei Potraghloo, Massoud PedramLanguage Model Bias Evaluation Prompt Sensitivity
University of Southern California (USC), USA
Typed Decision Models under Prevalence Shift: Decision Training, Calibration, and Implicit Priors
Oct 19, 2026Abdul Rahman Iqbal, Muhammad Fawad Nasim, Amir SaleemLanguage Model Calibration Model Calibration
Faculty of Computer Science and Information Technology, The Superior University, Lahore, Pakistan · Independent Researcher

Oct 26, 2026J/K move · Enter open · S save

Confidence Is Not a Promise: A Reference Audit of Typed Decision Models

Pedro Carvalho Brom

Department of Mathematics, Federal Institute of Brasília, Brasília, Brazil

LLM Reliability LLM Auditing Confidence Estimation in Language Models Selective Prediction

Open paper AskPDF arXiv

Typed decision models return a probability distribution over options supplied by the caller, and software tends to act on that number as if it were a guarantee. We measure what it licenses, comparing the open-weight Laya, in five checkpoints, with Jev 1.13, available only through an API. A probability licenses a statement about groups of decisions, not about one decision, and a gate that promises an accuracy among the cases it decides alone keeps that promise only with a lower bound checked on data like its traffic. Across 190 exchangeable evaluations, a threshold chosen by point estimate breaks the promise in 147, Learn then Test keeps it in all of them, and a Beta lower-bound rule decides more items on average than Learn then Test in 128 and fewer in none, without its guarantee. Calibrated in English, Jev keeps a promise of 95% in 8 of 51 locales. Ordinary typing noise breaks a promise calibrated on clean text, and, in exploratory runs, training with noise raises the noisy accuracy with no loss on clean text. In four further domains, a fine-tuned Laya needs from 250 to 8,000 labels to be non-inferior to Jev, and does not get there on LIAR2. Under drift, the preregistered claim for a Bayesian retraining loop is not supported: the loop avoids needless retraining where the change does not degrade the model, at a cost equivalent to recalibration alone, and re-qualifying the threshold in every window keeps the promise more often where it does. On the monthly series of M3, neither decider predicts the direction of the next value better than a constant answer. Confidence separates right from wrong answers with an AUROC from 0.80 to 0.96, but it does not select faithful explanations, and exact coalition games of single decisions show what an explanation can still say through an API that rounds to two decimals. We release the frozen answers with a datasheet, a promise check and the code.

Abstract

LLM Evaluation

Explore similar work

Confidence Is Not a Promise: A Reference Audit of Typed Decision Models

Labels Override Definitions in Jev-Style Typed Decision Models

Typed Decision Models under Prevalence Shift: Decision Training, Calibration, and Implicit Priors