How Reliable Are System-One Decisions in Medicine? A Comparative Study of Jev, Jev-like Models, and LLMs

Yanrui Du; Danyang Zhao; Sendong Zhao; Wenhao Li; Yibo Gao; Haochun Wang; Zhuoer Xu; Shiwen Cui

cs.CLOct 13, 2026

How Reliable Are System-One Decisions in Medicine? A Comparative Study of Jev, Jev-like Models, and LLMs

Authors: Yanrui Du, Danyang Zhao, Sendong Zhao, Wenhao Li, Yibo Gao, Haochun Wang, Zhuoer Xu, Shiwen Cui,

Organizations: Harbin Institute of Technology · Ant Group

Abstract

Jev has attracted considerable attention for its rapid, generalizable decision-making, yet its medical reliability remains underexplored. To address this gap, our study compares various types of system-one decision models across 15 medical datasets comprising 8,742 instances under a unified protocol. Beyond accuracy, we assess decision stability and the ability to recognize unreliable decisions. Our results show that Jev rivals leading models on primarily text-driven tasks but is less effective on numerically driven tasks, revealing a distinct capability boundary. Under question-format variations, Jev maintains strong decision consistency and small accuracy fluctuations. StartLux-27B exhibits comparable consistency, while D3-27B achieves higher overall accuracy and a lower mean decision reversal rate. But these strengths do not extend to reliable abstention awareness: most evaluated models show limitations in both recognizing missing correct answers and using optional abstention. Moreover, we find that Jev's probability-based answer selection can be improved by changing how decisions are scored: normalizing option-wise binary scores yields higher retained accuracy than choice scoring. Taken together, our findings provide early guidance for improving and applying decision models in medicine.

Figures & tables

Figure 1: Overview of the comparative evaluation. Three model families, four medical task families, and four evaluation dimensions. A shared clinical judgment case illustrates performance, format changes, missing-answer recognition, optional abstention, and probability-based selection.

Jev
Model	A.Triage and Navigation	B.Clinical Judgment	C.Safety Review	D.Physiological Assessment	Overall
Jev-1.13	55.9 / 50.8	74.9 / 74.3	59.8 / 58.9	35.9 / 27.2	61.6 / 58.7
Jev-1.14	54.4 / 50.1	76.8 / 76.2	58.9 / 58.0	39.4 / 32.0	62.3 / 59.7
General-purpose LLMs
Qwen3.5-0.8B	26.3 / 11.7	39.7 / 35.0	38.3 / 27.9	53.4 / 33.8	37.6 / 27.2
Qwen3.5-4B	43.2 / 35.7	60.4 / 59.7	51.4 / 45.9	54.8 / 39.1	53.2 / 47.8

Table 1: Original-choice accuracy / macro-F1 (%), averaged equally across datasets. Bold indicates column-wise best scores. More detailed dataset-level results appear in Figure 4 in Appendix E .

Format	Candidate presentation	Probability readout	Identity alignment
Original	A: appendicitis; B: cholecystitis; C: pancreatitis; D: diverticulitis.	Joint candidate probabilities $p_{k}$ ; select $\arg\max_{k}p_{k}$ .	–
Relabeling	1: appendicitis; 2: cholecystitis; 3: pancreatitis; 4: diverticulitis.		1 $\to$ A, 2 $\to$ B, 3 $\to$ C, 4 $\to$ D.
Reordering	A: pancreatitis; B: diverticulitis; C: appendicitis; D: cholecystitis.		A $\to$ C, B $\to$ D, C $\to$ A, D $\to$ B.
Binary (OBS)	Original A–D candidates retained. For each candidate: “Is [candidate] the correct diagnosis?” Yes / No.	Affirmative $q_{k}$ ; normalize $\tilde{p}_{k}=q_{k}/\sum_{j}q_{j}$ ; select $\arg\max_{k}\tilde{p}_{k}$ .	–

Table 2: Paired format interventions for one synthetic case: CT shows an inflamed appendix; what is the most likely diagnosis? Every row retains the same evidence and candidate meanings. The final column maps presented candidate identities to original identities.

Jev
Model	$\Delta$ Rel.	$\Delta$ Reord.	$\Delta$ OBS	$V_{\Delta}$	$F_{\mathrm{Rel}}$	$F_{\mathrm{Reo}}$	$F_{\mathrm{OBS}}$	$\overline{F}$
Jev-1.13	+0.45	-0.03	+0.05	1.70	3.9	6.3	8.3	6.2
Jev-1.14	+0.19	+0.03	+0.19	1.09	3.4	6.5	7.1	5.7
General-purpose LLMs
Qwen3.6-27B	-0.17	-0.34	-3.55	18.29	5.4	9.2	16.1	10.2
Qwen3.8-27B	+0.40	-0.20	-3.84	19.79	5.1	10.3	16.9	10.7

Table 3: Format robustness across all 15 datasets, with equal dataset weights.

\Delta

Rel.,

\Delta

Reord., and

\Delta

OBS denote accuracy changes (percentage points) under relabeling, reordering, and OBS, respectively.

V_{\Delta}

denotes mean within-dataset accuracy variance (squared percentage points).

F_{\mathrm{Rel}}

F_{\mathrm{Reo}}

, and

F_{\mathrm{OBS}}

denote prediction reversal rates (%) under the corresponding interventions and

\overline{F}

is their mean. Alongside both Jev versions, the three general LLMs, one medical LLM, and five Jev-like models with highest overall accuracy in Table 1 are shown. Table 15 reports all 23 models.

Jev
Model	Original Acc	NONE Acc	$\Delta$ Acc	Slot flip	Correctness flip	NONE Acc (orig. correct)
Jev-1.13	77.6	65.6	-12.0	18.4	16.2	70.1
Jev-1.14	77.7	66.3	-11.4	18.2	16.0	71.0
General-purpose LLMs
Qwen3.6-27B	78.9	69.0	-9.9	16.2	15.4	73.7
Qwen3.8-27B	78.6	68.4	-10.2	17.5	15.8	72.8

Table 4: Correct-answer replacement across four Clinical Judgment datasets (2,382 paired entries). Accuracy and reversal rates are percentages.

\Delta

Acc is NONE accuracy minus original accuracy in percentage points. NONE Acc (orig. correct) measures accuracy after replacement on items each model originally answered correctly. Model selection follows Table 3 ; complete results appear in Table 17 .

Jev
	Original Choice		With Abstention		Gains
Model	$A^{0}_{\mathrm{all}}$	$A^{0}_{S}$	$R$	$A^{+}_{S}$	$\Delta$	$\Delta_{S}$
Jev-1.13	74.9	75.5	4.5	75.2	+0.3	-0.3
Jev-1.14	76.8	77.4	4.4	77.3	+0.5	-0.1
General-purpose LLMs
Qwen3.6-27B	72.9	73.5	2.8	73.2	+0.3	-0.3

Table 5: Optional abstention across six Clinical Judgment datasets (2,501 entries).

A_{\mathrm{all}}^{0}

and

A_{S}^{0}

are original accuracy on the full cohort and retained subset.

R

is the mean abstention rate and

A_{S}^{+}

the mean retained accuracy (all in %). Gains

\Delta

and

\Delta^{S}

are in percentage points, computed before rounding. Model selection follows Table 3 ; complete results appear in Table 18 .

Figure 2: Jev 1.13’s probability-based selection. Choice (blue) and OBS (orange) compared through (a) threshold accuracy, (b) accuracy and coverage at 0.90, (c) accuracy at matched global coverage, and (d) confidence distributions (bin width 0.05). Accuracy pools retained entries; coverage uses all 8,742 entries. These are selection results, not calibration curves.

Figure 3: Jev 1.14’s probability-based selection. Choice (blue) and OBS (orange) compared through (a) threshold accuracy, (b) accuracy and coverage at 0.90, (c) accuracy at matched global coverage, and (d) confidence distributions (bin width 0.05). Accuracy pools retained entries; coverage uses all 8,742 entries. These are selection results, not calibration curves.

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Dataset	$N$	$K$	Input evidence	Classification target
A	CARE	135	4	Chronological patient messages	Clarify / self-care / nonurgent care / urgent care †
	ESI	1,000	5	Triage records and clinical vignettes	ESI 1 / ESI 2 / ESI 3 / ESI 4 / ESI 5 †
	MIMIC-IV-ED	1,000	5	Chief complaint, demographics, pain, vital signs	ESI 1 / ESI 2 / ESI 3 / ESI 4 / ESI 5 †
	Ramaswamy	960	4	Care-seeking narratives with available measurements	Home care / care within weeks / care within 24–48 hours / immediate ED †
	A subtotal	3,095			Triage and Navigation
B	ICD-proxy	1,000	4	Admission-related narrative excerpts	ICD candidates A / B / C / D ‡ ; e.g., D: J15 (bacterial pneumonia)

Appendix

Table 6: Benchmark composition and input evidence. Inputs include task instructions, candidate answers, and applicable definitions or constraints.

N

denotes the number of entries, not independent patients;

K

denotes the number of candidates in the baseline task.

\dagger

indicates fixed class meanings within a dataset, while

\ddagger

indicates question-specific candidates, with illustrative options shown in the table. Full task examples and adaptation details are provided in Appendix C .

Dataset / target	Example input (abridged)	Expected benchmark output
CARE / 4 actions	Ongoing root-canal treatment with severe tooth pain; the disclosed follow-up denies fever, facial swelling, pus, swallowing or breathing difficulty. Choose A: clarify; B: self-care; C: nonurgent care; D: urgent evaluation.	C: recommend nonurgent professional care.
ESI / 5 levels	An eight-month-old with fever, cough, vomiting, reduced intake, listlessness, and a sunken fontanel; respirations are regular and unlabored. Apply the supplied ESI decision rules. A–E correspond to ESI 1–5.	B: ESI 2, from the TriageAgent scenario reference.
MIMIC-IV-ED / 5 levels	Initial complaint: burn. Age 30, male, walk-in; temperature 98.2 F, pulse 75/min, respirations 14/min, oxygen saturation 100%, BP 124/77 mmHg, pain 4/10. Predict the recorded ESI among 1–5.	E: recorded ESI 5. This is a historical nurse label, not a newly inferred clinical gold standard.
Ramaswamy / 4 destinations	Two resolved episodes of unilateral arm numbness; currently normal neurologic examination and listed normal glucose. Options: A home; B physician in weeks; C physician within 24–48 hours; D immediate ED.	C or D is accepted in this source item; the model still selects one option.

Appendix

Table 7: A: Triage and Navigation, 3,095 entries. Output labels preserve each example’s original letter mapping. These tasks assess care urgency or routing, not a full differential diagnosis.

Dataset / target	Example input (abridged)	Expected benchmark output
ICD-proxy / 4 categories	Admission-related narrative describes cough, dyspnea, feverishness, and severe multifocal pneumonia. Candidates: A G40 epilepsy; B S02 skull/facial fracture; C S82 lower-leg fracture; D J15 bacterial pneumonia.	D: J15, the source’s first-listed three-character ICD category.
MedCalc / binary flag	Original question asks for admission GCS before treatment. Note describes no eye opening, incomprehensible vocalization, and abnormal flexion to pain. Supplied rules yield $1+2+3=6$ ; test whether GCS $\leq 8$ . A: flag absent; B: flag present.	B: threshold flag met. This does not itself specify airway treatment.
MedQA / 4 alternatives	Recurrent springtime itchy, watery eyes and sneezing; bilateral conjunctival injection, normal visual acuity, and unremarkable anterior chamber. Best treatment? A erythromycin; B ketotifen; C warm compresses; D fluorometholone.	B: ketotifen eye drops, retaining the source examination answer.
MedXpertQA / 10 alternatives	An oriented 19-year-old describes a specific plan to kill his father. A police officer requests the threatened person’s identity. The source asks for the most appropriate next step among ten original responses, including B: identify the father to the officer.	B: the original examination reference. All ten alternatives are retained during evaluation.
MIMIC-CDM / 4 diagnoses	Right upper quadrant and epigastric pain; focal tenderness; ultrasound describes gallbladder distension, stones/sludge, and wall thickening. The actual input also includes the remaining work-up. A appendicitis; B cholecystitis; C diverticulitis; D pancreatitis.	B: cholecystitis. This is a static diagnosis decision, not tool selection.
SCT / 5 evidence updates	A febrile three-week-old infant is being evaluated. Hypothesis: starting acyclovir. New evidence: a brief seizure-like episode. Choose change in appropriateness from $-2$ to $+2$ (A–E).	E ( $+2$ ) has maximum expert credit 1.00; D ( $+1$ ) receives partial credit 0.25, not an accuracy hit here.

Appendix

Table 8: B: Clinical Judgment, 2,501 entries. Different tasks retain examination answers, coding proxies, recorded diagnoses, explicit numerical rules, or expert-credit references; they are not interchangeable gold-standard constructions.

Dataset / target	Example input (abridged)	Expected benchmark output
MEDEC / error detection	Clinical note describes fever, diarrhea, abdominal pain and a rash after freshwater activities during travel to Brazil, and attributes the presentation to hepatitis A. Judge medical correctness: A no error; B an error is present.	B: medical error present, as labeled in the source. No localization or correction is required.
MedVAL / 4 risk grades	Source dialogue discusses urinary symptoms, a prostate evaluation, laboratory tests, Flomax and specialist referral. The candidate note additionally asserts a suspected urinary infection and an antibiotic course. The complete original and candidate documents are supplied for review. A no risk; B low; C moderate; D high.	C: moderate document risk; expert review required, according to the physician-assigned source grade.
RxSafe / 3 medication candidates	Consultation describes hepatitis C and concurrent atorvastatin use. Choose an appropriate medication while considering the described interaction risk. Candidates: A Harvoni; B Mavyret; C hyoscyamine.	A: Harvoni, the source reference among these candidates; not a general claim that the drug has no interaction risk.

Appendix

Table 9: C: Safety Review, 1,812 entries. The reviewed object is a clinical assertion, generated document, or candidate medication. Document risk is distinct from illness severity; candidate selection is distinct from unrestricted prescribing.

Dataset / target	Example input (abridged)	Expected benchmark output
CTG-21 / 3 states	Fetal heart-rate baseline 140; AC 0; FM 0.00142; UC 0.00284; DL 0.00709; DS/DP 0; ASTV 62; MSTV 1.2; ALTV 0; MLTV 3.9; histogram width 93, minimum 66, maximum 159, peaks 5, zeros 2, mode 142, mean 136, median 144, variance 26, tendency 1. Definitions and source units accompany every field.	A: normal fetal state. Other candidates are B: suspect and C: pathologic. All 21 features, not a raw CTG image, are used.
VitalDB / future event	An intraoperative case supplies age/sex and 60 pre-decision ten-second MAP medians, beginning $(-600,84),(-590,86),(-580,85),\ldots$ in seconds/mmHg. Predict whether the next five minutes contain six consecutive bins below 65 mmHg. A: no; B: yes.	A: no qualifying future episode in this item. The reference is derived from future measurements that are not part of the input.

Appendix

Table 10: D: Physiological Assessment, 1,334 entries. CTG values are rounded here for readability; evaluation preserves source precision. VitalDB is a probabilistic forecast with an observed future label, not a deterministically implied outcome from the abbreviated history.

Family	Evaluated configuration	Source	Decision readout
Jev	Jev-1.13 Jev-1.14	TypeSafe ( Almeida, 2026 )	C: native choice probabilities; B: native noul probabilities.
Jev-like	GLiNER2.5-Decide	Fastino ( fastino, 2026 )	C: candidate-logit softmax; B: two-class-logit softmax.
	SemIf	TheoLeeCJ ( TheoLeeCJ, n.d. )	C: candidate-token softmax; B: two-candidate-token softmax.
	Kev-4B	Jared Palmer ( jaredpalmer, 2026 )	C: native decision-head probabilities; B: native noul probabilities.
	AnyJev (raw)	Nokia Applied Research ( nokia-applied-research, n.d. )	C: candidate-slot softmax; B: native yes/no probabilities.
	OpenJev	OpenJev ( razorback16, n.d. )	C: answer-slot softmax; B: binary answer-slot softmax.

Appendix

Table 11: Model provenance and evaluated readouts. Multi-size rows identify separate configurations, totaling 23 models. C denotes the joint Choice distribution; B denotes the binary distribution for one candidate judgment. OBS extracts B’s affirmative probability for each candidate and normalizes across candidates. All LLMs use the same fixed-prefix, single-token-label protocol with model-specific chat templates. Backbone and adapter details appear in Appendix D .

Figure 4: Dataset-level original-choice accuracy (top) and macro-F1 (bottom) (%) for all 23 model configurations.

	Jev		General-purpose LLMs
Dataset	Jev- 1.13	Jev- 1.14	Qwen3.5- 0.8B	Qwen3.5- 4B	Qwen3.5- 9B	Qwen3.6- 27B
A CARE	60.0 / 57.6	57.8 / 55.2	27.4 / 10.8	47.4 / 43.1	53.3 / 52.2	65.9 / 65.8
A ESI	42.6 / 39.3	39.2 / 37.7	23.6 / 11.2	27.8 / 21.3	33.6 / 29.3	43.9 / 43.1
A MIMIC-IV-ED	38.1 / 35.3	37.1 / 35.0	21.0 / 9.2	24.9 / 20.5	29.8 / 29.2	37.6 / 37.8
A Ramaswamy	82.9 / 70.9	83.6 / 72.6	33.0 / 15.8	72.5 / 58.0	81.2 / 75.1	81.9 / 71.0
B ICD-proxy	94.6 / 94.6	94.5 / 94.5	76.3 / 76.7	91.9 / 91.9	92.3 / 92.3	94.0 / 94.0

Appendix

Table 12: Original choice: Acc / macro-F1 (%), panel 1 of 3. Model-family headings follow Table 1 . All 15 datasets are retained. A–D identify the four decision families. Ramaswamy F1 uses 480 unique-reference items and SCT F1 uses 76; accuracy uses all respective items.

	Medical LLMs			Jev-like models
Dataset	MedGemma- 4B-IT	MedGemma- 27B-IT	Baichuan- M3-235B	GLiNER2.5- Decide
A CARE	54.1 / 51.9	51.1 / 50.4	45.9 / 41.3	27.4 / 10.8
A ESI	23.7 / 16.8	19.8 / 15.4	32.8 / 28.0	24.9 / 9.1
A MIMIC-IV-ED	26.4 / 22.3	22.3 / 17.2	31.2 / 31.2	22.5 / 7.3
A Ramaswamy	56.9 / 32.4	69.8 / 57.4	82.1 / 70.6	16.7 / 10.5
B ICD-proxy	87.4 / 87.3	91.9 / 91.9	93.7 / 93.7	31.1 / 23.7

Appendix

Table 13: Original choice: Acc / macro-F1 (%), panel 2 of 3, under the same scoring convention and model-family organization as Table 12 .

	Jev-like models
Dataset	StartLux-0.8B	StartLux-4B	StartLux-9B	StartLux-27B	D3-27B
A CARE	38.5 / 36.1	61.5 / 56.6	58.5 / 55.7	52.6 / 49.8	66.7 / 66.4
A ESI	25.8 / 17.8	29.3 / 25.9	36.6 / 32.3	43.0 / 39.6	39.1 / 35.2
A MIMIC-IV-ED	22.1 / 12.1	26.2 / 25.6	32.0 / 29.2	40.4 / 36.5	35.7 / 34.0
A Ramaswamy	61.0 / 33.8	82.0 / 69.7	79.2 / 64.3	81.8 / 65.6	81.6 / 60.2
B ICD-proxy	87.4 / 87.4	92.2 / 92.1	93.3 / 93.3	94.6 / 94.6	94.9 / 94.9

Appendix

Table 14: Original choice: Acc / macro-F1 (%), panel 3 of 3, completing the Jev-like model block with the four StartLux configurations and D3-27B. Scoring and model-family organization follow Table 12 .

Jev
Model	$\Delta$ Rel.	$\Delta$ Reord.	$\Delta$ OBS	$V_{\Delta}$	$F_{\mathrm{Rel}}$	$F_{\mathrm{Reo}}$	$F_{\mathrm{OBS}}$	$\overline{F}$
Jev-1.13	+0.45	-0.03	+0.05	1.70	3.9	6.3	8.3	6.2
Jev-1.14	+0.19	+0.03	+0.19	1.09	3.4	6.5	7.1	5.7
General-purpose LLMs
Qwen3.5-0.8B	-0.87	+1.05	-3.31	50.95	20.6	45.3	44.2	36.7
Qwen3.5-4B	+0.28	-0.05	-4.63	21.51	10.7	18.9	40.9	23.5

Appendix

Table 15: Format robustness across all 15 datasets, with equal dataset weights.

\Delta

Rel.,

\Delta

Reord., and

\Delta

OBS denote accuracy changes (percentage points) under relabeling, reordering, and OBS, respectively.

V_{\Delta}

denotes mean within-dataset accuracy variance (squared percentage points).

F_{\mathrm{Rel}}

F_{\mathrm{Reo}}

, and

F_{\mathrm{OBS}}

denote prediction reversal rates (%) under the corresponding interventions and

\overline{F}

is their mean.

Jev
Model	Original	Relabeling	Reordering	OBS	$F_{\mathrm{Rel}}$	$F_{\mathrm{Reo}}$	$F_{\mathrm{OBS}}$
Jev-1.13	61.6 / 58.7	62.1 / 58.9	61.6 / 58.4	61.7 / 58.5	3.9	6.3	8.3
Jev-1.14	62.3 / 59.7	62.5 / 60.0	62.3 / 60.0	62.5 / 60.1	3.4	6.5	7.1
General-purpose LLMs
Qwen3.5-0.8B	37.6 / 27.2	36.8 / 26.6	38.7 / 31.9	34.3 / 19.0	20.6	45.3	44.2
Qwen3.5-4B	53.2 / 47.8	53.5 / 48.2	53.2 / 48.3	48.6 / 38.8	10.7	18.9	40.9

Appendix

Table 16: Complete original, relabeling, reordering, and OBS accuracy / macro-F1 (%), plus the three paired reversal rates, for all 23 models. All scores use equal dataset weights. The main text adds the within-dataset four-format variance and the three-format mean reversal.

Figure 5: Three distinct views of each interface change for all models, with equal dataset weighting. Left: accuracy change relative to original choice, in percentage points. Middle: the percentage of predictions that change after semantic alignment. Right: total-variation distance between aligned candidate distributions. A small accuracy difference can coexist with many flips or a large probability shift.

Jev
Model	Original Acc	NONE Acc	$\Delta$ Acc	Slot flip	Correctness flip	NONE Acc (orig. correct)
Jev-1.13	77.6	65.6	-12.0	18.4	16.2	70.1
Jev-1.14	77.7	66.3	-11.4	18.2	16.0	71.0
General-purpose LLMs
Qwen3.5-0.8B	46.5	34.7	-11.8	52.3	47.9	48.5
Qwen3.5-4B	66.5	51.9	-14.5	29.8	26.2	51.9

Appendix

Table 17: Correct-answer replacement across four Clinical Judgment datasets (2,382 paired entries). Accuracy and reversal rates are percentages.

\Delta

Acc is NONE accuracy minus original accuracy in percentage points. NONE Acc (orig. correct) measures accuracy after replacement on items each model originally answered correctly.

Jev
	Original Choice		With Abstention		Gains
Model	$A^{0}_{\mathrm{all}}$	$A^{0}_{S}$	$R$	$A^{+}_{S}$	$\Delta$	$\Delta_{S}$
Jev-1.13	74.9	75.5	4.5	75.2	+0.3	-0.3
Jev-1.14	76.8	77.4	4.4	77.3	+0.5	-0.1
General-purpose LLMs
Qwen3.5-0.8B	39.7	44.9	35.2	44.7	+5.0	-0.2

Appendix

Table 18: Optional abstention across six Clinical Judgment datasets (2,501 entries).

A_{\mathrm{all}}^{0}

and

A_{S}^{0}

are original accuracy on the full cohort and retained subset.

R

is the mean abstention rate and

A_{S}^{+}

the mean retained accuracy (all in %). Gains

\Delta

and

\Delta^{S}

are in percentage points, computed before rounding.

Model	Abstain	Retained Acc/F1	Original: all	Original: retained	Retained $N$
Jev
Jev-1.13	4.5	75.2 / 74.5	74.9 / 74.3	75.5 / 74.9	2241
Jev-1.14	4.4	77.3 / 76.6	76.8 / 76.2	77.4 / 76.8	2270
General-purpose LLMs
Qwen3.5-0.8B	35.2	44.7 / 38.6	39.7 / 35.0	44.9 / 39.9	1469
Qwen3.5-4B	2.8	62.4 / 62.0	60.4 / 59.7	60.8 / 60.1	2333

Appendix

Table 19: All-model direct-choice abstention comparison in B only (six datasets; 2,501 entries). Accuracy, macro-F1, and abstention rates are computed within each dataset and averaged equally. Original: all uses choice without an abstention option on the complete B cohort; Original: retained uses that protocol on exactly the modified version’s retained items. Retained

N

is the pooled count across datasets and excludes refusals. Retained-set metrics omit a dataset only when no entries are retained there (GLiNER2.5-Decide on SCT).

Figure 6: Retained accuracy across confidence thresholds for all 23 models. Blue circles denote Choice; orange squares denote normalized OBS. Missing points indicate empty retained sets. Coverage and counts are reported in Figure 7 and Table 22 .

Jev
Model	Choice Acc	Cov.	OBS Acc	Cov.
Jev-1.13	79.4	42.3	97.0	18.8
Jev-1.14	80.3	39.9	90.3	24.7
General-purpose LLMs
Qwen3.5-0.8B	77.1	3.1	–	0.0
Qwen3.5-4B	76.7	44.0	100.0	0.0

Appendix

Table 20: Retained accuracy and coverage (%) at

\tau=0.90

. Accuracy pools retained entries; coverage uses all 8,742 entries. Results for all five thresholds are in Table 21 .

Jev
Model	Mode	0.70	0.75	0.80	0.85	0.90
Jev-1.13	C	72.9	74.1	75.6	77.3	79.4
	OBS	78.6	81.7	85.7	91.6	97.0
Jev-1.14	C	74.0	75.4	77.3	78.3	80.3
	OBS	77.0	79.5	81.7	84.9	90.3
General-purpose LLMs

Appendix

Table 21: Threshold comparison with adjacent choice (C) and normalized OBS (O) rows. Cells are pooled retained accuracy (%). Empty subsets have undefined scores (–). Retained populations are determined separately for each model and scoring protocol. Table 22 reports exact counts.

Jev
Model	Mode	0.70	0.75	0.80	0.85	0.90
Jev-1.13	C	5659	5139	4683	4209	3700
	OBS	3712	3236	2782	2262	1647
Jev-1.14	C	5483	5019	4532	4024	3484
	OBS	4030	3576	3144	2682	2163
General-purpose LLMs

Appendix

Table 22: Exact retained

N

for the threshold scores in Table 21 . C: choice; O: normalized OBS. The pre-threshold denominator is 8,742 for every model and protocol.

Figure 7: Rounded coverage percentages at every threshold for all models; $<1$ denotes nonzero coverage below 1%. The denominator is 8,742 complete parent items. Accuracy and coverage jointly characterize the operating points of probability-based selection.

Figure 8: OBS minus Choice accuracy (percentage points) at matched global coverage for all 23 models. Positive values favor OBS. Absolute accuracies and retained counts are reported in Table 23 .

Jev
Model	Mode	5%	10%	15%	20%	25%
Jev-1.13	C	94.05	93.59	93.90	94.34	90.07
	OBS	99.31	98.28	97.71	96.22	92.45
Jev-1.14	C	95.42	96.22	96.34	93.59	89.98
	OBS	99.54	98.74	97.56	95.65	90.25
General-purpose LLMs

Appendix

Table 23: Global fixed-coverage selection, all models. Adjacent rows compare original choice (C) and normalized OBS. Cells are pooled retained accuracy (%). The five columns retain exactly 437, 874, 1,311, 1,748, and 2,185 entries, respectively, for every row. Task mixtures may differ.

Figure 9: Choice versus normalized OBS confidence distributions for all 23 models on the same 8,742 entries. Confidence is the probability assigned to the predicted label. Lines show the percentage of entries in confidence bins of width 0.05. Blue solid lines denote Choice; orange dashed lines denote OBS.

Jev
Model	Mode	Mean $c$	SD( $c$ )	IQR( $c$ )	Mean SD d	Mean $H$	$c_{\rm correct}$	$c_{\rm wrong}$
Jev-1.13	C	0.790	0.192	0.370	0.152	0.378	0.847	0.700
	OBS	0.661	0.198	0.372	0.135	0.621	0.724	0.563
Jev-1.14	C	0.781	0.191	0.380	0.147	0.395	0.837	0.690
	OBS	0.685	0.202	0.401	0.136	0.576	0.741	0.592
General-purpose LLMs

Appendix

Table 24: All-model distribution summaries on the full paired cohort. Adjacent rows compare choice (C) and normalized OBS, each with

N=8,742

. All values are on a 0–1 scale. Mean

c

, SD(

c

), and IQR(

c

) pool entries; Mean SD d averages within-dataset SDs equally. Mean

H

is pooled normalized within-item entropy. The last columns give mean confidence conditional on correct or incorrect predictions under that row’s protocol; those subsets need not match between rows.

Family	Task	$N$	Readout	$0.70$	$0.75$	$0.80$	$0.85$	$0.90$
A	Triage / navigation	3095	Choice	1879 (60.7)	1607 (51.9)	1371 (44.3)	1146 (37.0)	881 (28.5)
			OBS	669 (21.6)	474 (15.3)	314 (10.1)	140 (4.5)	26 (0.8)
B	Clinical judgment	2501	Choice	2235 (89.4)	2183 (87.3)	2127 (85.0)	2065 (82.6)	1997 (79.8)
			OBS	1994 (79.7)	1902 (76.0)	1811 (72.4)	1660 (66.4)	1361 (54.4)
C	Safety review	1812	Choice	1139 (62.9)	1006 (55.5)	873 (48.2)	713 (39.3)	557 (30.7)
			OBS	742 (40.9)	580 (32.0)	422 (23.3)	274 (15.1)	132 (7.3)

Appendix

Table 25: Jev 1.13 coverage by task family at probability thresholds. Cells report retained counts followed by within-task coverage (%) in parentheses. The denominator is the row’s

N