Native Probability Outputs for Clinical Triage: An Evaluation of Calibration and Decision Accuracy

Liang You; Zhiyin Zhou; Dawei Tang; Chuchu Wu; Jingzhuo Hu

cs.LGOct 17, 2026

Native Probability Outputs for Clinical Triage: An Evaluation of Calibration and Decision Accuracy

Authors: Liang You, Zhiyin Zhou, Dawei Tang, Chuchu Wu, Jingzhuo Hu

Organizations: University of Pittsburgh · Pratt Institute · Icahn School of Medicine at Mount Sinai · Carnegie Mellon University ·

Abstract

Patient-facing triage requires an appropriate care recommendation and recognition of when information is insufficient. We evaluate whether Jev, a decision model that returns probabilities over predefined actions, supports both judgments. Using CARE-Bench, we compare Jev with four general-purpose language models and three input-independent baselines across 925 decision points from 439 public cases. On the public test subset, Jev ranks fourth of five models by macro-F1 (0.461) but has the lowest observed multiclass Brier score (0.610), an 18.7% improvement over uniform prediction (case-bootstrap 95% interval, 4.9--31.1%). Noul has lower binary Brier than Choice (0.215 versus 0.276). A evaluation of 453 HealthBench conversations again gives Jev the lowest observed Brier score (0.299), but measures a distinct emergency-referral judgment. Jev combines lower probability error with missed clarification needs in the four-action task. Asking about information sufficiency separately increases recall but also produces frequent unnecessary clarification assignments.

Figures & tables

Model	Macro-F1 $\uparrow$	Brier $\downarrow$	Clarification detected (out of 37) $\uparrow$
Jev	0.461	0.610	3
DeepSeek Flash	0.510	0.630	3
Gemini 2.5 Flash	0.416	0.793	0
GPT Luna	0.512	0.667	4
GPT Sol	0.564	0.632	10

Table 1: Main results on 135 test decision points from 62 cases. Macro-F1 summarizes action selection; Brier measures probability error; clarification detection counts reference clarification points assigned the correct action. Arrows indicate the better direction. Bold marks the best observed model value, not statistical significance. Complete results and input-independent baselines appear in Supplementary Table S1 .

Figure 1: Confidence versus correctness on 135 test decision points per model. Panels: A Jev, B DeepSeek Flash, C GPT Luna, D GPT Sol, E Gemini 2.5 Flash. Upper plots show mean confidence (horizontal) versus observed accuracy (vertical); the dashed diagonal indicates perfect agreement. Lower plots show counts in ten confidence bins, using common scales. Confidence here is the largest action probability, not the confidence value Jev itself reports. Sparse bins are descriptive; bin-level confidence intervals are not shown.

Figure 2: Action errors on the public test subset. Rows show reference actions and columns show predictions; cells give row percentages and counts. Jev (A) and DeepSeek Flash (B) each identify only 3 of 37 decision points requiring clarification, compared with 22 of 26 requiring urgent care.

Model	Macro-F1 $\uparrow$	Brier $\downarrow$	Conditional recall $\uparrow$	N $\to$ E $\downarrow$
Jev	0.788	0.299	0.856	0 /134
DeepSeek Flash	0.718	0.395	0.700	0 /134
Gemini 2.5 Flash	0.763	0.372	0.789	1/134
GPT Luna	0.696	0.434	0.617	6/134
GPT Sol	0.812	0.306	0.761	3/134

Table 2: HealthBench emergency-referral action classification on 453 conversations with a unique physician-majority category (Section 3.5). Conditional recall measures recognition of the physician conditional category, not actual clarification behavior. N

\to

E counts conversations labeled non-emergent incorrectly assigned immediate emergency care. Complete results appear in Supplementary Table S2 .

Figure 3: HealthBench emergency-referral confusion matrices for Jev, GPT Sol, and Gemini. Rows are physician-majority categories; columns are predicted categories. Cells give counts and row percentages. Conditional denotes conditional emergency referral, not observed questioning. The historical five-model protocol used a narrower seek-context-before-deciding criterion; the reference-aligned-definition sensitivity analysis is reported separately.

Decision rule	Recall	Specificity	F1	Brier
Choice argmax	.081	.990	.146	.203
Choice threshold	.027	.990	.051	.203
Noul threshold	.973	.449	.567	.212

Table 3: Original information-sufficiency comparison on 135 Public Test 1 decision points from 62 cases. The positive reference class comprises 37 points requiring information. Threshold rows use

P(\text{Info})\geq.5

for the four-action Choice marginal and

1-P(\mathrm{sufficient})\geq.5

for Noul. Brier is computed from probabilities, so both Choice decision rules have the same binary score. Specificity measures recognition of sufficient information. These rows precede the matched binary follow-up.

Model	USD per 1,000 calls	Total time (seconds)
Jev	0.0193	106.56
DeepSeek Flash	0.0954	252.17
Gemini 2.5 Flash	0.1245	180.41
GPT Luna	0.0774	381.22
GPT Sol	0.7192	484.03

Table 4: Service efficiency for 925 inputs with four concurrent requests. Lower is better. Jev and DeepSeek costs use vendor tariffs; other costs are reported by OpenRouter.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Model / baseline	F1	Acc.	Clar.	Brier
Jev	0.461	53.3	8.1	0.610
DeepSeek Flash	0.510	57.0	8.1	0.630
Gemini 2.5 Flash	0.416	48.9	0.0	0.793
GPT Luna	0.512	55.6	10.8	0.667
GPT Sol	0.564	60.0	27.0	0.632
Uniform	0.108	27.4	100.0	0.750

Appendix

Table S1: Complete results on 135 test decision points. Acc.: overall accuracy (%); Clar.: clarification recall (%). Higher is better except for Brier. Baselines ignore patient information.

Model	Macro-F1	Acc.	Brier	ECE	E recall	C recall	N recall	C $\to$ E / N $\to$ E
Jev	0.788	.795	0.299	.081	.957	.856	.545	.117 / 0
DeepSeek Flash	0.718	.724	0.395	.096	.964	.700	.507	.233 / 0
Gemini 2.5 Flash	0.763	.768	0.372	.136	.942	.789	.560	.150 / .007
GPT Luna	0.696	.700	0.434	.109	.971	.617	.530	.350 / .045
GPT Sol	0.812	.812	0.306	.069	.986	.761	.701	.167 / .022

Appendix

Table S2: Complete HealthBench emergency-referral results on 453 conversations with a unique physician-majority category (Section 3.5, Section 4.4). Acc.: overall accuracy. E/C/N recall: per-class recall for emergent, conditionally-emergent, and non-emergent conversations. C

\to

E: share of conditionally-emergent conversations assigned immediate emergency care; N

\to

E: share of non-emergent conversations assigned immediate emergency care. Macro-F1, Brier, and C recall bootstrap 95% intervals appear in the main text (Section 4.4).

Figure 4: Binary clarification decisions. Rows are references and columns are predictions; cells give counts and row percentages. The historical four-action marginal and both contemporaneous matched binary interfaces use the same 0.5 threshold. The historical panel also differs in date and formulation.

Figure 5: Jev conditional-definition sensitivity. Only the two same-session panels form the paired comparison; the historical panel is a stability reference. Conditional denotes a referral category, not necessarily a request for context. Cells give counts and row percentages.

Figure 6: Info-class calibration for the three displayed models using ten equal-width bins. Point area encodes bin support; vertical bars are descriptive 95% whole-case-bootstrap intervals for observed frequency, conditional on fixed predictions and bins. Sparse homogeneous bins may have degenerate intervals. Results for all five models and 5/10/15-bin data are retained in the analysis exports.

Figure 7: Descriptive precision–recall curves from the historical public-test predictions. Tied probabilities enter together; the no-positive-prediction point has undefined precision and is omitted. These sweeps were not used to select policy thresholds, which were fitted on development data.

Variant	TP / FP		Changes
	Choice	Noul	Choice	Noul
Primary	36 / 54	36 / 53	–	–
Reverse order	35 / 50	36 / 52	5	1
Alternate wording	31 / 42	34 / 40	17	15
Identical repeat	36 / 54	36 / 52	2	1

Appendix

Table S3: Matched binary sensitivity on the public test split. TP/FP are counts; changes are relative to each interface’s primary decisions.

Pair	Agreement	$\kappa$
Jev–DeepSeek	.763	.619
Jev–Luna	.748	.619
Jev–Sol	.807	.703
Jev–Gemini	.763	.619
DeepSeek–Luna	.674	.516
DeepSeek–Sol	.681	.515

Appendix

Table S4: All pairwise Public Test 1 agreements.

Model	Info	Self-care	Nonurgent	Urgent
Jev	.081	.375	.792	.846
DeepSeek	.081	.500	.833	.846
Gemini	.000	.458	.646	.923
Luna	.108	.625	.646	.962
Sol	.270	.417	.729	1.000

Appendix

Table S5: Test recall for all four classes.

Model / baseline	Development	Validation
Jev	.515 / .615	.508 / .607
DeepSeek	.499 / .640	.528 / .621
Gemini	.465 / .753	.447 / .752
Luna	.474 / .673	.457 / .711
Sol	.572 / .646	.518 / .679
Uniform	.106 / .750	.105 / .750

Appendix

Table S6: Secondary splits: macro-F1 / Brier. Development evaluation of fitted baselines is in-sample.

Model	Input $/M tok	Output $/M tok	Cost (925 calls)	Latency med./P95
Jev	0.042	0	0.017875	0.401 / 0.499
DeepSeek Flash	0.300	1.200	0.088243 a	0.916 / 1.192
Gemini 2.5 Flash	0.300	2.500	0.115167	0.631 / 0.826
GPT Luna	0.200	1.200	0.071616	1.058 / 1.814
GPT Sol	2.000	10.000	0.665220	1.410 / 2.210

Appendix

Table S7: Vendor tariffs, estimated or service-reported cost, and latency for the 925-call quality runs. Costs are USD; latency is seconds for successful requests; these are estimates, not invoices. Cost figures are exact quotients of metered token counts and published tariff rates, so their apparent precision reflects the arithmetic, not billing-level certainty.

Model	Wall time (s)	Median latency	P95 latency	Cost (USD)
Jev	50.67	0.400	0.482	0.010069
DeepSeek Flash	129.71	0.973	1.249	0.022075
Gemini 2.5 Flash	126.03	0.705	0.996	0.062663
GPT Luna	158.21	1.079	1.570	0.039694
GPT Sol	206.38	1.287	2.189	0.375696

Appendix

Table S8: Service efficiency for the 453-conversation HealthBench emergency-referral run at four-way concurrency (Section 3.5). Latency is seconds for successful requests. Jev and DeepSeek costs use vendor tariffs (DeepSeek off-peak cache-adjusted); the remaining costs are OpenRouter-reported. These are estimates, not invoices.

Clinical Language Model Evaluation

Clinical language model evaluation measures whether AI systems are accurate, safe, fair, and useful across medical tasks and care settings.

See all 419 papers

Jev in Medicine: A Benchmark Evaluation
Sep 27, 2026Alfredo Madrid-García, Beatriz Merino-BarbanchoMedical QA Clinical Reasoning
Independent researcher, Madrid, Spain
Decision language models as affordable and rapid alternatives to LLM judges of clinical generative AI
Oct 20, 2026Rotem Gershon, Seffi Cohen, Dean Ariel +11LLM-as-a-Judge Clinical Language Model Evaluation
Innovation Division, Clalit Research Institute, Clalit Health Services, Ramat-Gan, Israel. · Stein Faculty of Computer and Information Science, Ben Gurion University of the Negev, Be’er Sheva, Israel. · Department of Family Medicine, Clalit Health Services, Tel Aviv, Israel. +4
OmniMed-Jev: Calibrating LVLM Confidence for Trustworthy Medical Multimodal Decisions via System One
Sep 30, 2026Luyao Tang, Cheng ChenProbability Calibration Multimodal Classification
Department of Electrical and Computer Engineering The University of Hong Kong, Hong Kong SAR, China

Sep 27, 2026J/K move · Enter open · S save

Jev in Medicine: A Benchmark Evaluation

Alfredo Madrid-García, Beatriz Merino-Barbancho

Independent researcher, Madrid, Spain

Medical QA Clinical Reasoning Clinical Language Model Evaluation

Open paper AskPDF arXiv

Jev is a non-generative System One model that assigns probabilities to predefined answer options and cannot answer outside them. Its accuracy and calibration on medical question-answering and case-based diagnostic-reasoning tasks are unknown. We evaluated Jev 1.13 on four medical benchmarks: MetaMedQA, PubMedQA, DiagnosisArena-MCQ and the NEJM Case Challenges. GPT-6 Sol, with (medium) and without reasoning, was the reference. The primary outcome was top-1 accuracy; key secondary outcomes were calibration, selective prediction and recognition of unanswerable questions. All 8,469 requests returned a valid answer. Jev's accuracy was similar to that of GPT-6 Sol with medium reasoning on PubMedQA (78.4% vs 78.2%;), lower on MetaMedQA (74.8% vs 82.7%) and much lower on DiagnosisArena-MCQ (59.8% vs 82.4%;) and the NEJM cases (61.8% vs 82.4%). On MetaMedQA, Jev's probabilities were the best calibrated (expected calibration error 0.063 vs 0.146), and its answers with a probability of at least 0.9 (52.9% of questions) were 93.4% accurate, but GPT-6 Sol was as accurate when it accepted a similar proportion of questions. On DiagnosisArena-MCQ, Jev's probabilities discriminated poorly (AUROC 0.645 vs 0.768). Of the 162 questions whose correct answer was "I don't know or cannot answer", Jev chose that option for 10.5% (GPT-6 Sol, 8.6%). Median latency was 0.27-0.31 s; all 2,823 items cost USD 0.08. Jev was fast and inexpensive, and its accuracy was similar to that of a frontier LLM on research abstracts but lower on examination questions and much lower on complex diagnostic cases. Task-specific validation is required before clinical use.

Abstract

Clinical Language Model Evaluation

Explore similar work

Jev in Medicine: A Benchmark Evaluation

Decision language models as affordable and rapid alternatives to LLM judges of clinical generative AI

OmniMed-Jev: Calibrating LVLM Confidence for Trustworthy Medical Multimodal Decisions via System One