Can JEV Support Clinical Trial Criteria Assessment?

Mengxian He; Ziyue Gao; Chun Li; Shunyi Zhang; Hao Wei; Minqing Zhang; Wu Yuan

cs.LGOct 26, 2026

Can JEV Support Clinical Trial Criteria Assessment?

Authors: Mengxian He, Ziyue Gao, Chun Li, Shunyi Zhang, Hao Wei, Minqing Zhang, Wu Yuan

Organizations: The Chinese University of Hong Kong · Capital Medical University

Abstract

Clinical trial prescreening requires distinguishing unmet conditions from insufficient evidence in patient records. Given patient records, criterion questions, and predefined answer options, we retrospectively compared JEV, a non-generative decision model returning native option probabilities, with schema-constrained Qwen Flash across 2,048 test patient-condition pairs from TrialGPT and question-answer datasets derived from MIMIC-III and MIMIC-IV, all evaluated under strict Zero Data Retention (ZDR) policies to ensure clinical data privacy. Both models assessed conditions individually or in batches sharing the same patient context, with performance measured against reference labels. In batch mode, JEV and Qwen achieved similar macro-F1 scores on TrialGPT (0.652 vs. 0.660) and MIMIC-IV (0.807 vs. 0.814), whereas JEV substantially outperformed Qwen on MIMIC-III (0.768 vs. 0.534; $Δ$ = +0.234, 95% CI: 0.187-0.281). This advantage largely reflected higher recall for conditions labeled as having insufficient information (56.3% vs. 9.9%). Across all three datasets, JEV also had lower observed median job latency (1.18-1.40 vs. 2.80-8.90 s) and 53%-74% lower estimated cost per condition. These findings position JEV as an initial assessment component that provides criterion-level judgments and flags potential information gaps for review. JEV's observed errors in interpreting clinical evidence underscore the need to verify its outputs against patient records before they inform eligibility decisions.

Figures & tables

Figure 1: Proposed role of JEV in clinical trial prescreening. (a) Reviewers assess the patient record against individual inclusion and exclusion criteria and document their judgments. (b) JEV assesses criteria sharing the same patient context in a batch, returning provisional labels and native option probabilities. Clinicians verify these judgments against the patient record and follow up on potential information gaps before making a screening decision.

Label	Meaning
met	The stated condition holds for the patient.
not_met	The stated condition does not hold under the specified policy.
insufficient	Required information is missing or unresolved under that policy.
not_applicable	The criterion does not apply to the patient or situation; TrialGPT only.

Table 1: Each label refers to the literal condition, including its qualifiers and the source annotation policy. INSUFFICIENT describes the available evidence, which is not assigned by applying a threshold to model confidence.

Figure 2: TrialGPT retains 1,014 of 1,015 source patient–criterion pairs after exclusion of one empty criterion. MIMIC-III retains 532 Boolean questions from 1,000 source pairs. MIMIC-IV retains 1,500 Boolean questions and 89 derived LVEF conditions, comprising 37 retained numerical references and 52 references indicating unavailable information.

Source	Retained reference	Mapping or boundary
TrialGPT inclusion	included; not included	met ; not_met
TrialGPT exclusion	excluded; not excluded	met ; not_met
TrialGPT, either type	not enough information; not applicable	insufficient ; not_applicable
MIMIC Boolean	Available Yes or No	met or not_met ; preserve source defaults
MIMIC missing answer	Source unavailable flag and blank answer	insufficient ; inconsistent availability/answer pairs excluded
MIMIC-IV LVEF	Comparable lowest recorded value $v$ in percent	met if $v\leq 40$ ; not_met otherwise. Source-valid capped range: $0\leq v\leq 55$ .

Table 2: Source-to-reference transformations. Labels describe the stated condition; they do not encode final trial eligibility. The LVEF reference range reflects the source response convention and the retained construction rules.

Track	Groups	Jobs	Total	Dev	Test	Confirmation
TrialGPT	53	105	1,014	240	774	0
MIMIC-III	522	529	532	105	322	105
MIMIC-IV	100	100	1,589	319	952	318
Total	675	734	3,135	664	2,048	423

Table 3: Standardized collection and retained partitions. Group counts are summed within datasets and do not represent a deduplicated count of people across sources.

Track	Groups	Jobs	Conditions	M	N	U	A
TrialGPT	43	85	774	129	366	243	36
MIMIC-III	314	319	322	83	97	142	–
MIMIC-IV	60	60	952	255	667	30	–

Table 4: Composition of the primary test partitions. M, N, U, and A denote MET, NOT_MET, INSUFFICIENT, and NOT_APPLICABLE, respectively.

Figure 3: Benchmark evaluation pipeline.

Dataset	Mode	$n$	Accuracy	Macro-F1	95% CI	Failed
TrialGPT	JEV Single	774	0.720	0.646	[0.594, 0.693]	1
TrialGPT	JEV Batch	774	0.726	0.652	[0.601, 0.702]	0
TrialGPT	Qwen Single	774	0.730	0.645	[0.590, 0.699]	0
TrialGPT	Qwen Batch	774	0.749	0.660	[0.602, 0.714]	0
MIMIC-III	JEV Single	322	0.758	0.771	[0.728, 0.812]	0
MIMIC-III	JEV Batch	322	0.755	0.768	[0.724, 0.811]	1

Table 5: Primary historical test results under the native label spaces. Macro-F1 and accuracy are on

[0,1]

. The 95% descriptive intervals use grouped bootstrap resampling.

Figure 4: Macro-F1 comparison on the historical test partitions. TrialGPT uses four classes and the MIMIC tracks three. Error bars denote 95% descriptive group-bootstrap intervals.

Dataset	$n$	Off F1	On F1	$\Delta$ F1 (pp)	95% CI (pp)
TrialGPT	774	0.677	0.701	+2.39	[-1.50, +6.74]
MIMIC-III	322	0.531	0.560	+2.95	[-1.57, +7.35]
MIMIC-IV	952	0.827	0.798	-2.89	[-6.10, +0.11]

Table 6: Separate test-only Qwen Batch thinking comparison. Paired F1 differences are in percentage points.

	Cost / 1,000 (USD)		Median job time (s)
Dataset	Off	On	Off	On
TrialGPT	0.0645	0.1249	4.45	19.12
MIMIC-III	0.7264	0.8991	2.96	6.90
MIMIC-IV	0.0876	0.1299	8.65	26.66

Table 7: Matched Qwen thinking resources on the test split only. Cost is the estimated upper bound in USD per 1,000 planned conditions; median latency is seconds per complete timing-valid job.

Figure 5: Primary baseline quality versus estimated upper-bound cost (top) and observed median job latency (bottom). Horizontal axes are logarithmic. Filled markers denote Batch and hollow markers Single; sizes only improve visibility at overlapping locations.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Partition	$n$	JS	JB	QS	QB
TrialGPT	Development	240	0.748	0.749	0.749	0.779
TrialGPT	Historical test	774	0.646	0.652	0.645	0.660
MIMIC-III	Development	105	0.680	0.680	0.464	0.459
MIMIC-III	Historical test	322	0.771	0.768	0.523	0.534
MIMIC-III	Confirmation	105	0.814	0.804	0.602	0.637
MIMIC-IV	Development	319	0.813	0.816	0.722	0.838

Appendix

Table 8: Native-label macro-F1 by frozen partition. JS/JB denote JEV Single/Batch; QS/QB denote Qwen Single/Batch. All partitions were previously inspected; “confirmation” is a historical identifier.

Dataset	Mode	$\Delta$ Macro-F1 (pp)	95% CI (pp)	Net correct
TrialGPT	Single	+0.05	[-5.15, +5.08]	-8
TrialGPT	Batch	-0.81	[-5.73, +4.34]	-18
MIMIC-III	Single	+24.89	[+20.16, +29.62]	+68
MIMIC-III	Batch	+23.42	[+18.73, +28.05]	+62
MIMIC-IV	Single	+7.66	[+1.82, +13.39]	+13
MIMIC-IV	Batch	-0.65	[-4.24, +2.92]	-7

Appendix

Table 9: JEV minus Qwen in the same request mode on test. Macro-F1 differences and their 95% paired intervals are percentage points. Net correct is corrections minus damages relative to Qwen.

Dataset	Model	Changed	Corrected	Damaged	Net	$\Delta$ F1 (pp)
TrialGPT	JEV	31	17	12	+5	+0.64
TrialGPT	Qwen	97	51	36	+15	+1.50
MIMIC-III	JEV	10	3	4	-1	-0.33
MIMIC-III	Qwen	28	11	6	+5	+1.14
MIMIC-IV	JEV	13	7	6	+1	-0.10
MIMIC-IV	Qwen	77	48	27	+21	+8.22

Appendix

Table 10: Changes from Single to Batch on the historical test partitions. Corrections and damages are paired counts; a changed label can remain incorrect. These counts do not establish stability under repeated calls.

Track	Arm	Timed/planned	Cost/1,000	Req. P50	Job P50	Job P95
TrialGPT	JEV Single	85/85	0.0323	0.94	8.08	19.76
TrialGPT	JEV Batch	85/85	0.0170	1.05	1.20	1.58
TrialGPT	Qwen Batch	85/85	0.0645	4.55	4.55	8.12
TrialGPT	Qwen Single	85/85	0.0999	2.33	8.95	22.70
MIMIC-III	JEV Single	319/319	0.2064	1.09	1.19	1.49
MIMIC-III	JEV Batch	319/319	0.2045	1.08	1.18	1.44

Appendix

Table 11: Primary test resources. Cost is the estimated upper bound in USD per 1,000 planned conditions. Timed/planned gives timing-audit-valid jobs over planned jobs; terminal failures remain included. Request and job percentiles are seconds. These are observed service measurements under different acquisition conditions.

Track	Mode	Valid / planned	Brier	NLL	ECE	High- $p$ errors
TrialGPT	Single	773/774	0.3787	0.8391	0.0475	19/195
TrialGPT	Batch	774/774	0.3756	0.8324	0.0336	20/196
MIMIC-III	Single	322/322	0.3615	0.9788	0.0974	20/174
MIMIC-III	Batch	320/322	0.3611	0.9044	0.0958	20/172
MIMIC-IV	Single	948/952	0.1565	0.5359	0.0376	19/549
MIMIC-IV	Batch	952/952	0.1556	0.5073	0.0346	19/560

Appendix

Table 12: JEV native probability diagnostics on historical test conditions. Brier is the summed multiclass score; NLL is negative log-likelihood; ECE uses ten equal-width bins. High-

p

errors count incorrect labels among selected-label probabilities

\geq 0.9

. Metrics use valid vectors only. These values do not establish clinical risk thresholds or a cross-model calibration ranking.

Role	Retained families	Scope limitation
Six inclusion-related components	Atrial fibrillation; prior stroke; heart failure; arterial hypertension; type 2 diabetes; derived LVEF	Source evidence covers only parts of protocol requirements, not every qualifier or the complete inclusion logic.
Ten contextual questions	Major depressive disorder; schizophrenia; bipolar disorder; hemorrhagic tendency; recent stroke; peptic ulcer disease; bleeding; atrial fibrillation ablation; surgical valvular disease; medical decision-making	Source questions differ from protocol gates in timing, required qualifiers, or meaning. A source-positive answer is not itself a disqualifying event.

Appendix

Table 13: MIMIC-IV adapted family roles, inherited from the saved task-construction review. These descriptions are not independent clinical adjudications or complete protocol criteria.

Figure 6: All cell values are percentages, not counts. Row-normalized Batch confusion matrices on the test partitions; rows denote reference labels and columns model outputs. Parentheses show reference counts. M, N, U, A, and F denote met , not_met , insufficient , not_applicable , and failed or missing output. Values are rounded to integers, except nonzero values below $1\%$ , which retain one decimal. In MIMIC-III JEV Batch, the U-to-F cell is $0.7\%$ (1 of 142 conditions). Rounded rows may not sum to $100\%$ . F is a technical outcome, not a clinical class; absent clinical classes are not added.

Dataset	Arm	P (%)	R (%)	F1 (%)	U $\to$ D	D $\to$ U
TrialGPT	JEV Single	63.2	67.1	65.1	75/243	85/495
	JEV Batch	64.6	67.5	66.0	75/243	80/495
	Qwen Single	80.3	40.3	53.7	144/243	22/495
	Qwen Batch	81.5	49.0	61.2	122/243	25/495
MIMIC-III	JEV Single	88.9	56.3	69.0	62/142	10/180
	JEV Batch	88.9	56.3	69.0	61/142	10/180

Appendix

Table 14: Evidence-state errors on the historical test split. Precision (P), recall (R), and F1 refer to the scored INSUFFICIENT class. U

\to

D counts reference INSUFFICIENT predicted MET or NOT_MET; D

\to

U counts the reverse direction. These are benchmark label errors, not abstention decisions.

Case	Dataset	Evidence issue	Gold	JS	JB	QS	QB
C1	MIMIC-III	Undocumented historical symptoms	U	U	U	N	N
C2	MIMIC-III	Explicit negation across a symptom list	N	N	N	M	M
C3	TrialGPT	Applicability versus nonfulfillment	A	N	N	N	N
C4	MIMIC-III	Symptom onset before hospitalization	M	U	U	M	M
C5	MIMIC-III	Laboratory values below a threshold	N	N	N	M	N
C6	TrialGPT	Symptoms without required duration	U	U	U	M	M

Appendix

Table 15: Illustrative test-set conditions. M: MET; N: NOT_MET; U: INSUFFICIENT; A: NOT_APPLICABLE. JS/JB denote JEV 1.14.0 Single/Batch, and QS/QB denote Qwen Single/Batch. Agreement with gold is a benchmark result, not independent clinical adjudication.

Clinical Decision Support

Clinical Decision Support uses data, medical knowledge, and AI to help clinicians assess patients and choose tests, diagnoses, and treatments.

See all 242 papers

Native Probability Outputs for Clinical Triage: An Evaluation of Calibration and Decision Accuracy
Oct 17, 2026Liang You, Zhiyin Zhou, Dawei Tang +2Clinical Language Model Evaluation Clinical Triage
University of Pittsburgh · Pratt Institute · Icahn School of Medicine at Mount Sinai +2
Jev in Medicine: A Benchmark Evaluation
Sep 27, 2026Alfredo Madrid-García, Beatriz Merino-BarbanchoMedical QA Clinical Reasoning
Independent researcher, Madrid, Spain
Decision language models as affordable and rapid alternatives to LLM judges of clinical generative AI
Oct 20, 2026Rotem Gershon, Seffi Cohen, Dean Ariel +11LLM-as-a-Judge Clinical Language Model Evaluation
Innovation Division, Clalit Research Institute, Clalit Health Services, Ramat-Gan, Israel. · Stein Faculty of Computer and Information Science, Ben Gurion University of the Negev, Be’er Sheva, Israel. · Department of Family Medicine, Clalit Health Services, Tel Aviv, Israel. +4

Oct 17, 2026J/K move · Enter open · S save

Native Probability Outputs for Clinical Triage: An Evaluation of Calibration and Decision Accuracy

Liang You, Zhiyin Zhou, Dawei Tang, Chuchu Wu+1

University of Pittsburgh · Pratt Institute · Icahn School of Medicine at Mount Sinai+2

Clinical Language Model Evaluation Clinical Triage Clinical Decision-Making Probability Calibration Selective Prediction

Open paper AskPDF arXiv

Patient-facing triage requires an appropriate care recommendation and recognition of when information is insufficient. We evaluate whether Jev, a decision model that returns probabilities over predefined actions, supports both judgments. Using CARE-Bench, we compare Jev with four general-purpose language models and three input-independent baselines across 925 decision points from 439 public cases. On the public test subset, Jev ranks fourth of five models by macro-F1 (0.461) but has the lowest observed multiclass Brier score (0.610), an 18.7% improvement over uniform prediction (case-bootstrap 95% interval, 4.9--31.1%). Noul has lower binary Brier than Choice (0.215 versus 0.276). A evaluation of 453 HealthBench conversations again gives Jev the lowest observed Brier score (0.299), but measures a distinct emergency-referral judgment. Jev combines lower probability error with missed clarification needs in the four-action task. Asking about information sufficiency separately increases recall but also produces frequent unnecessary clarification assignments.

Abstract

Clinical Decision Support

Explore similar work

Native Probability Outputs for Clinical Triage: An Evaluation of Calibration and Decision Accuracy

Jev in Medicine: A Benchmark Evaluation

Decision language models as affordable and rapid alternatives to LLM judges of clinical generative AI