A general-purpose System One decision model for clinical records
Authors: Andrew Zhang, Trinh T. L. Vuong, Sophia J. Wagner, Dongmin Bang, Tong Ding, Yuchang Su, Mary Cipperman, Christina S. Chen, Judith Lefkes, Ramin Kahidi, Ming Y. Lu, Caiwei Tian, Tianhao Luo, Jingnan Ma, Rowland Pettit, Long Phi Le, Faisal Mahmood
Organizations: Mass General Brigham AI Institute, Boston, MA. · Harvard Medical School, Boston, MA. · Massachusetts Institute of Technology, Cambridge, MA.
Clinical care, health-system operations and biomedical discovery depend on extracting reliable answers from patient records, yet applying diverse clinical questions across entire populations remains computationally expensive. Here we present Clinical System One (CS1), a 4-billion-parameter decision model for fast, typed judgments from clinical records, along with two agentic harnesses in which CS1 works together with a generalist language model. CS1-4B is trained on 12.4 million questions and answers from 3.6 million patients across a multi-hospital health system, combining structured-data keys, observed outcomes and language-model supervision across clinical notes, diagnostic reports and patient timelines. It answers multiple typed questions in a single forward pass without task-specific retraining, and each answer carries a probability that can route uncertain cases to human review. On 963,408 held-out patients, CS1-4B outperformed open decision models up to 27 billion parameters on 13 of 14 tasks scored independently of its training teacher. Across 21 external data sets, its aggregate performance approached that of the strongest tested 27-billion-parameter model, with 6.4-fold faster inference in a matched workload. Integrating CS1-4B with a language model agent produced cited answers over complete multi-year patient histories and screened a patient population against the eligibility criteria of 17 registered clinical trials, placing four times as many judged-eligible patients among its top 20 candidates as keyword search while using 690-fold fewer language-model tokens than a language model reading every record. These findings support a division of labour in clinical AI, with small decision models performing repeated record reading and generative models formulating questions and synthesizing evidence.
Figures & tables
Fig. 1: Clinical System One reads a clinical record once and answers many typed questions with calibrated probabilities. a , The Klaros-12M training data: 12.4 million question–answer (QA) pairs from 3.64 million patients at 24 hospital sites (median follow-up 12 years, interquartile range 5–20), asked of 7.47 million documents. One patient, items per year for a synthetic patient whose totals equal the cohort medians. Below, the share of pairs by question type and by modality (colour, input type, as in Fig. 2 a), and the five most frequent conditions in the 3.01 million structured training records. b , CS1 answers any number of typed questions about a document in one forward pass, whereas a generative LLM can write hundreds of tokens per question. c , The three question types, each option shaded by CS1-4B’s probability (darkest, the answer). d , Ground truth. Left, soft labels for 4.51 million note and 2.46 million report pairs: the GLM-5.3-Flash teacher’s next-token probabilities, averaged over rotated option orders. Middle, a record timeline serialized into a budgeted text document. Right, exact labels for 5.01 million timeline and 433,000 forecasting pairs from rules or observed outcomes, kept only when the facts are visible in the document. e , Shared-document packing: the document is read once and each question attends only to the document and itself; the candidate head returns one logit per option. CS1-4B has 4.2 billion parameters and was initialized from Decision 2.0 Nox-4B. f , Top, one synthetic note read once by CS1-4B: answers at P≥0.95 are accepted automatically and the rest go to a reviewer; highlighted, the evidence for the cost question. Bottom, the same output serves six user groups. Patient examples are synthetic.
Fig. 2: Performance on held-out patient records and transfer to public data sets. a , Six task types, with example questions and the source of their answer keys (Extended Data Table 4); colour, input type. Padlock, private health-system records, never public; globe, public data sets, whose records may be in open models’ training data. b , MGBQA-1M (963,408 held-out patients): mean balanced accuracy over each task type’s tasks keyed by rules, observed outcomes or an open LLM (14 of 28 tasks; Extended Data Table 3) against model size; blue, CS1-4B; grey, the best of the ten open decision models at each size. c , The structured-record public sets by source institution: Nox-4B (grey) and CS1-4B (blue), with their difference (points); balanced accuracy (%), prognosis mean AUROC × 100. Other public sets and every open model, Extended Data Table 4. d , CS1-4B minus the best open model on each of the 14 tasks of b (balanced-accuracy points). e , CS1-4B minus Nox-4B per task type, on MGBQA-1M and on the 21 public data sets. f , Mean of b over the six task types against GPU-hours per million documents at 40 questions on one GB10 (Extended Data Table 2); dashed, Pareto frontier. g , The three public prognosis sets (27 outcomes): observed incidence against predicted risk, raw and after one log-odds shift per outcome to its incidence in the other half of the patients. h , Expected calibration error (ECE) of the top answer per task type, pooled over the public data sets. Dots in c–e and h, task type. CS1-4B: fine-tuned on Klaros-12M only, raw probabilities unless stated.
Fig. 3: CS1 provides evidence attribution over notes and timelines. a , Deletion attribution: a sentence’s necessity is P(a∣d)−P(a∣d∖s) , the drop in the probability of the answer a when the sentence is deleted and the document is re-encoded; its sufficiency is P(a∣s alone)−P(a∣empty document) ; the bars and numbers are those of the denial sentence in d. Re-encoding reuses the model’s state for the text before the deleted sentence; long documents are screened in 10-sentence chunks before sentences are measured individually in the top chunks. b , Faithfulness on the 90 evidence-dependent note–question pairs (answer probability below 0.5 for an empty document) of 200 pairs from 100 MIMIC-IV discharge notes not used to train CS1-4B (aggregate only): mean probability of the original answer after deleting the top- k sentences by necessity or sufficiency (lower is more faithful) or after keeping only them (higher is more faithful); dotted, random order. Median time per answer on one GB10: 24 s (necessity) and 10 s (sufficiency). c , Answers flipped by deleting the top three sentences of the same pairs: 41% (necessity) and 59% (sufficiency), against 1% for random order. d – f , Evidence in two synthetic notes and a synthetic structured record: for each sentence or line, its necessity (left) and sufficiency (right), as defined in a; values below zero (a sentence that lowers the answer’s probability) are drawn as zero. d , An explicit denial: deleting one sentence drops the answer from 0.98 to 0.07. e , A fact stated twice: deleting either mention lowers it by at most 0.21, while each alone gives 0.97 or 0.94. f , The laboratory lines of a day-by-day timeline, rendered as CS1 reads structured data: for the course of creatinine over the stay (rose and recovered, P=0.62 ), the evidence spans days. Deleting the last value (D0), the admission value (D − 6) or the earlier outpatient value (D − 400) drops P to 0.19, 0.39 and 0.44, while no single line alone gives the answer. Faithfulness was measured on notes only. All text in d–f is synthetic.
Fig. 4: Evidence-backed answers over a whole patient record. a , A clinical question and the summarized record of a MIMIC-IV test patient. b , An LLM (GLM-5.3) plans 5 typed sub-questions, each with an aggregation rule and a scope. c , CS1-4B answers every sub-question for every document (402 typed decisions; long documents chunked, nothing truncated); rows are labelled by topic. Level rows (trajectory and latest rules) show each document’s level, light to dark in option order; the other rows show whether a document reports the finding (blue, shaded by probability), reports it absent (orange), gives another answer or is silent. Right, code aggregates each sub-question across documents and time: the aggregated answer and the documents that support it; a warning triangle marks conflicting documents, explained in d. d , The LLM composes a cited answer from the aggregated answers and the evidence sentences (one bullet per sentence). e , CS1 attribution selects 16 evidence sentences; the 12 cited in d are shown, grouped by document (a sentence cited twice shares a line). The LLM plans from a manifest of the record (document types, counts and dates) and never reads the record; at the last step it sees only the aggregated answers and the evidence sentences. Aggregation rules: any, the highest probability over documents; latest, the most recent document that addresses the sub-question; trajectory, the per-document level over time; a conflict is confident support and confident contradiction in different documents. Sub-questions scoped to this admission are asked only of its documents (white cells, out of scope). Evidence sentences are selected by CS1 sufficiency attribution; in compliance with the MIMIC-IV data-use policy, the evidence sentences and any record text quoted in the answer are shown as paraphrases, not verbatim. Dates are shifted by MIMIC-IV de-identification and documents are relabelled D001…in date order. Results shown are from a single run, with GLM-5.3 at its default reasoning effort and CS1-4B (raw outputs).
Fig. 5: Population-level cohort building with real clinical trial criteria. a , The pipeline on one of the 17 evaluated trials, VITC-SHOCK (NCT03338569 [ 56 ] ), and 1,000 MIMIC-IV test patients. (1) Eligibility criteria as written (green, inclusion; orange, exclusion; grey, not checkable from records; 12 of 17 shown, shortened). (2) An LLM (GLM-5.3) turns them into atomic typed questions, each with a time scope. (3) CS1-4B reads every document of every patient (21,866 documents); a question’s probability is its maximum over the parts of the record within its scope. Shown are CS1-4B’s top 20 patients and four drawn at random from the rest (P01–P24). (4) Code combines the answers into P(eligible) and ranks the patients (shaded, top 20). (5) A blind LLM judge reading the full record graded the top 20: 10 eligible, 2 possibly eligible and 8 not eligible. b , Criterion-level balanced accuracy for seven trials on 200 patients, for the LLM reading every record and for the agentic harness (95% intervals by patient bootstrap). Filled, 70 structured criteria with rule-computed answers: 0.869 against 0.884, a difference of 0.015 ( − 0.005 to 0.039). Open, 36 note-only criteria, against labels on which two full-record LLM reads agree: 0.796 against 0.766, a difference of − 0.030 ( − 0.060 to − 0.009). c , The criteria of b by topic (dark, structured; light, note-only). d , Ranking all 1,000 patients for each trial by CS1-4B’s P(eligible) and by BM25 keyword search over the full record. The union of both top 20s was graded by a blind GLM-5.3 judge in two passes (663 pairs; weighted κ [ 57 ] between passes 0.84). Left, nDCG@20 [ 58 ] for the 13 trials with any judged-eligible or possibly eligible patient, grouped by therapeutic area (numbers, CS1-4B minus BM25), and the mean with its 95% interval: 0.64 against 0.11 (CS1-4B higher in 12 of 13 trials). Right, judged-eligible (dark) and possibly eligible (light) patients in each top 20; ticks, judged-eligible patients in the pool. e , LLM tokens per screened patient to rank patients for the 13 trials of d: 421,136 for the LLM reading every record against 610 with CS1-4B, 690 times fewer (verification, CS1-4B’s own compute and judge calls not counted).
Extended Data Fig. 1: Logit-readout teacher. For one document–question pair (synthetic excerpt; illustrative probabilities), the teacher (GLM-5.3-Flash) receives one raw completion prompt per cyclic rotation of the options and generates no text: the probabilities of the valid letters are read from one next-token distribution after the prefill Answer: ( . Rotation s lists the option keys starting from the s -th, each with its criterion. Letter probabilities are renormalized over the valid letters (their sum, the valid-letter mass, is logged as a quality check), mapped back to option-key order and averaged over the n rotations, which cancels purely positional bias. A temperature can then sharpen the under-confident average; the Klaros-12M training labels use none. For MGB data, the teacher runs on institutional servers.
Extended Data Fig. 2: Shared-document packing on the hybrid backbone of CS1-4B. a , One packed sequence holds the document prefix once, in the Decision 2.0 runtime’s prompt format ( Context: and the document), followed by one segment per question: the task type and question, one block per option, and a closing line ending in Decision: . Segment position indices restart at the end of the document, so the prefix and any one segment are token for token the runtime’s input for that question. The candidate head takes the hidden states at the last token of every option block (option endpoints) and at the segment’s last token (query) and returns one logit per option. b , The block attention mask of the full-attention layers: causal within the document; each segment attends to the whole document and causally to itself, never to another segment. c , The Gated DeltaNet layers run the document once from a zero state and copy its final recurrent state to the start of every segment; the causal convolution at each segment’s first tokens takes its left context from the document’s last tokens. Every question therefore sees the document exactly as in a single-question prompt, its answer does not depend on the other questions, and an extra question costs only its own tokens. The same structure supports deletion attribution: because the text before a deleted sentence is unchanged, the model’s state up to it is computed once and reused, and only the rest of the document and the questions are re-encoded (Methods).
Extended Data Fig. 3: Speed against document length. a , Latency per document at 40 questions on one idle NVIDIA GB10 for synthetic documents of 512 to 16,384 tokens: CS1-4B and the six Decision 2.0 models, each through its released inference code (Decision 2.0: one system_one call per document, not further optimized; Methods); median of 10 timed repetitions (5 for runs longer than 5 s). Open markers, input truncated to the model’s context (at 16,384 tokens; Kai-0.6B at 8,192). b , CS1-4B’s speed-up over each Decision 2.0 model (ratio of latencies; 1, parity). c , CS1-4B reading the document once for all 40 questions (its default) against one forward pass per question, with the same weights; both modes gave the same top answers.
Extended Data Fig. 4: More evidence attributions by CS1-4B on synthetic documents. Drawn as Fig. 3 d–f: necessity left and sufficiency right, as defined in Fig. 3 a; values below zero (a sentence that lowers the answer’s probability) are drawn as zero; bold, the sentences that carry the answer. All text is synthetic, and the bars are model attributions, not ground truth. a , A hedged finding (pneumonia this admission; suspected, P=0.88 ): one radiograph sentence, “atelectasis versus early pneumonia”, carries the answer (necessity 0.71, sufficiency 0.86). b , A laboratory value against a threshold (sodium below 130 mEq/L; yes, P=0.94 ): the sentence with sodium 124 (necessity 0.61, sufficiency 0.57); the later values of 131 and 134 alone point the other way. c , An answer from silence (smoking status; not documented, P=0.93 ): no sentence is necessary, and every sentence alone raises the answer a little, because none mentions smoking. d , A medication stopped in hospital (non-insulin glucose-lowering drug at discharge; no, P=0.78 ): deleting the sentence that metformin was stopped and not restarted flips the answer (necessity 0.56). e , A condition in a relative only (breast cancer; no, P=0.93 ): no sentence is necessary; the family-history sentence does not move the answer towards yes. f , g , The structured record of Fig. 3 f, other questions. f , Highest creatinine of the stay (1.5–2.9 mg/dL, P=0.95 ): the measurements are redundant; no line is necessary, and each of the four elevated values alone gives the answer (sufficiency 0.71–0.77). g , ACE inhibitor or ARB at the end of the stay (stopped before the end, P=0.97 ): no single line moves the answer, which rests on the pattern across days, a limit of single-sentence attribution.
Extended Data Fig. 5: Patient example 2: anticoagulation and bleeding over a whole record. As Fig. 4 , for another MIMIC-IV test patient and question. a , A clinical question and the summarized record of a MIMIC-IV test patient. b , An LLM (GLM-5.3) plans 5 typed sub-questions, each with an aggregation rule and a scope. c , CS1-4B answers every sub-question for every document (620 typed decisions; long documents chunked, nothing truncated); rows are labelled by topic. Level rows (trajectory and latest rules) show each document’s level, light to dark in option order; the other rows show whether a document reports the finding (blue, shaded by probability), reports it absent (orange), gives another answer or is silent. Right, code aggregates each sub-question across documents and time: the aggregated answer and the documents that support it; a warning triangle marks conflicting documents, explained in d. d , The LLM composes a cited answer from the aggregated answers and the evidence sentences (one bullet per sentence). e , CS1 attribution selects 18 evidence sentences; the 13 cited in d are shown, grouped by document (a sentence cited twice shares a line). The LLM plans from a manifest of the record (document types, counts and dates) and never reads the record; at the last step it sees only the aggregated answers and the evidence sentences. Aggregation rules: any, the highest probability over documents; latest, the most recent document that addresses the sub-question; trajectory, the per-document level over time; a conflict is confident support and confident contradiction in different documents. Sub-questions scoped to this admission are asked only of its documents (white cells, out of scope). Evidence sentences are selected by CS1 sufficiency attribution; in compliance with the MIMIC-IV data-use policy, the evidence sentences and any record text quoted in the answer are shown as paraphrases, not verbatim. Dates are shifted by MIMIC-IV de-identification and documents are relabelled D001…in date order. Results shown are from a single run, with GLM-5.3 at its default reasoning effort and CS1-4B (raw outputs).
Extended Data Fig. 6: Patient example 3: cancer course over a whole record. As Fig. 4 , for another MIMIC-IV test patient and question. a , A clinical question and the summarized record of a MIMIC-IV test patient. b , An LLM (GLM-5.3) plans 6 typed sub-questions, each with an aggregation rule and a scope. c , CS1-4B answers every sub-question for every document (756 typed decisions; long documents chunked, nothing truncated); rows are labelled by topic. Level rows (trajectory and latest rules) show each document’s level, light to dark in option order; the other rows show whether a document reports the finding (blue, shaded by probability), reports it absent (orange), gives another answer or is silent. Right, code aggregates each sub-question across documents and time: the aggregated answer and the documents that support it; a warning triangle marks conflicting documents, explained in d. d , The LLM composes a cited answer from the aggregated answers and the evidence sentences (one bullet per sentence). e , CS1 attribution selects 19 evidence sentences; the 13 cited in d are shown, grouped by document (a sentence cited twice shares a line). The LLM plans from a manifest of the record (document types, counts and dates) and never reads the record; at the last step it sees only the aggregated answers and the evidence sentences. Aggregation rules: any, the highest probability over documents; latest, the most recent document that addresses the sub-question; trajectory, the per-document level over time; a conflict is confident support and confident contradiction in different documents. Sub-questions scoped to this admission are asked only of its documents (white cells, out of scope). Evidence sentences are selected by CS1 sufficiency attribution; in compliance with the MIMIC-IV data-use policy, the evidence sentences and any record text quoted in the answer are shown as paraphrases, not verbatim. Dates are shifted by MIMIC-IV de-identification and documents are relabelled D001…in date order. Results shown are from a single run, with GLM-5.3 at its default reasoning effort and CS1-4B (raw outputs).
Klaros-12M source
Labels
Documents
Unique questions
Patients
Pairs
Clinical notes
teacher readout
3,765,728
480,238
690,136
4,511,638
Diagnostic reports
teacher readout
687,809
23,063
458,666
2,460,078
Structured records: outcome forecasting
observed outcomes
431,890
6,013
431,862
433,430
Structured records: structured criteria
exact (rules)
2,581,620
497,312
2,425,282
5,011,302
Total
7,467,047
1,006,626
3,643,798
12,416,448
Extended Data Table 1: Fine-tuning data of CS1-4B. Klaros-12M only, in four groups; no MIMIC, other public or synthetic data. Splits are by patient; the test split, MGBQA-1M, is used only for the internal evaluation (Extended Data Table 3). Pairs are document–question training pairs; a patient can contribute to several groups, so the total counts patients once.
Internal
Open decision models
Task
Key
Pairs
CS1- 4B
JEV- 27B
Clef- 27B
pplx- 27B
Vega- 27B
Lux- 9B
Nox- 4B
Sol- 2B
Eos- 0.8B
Kai- 0.6B
BioDec- 4B
Most common
1 Patient timelines (rule keys)
Care process
rule
47,091
95.4
72.1
66.2
74.5
64.2
58.7
49.2
47.8
43.9
38.6
31.2
46.6
Conditions
rule
97,491
98.4
93.1
90.9
92.2
94.3
89.0
85.3
74.2
69.1
60.1
38.7
64.1
Demographics
rule
31,782
95.8
90.0
89.7
90.3
88.7
85.6
83.9
76.7
81.4
75.7
48.3
46.4
Measurements
rule
102,182
98.7
90.1
89.2
90.9
90.5
81.9
79.5
60.8
59.1
51.0
29.7
49.9
Extended Data Table 3: Internal test on held-out MGB patients, per task. Balanced accuracy (%; mean recall over the key’s answer classes per question, averaged over questions) of every system on the 28 tasks of the conformed MGBQA-1M test set (963,408 held-out patients, one test question each), grouped by task type (Fig. 2a) and kind of key; the means of the rule-, outcome- and LLM-keyed rows are plotted in Fig. 2b. Key: rule, computed by a rule from the record or structured tables; outcome, observed after the decision point; LLM, written by an open LLM (Gemma-4-26B-A4B [ 79 , 80 ] ); teacher (task in italics), the GLM-5.3-Flash teacher’s answer, so these tasks measure agreement with the teacher and are secondary. Raw outputs; per-type temperature scaling leaves the top answer, and so balanced accuracy, unchanged on every task. Most common, the most common training answer per question (scored on pairs whose question occurs in training). Raw accuracy is dominated by the most common answer on several note tasks (for example 98.6% on addenda and 86.3% on coding review), so balanced accuracy is the primary metric. Bold, best system per task.
Extended Data Table 5: Serializer comparison on MIMIC-IV. Balanced accuracy (%) or mean AUROC of every system on the same documents rendered in the timeline layout (periods counted back from the index day; the layout of the Klaros-12M training timelines and of the eICU-CRD and EHRSHOT sets) and in the encounter layout (admissions; the layout of the MIMIC-IV sets), scored on the identical pairs valid under both; encounter − timeline, the paired difference with its 95% interval. Timelines: 2,377 MIMIC-IV timeline questions built as the public timeline sets (one per document; 352 concepts), not one of the public data sets. Composite definitions and prognosis: the documents of MIMIC-Composite and MIMIC-Prognosis, whose encounter-layout values are those of Extended Data Table 4. On timelines every model with valid predictions, CS1-4B included, scored higher in the encounter layout.
Decision 2.0
Outcome ( n / positives)
JEV- 27B
Clef- 27B
pplx- 27B
Kai- 0.6B
Eos- 0.8B
Sol- 2B
Nox- 4B
Lux- 9B
Vega- 27B
BioDec- 4B
MedDec- 9B
MedDec- 27B
CS1- 4B
MIMIC-Prognosis (MIMIC-IV, TD2 encounter layout; 2,398 patients, one question each)
AKI within 48 h (224/16)
0.670
0.633
0.701
0.664
0.325
0.582
0.697
0.669
0.750
0.709
0.639
0.674
0.690
1-year death (165/17)
0.913
0.909
0.908
0.631
0.604
0.742
0.860
0.879
0.907
0.896
0.885
0.909
0.890
30-day emergency revisit (178/30)
0.613
0.592
0.636
0.552
0.579
0.616
0.646
0.668
0.627
0.557
0.553
0.585
0.624
Intensive care after 24 h (174/20)
0.797
0.811
0.755
0.526
0.375
0.440
0.571
0.808
0.771
0.762
0.611
0.868
0.643
Extended Data Table 6: Outcome forecasting on the three prognosis sets. AUROC per outcome (raw probabilities; risk = P(yes), or the probability of the event option for EHRSHOT question kinds with a clear event option) for the outcomes with at least 10 positives and 10 negatives (MIMIC-Prognosis, 7 of 12 outcomes, eICU-Prognosis, 5 of 6 outcomes and EHRSHOT-Prognosis, 15 of 20 outcomes (19 with any positive); n / positives per outcome in parentheses), and their mean, the value of Fig. 2c and Extended Data Table 4. Mean ECE, over all outcomes with positives; Brier > prev., outcomes with positives whose Brier score is worse than always predicting the prevalence. One question per patient. The LLM teacher was not run on these sets.
Large language models (LLMs) often make discrete decisions, such as flagging phishing or escalating to a human. System-1 models, such as Jev and the OpenAI Decisions API, return probabilities over options in a single forward pass, while System-2 approaches generate reasoning steps. Which decisions can System-1 models handle, and which require generative reasoning? We introduce System1-Arena, a benchmark that gives all models the same decision requests and scores their probabilities. It contains 92k questions across 105 tasks from 268 public sources, in four groups: text classification and knowledge, applied decisions, evaluation and control, and agent and action. It also includes 138 closed-loop environments, from web shopping and SWE-bench coding control to robotics, games, and driving; in real-time mode, the world evolves while a model decides. We evaluate 30 models: 7 closed LLMs, 13 open LLMs read at the answer token, and 10 System-1 models. The largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency. A reasoning LLM is the most accurate, at about 70x the cost of a fast LLM. In closed loops, large open LLMs succeed most often (up to 32.9%); small System-1 models rarely succeed.
LLM-based decision-making systems, also called System One models, have recently gained popularity, most notably through Jev (Almeida, 2026). Given a typed input state, a question, and a set of options, they return a probability for each option in a single forward pass, without text generation. We show that existing open instruction-tuned language models can achieve comparable performance without any additional training, by reading out answers from letters (A, B, C, etc.) in the model's output logits at the last token position. Furthermore, we show how a simple supervised fine-tuning recipe can improve their performance further. We evaluate four open instruction-tuned language models: Gemma 4 31B, Gemma 4 12B, Qwen3.6-27B, and Qwen3.5-4B. We fine-tune each using LoRA and a decision head initialized from a slice of the language model head. Furthermore, we introduce OpenDecisions, a novel 2,352-question validation dataset designed to reflect the task composition of JevBench (JevBench main- tainers, 2026) and which computes a Capability score based on accuracy and calibration metrics. Without fine-tuning, Gemma 4 31B scores 79.6 on OpenDecisions, compared to Jev's 80.1. After undergoing our fine-tuning recipe, it scores 84.4 on OpenDecisions and 81.5 on JevBench v1.7, outperforming Jev by 4.3 and 4.4 Capability points respectively and setting a new state-of-the-art on OpenDecisions. On a suite of public decision datasets, our models achieve accuracy comparable to Jev, with lower calibration error. Our results show that open language models already possess System One capabilities, and with simple fine-tuning they outperform Jev on a wide range of decision tasks. We release our models, serving code, and OpenDecisions.
Jonathan Geuter, Andrew Cheng, David Alvarez-Melis
Jev has attracted considerable attention for its rapid, generalizable decision-making, yet its medical reliability remains underexplored. To address this gap, our study compares various types of system-one decision models across 15 medical datasets comprising 8,742 instances under a unified protocol. Beyond accuracy, we assess decision stability and the ability to recognize unreliable decisions. Our results show that Jev rivals leading models on primarily text-driven tasks but is less effective on numerically driven tasks, revealing a distinct capability boundary. Under question-format variations, Jev maintains strong decision consistency and small accuracy fluctuations. StartLux-27B exhibits comparable consistency, while D3-27B achieves higher overall accuracy and a lower mean decision reversal rate. But these strengths do not extend to reliable abstention awareness: most evaluated models show limitations in both recognizing missing correct answers and using optional abstention. Moreover, we find that Jev's probability-based answer selection can be improved by changing how decisions are scored: normalizing option-wise binary scores yields higher retained accuracy than choice scoring. Taken together, our findings provide early guidance for improving and applying decision models in medicine.