Authors: Xiufeng Yang, Konstantinos Vrettos, Zeineb Ben Chaaben, Fabian Drexel, Jannik Lübberstedt, Johannes Moll, Anirudh Narayanan, Frederike Stumpf, Cosmin I. Bercea, Jiawen Wang, Evangelia E. Vassalou, Krischan Braitsch, Aymen Meddeb, Sebastian Ziegelmayer, Markus Graf, Marcus R. Makowski, Lisa Adams, Nassir Navab, Michail E. Klontzas, Keno Bressem
Organizations: Department of Diagnostic and Interventional Radiology, School of Medicine and Health, Technical University of Munich, Munich, Germany · Artificial Intelligence and Translational Imaging Lab, Department of Radiology, School of Medicine, University of Crete, Heraklion, Greece · Department of Medical Imaging, University Hospital of Heraklion, Heraklion, Crete, Greece · Department of Medicine III, School of Medicine and Health, Technical University of Munich, Munich, Germany · Chair for Computer Aided Medical Procedures and Augmented Reality, School of Computation, Information and Technology, Technical University of Munich, Munich, Germany · Division of Radiology, Department for Clinical Science, Intervention and Technology (CLINTEC), Karolinska Institutet, Stockholm, Sweden · National Center for Tumor Diseases West, Essen, Germany · Institute of Artificial Intelligence in Medicine, University Hospital Essen, Essen, Germany · Department of Diagnostic and Interventional Radiology and Neuroradiology, University Hospital Essen, Essen, Germany
Many clinical decisions are a choice among a few fixed options, taken quickly and in large numbers. Such decisions also need an uncertainty estimate to know when to defer to a clinician. Generative language models answer slowly in free text and state their confidence poorly. In contrast, System 1 models answer zero-shot with a probability for each answer. However, their performance across clinical tasks is unknown. Four System 1 models, two proprietary and two with open weights, were evaluated zero-shot on 86 public tasks from 14 clinical domains in 20 languages, spanning examination questions and medical imaging. Tasks were assigned to six capability tiers before analysis. The primary outcome was the area under the receiver-operating-characteristic curve (AUROC). Secondary outcomes included calibration, language effects and comparisons with published results and generative models. Across 179,111 cases, the mean AUROC of the best model per task was 0.94 for medical knowledge, 0.93 for text understanding, 0.83 for clinical judgement from text, 0.89 for image perception, 0.84 for 2D image findings and 0.69 for volumetric findings. Imaging tasks scored 0.09 (95% CI, 0.06 to 0.11) lower than text tasks (P < 0.001). The best model matched or exceeded the best published result on 13 of 26 text tasks and on 8 of 35 imaging tasks. Triage nurses outperformed the best model (accuracy, 0.85 vs. 0.56). One decision took 0.34 to 0.44 seconds and 24 to 81 joules on open-weight models run locally. Accuracy on identical questions fell by up to 39 percentage points from English to Yoruba. Generative models of the same family were 11 to 76 times slower, with 3 to 5 percentage points higher balanced accuracy. System 1 models decide quickly, cheaply and with usable confidence when the answer is stated in a text or recalled from medical knowledge, but not when it must be judged or read from an image.
Figures & tables
Figure 1: Study Design. Panel A shows how a System 1 model answers. Text, one or more images, or both enters the model with typed questions, and the model returns a probability for each option in one forward pass. The generative model of the same family served as comparator on 32 tasks. Panel B shows the 86 tasks by capability tier and input type. Hatched tiles are language versions of another task and were analysed for language effects only.
Figure 2: Capability Map. Panel A shows the AUROC of each primary task (columns) for the four System 1 models, with tasks grouped by capability tier and sorted by the best model. Grey vertical lines span the four models. Horizontal bars and shaded bands show the random-effects mean AUROC of the best model per task with its 95% CI, and the weakest task of each tier is labelled. Panels B and C show the random-effects mean per tier and model with 95% CIs for AUROC and for chance-corrected balanced accuracy, where 0 is chance and 1 is perfect. Refused or failed questions were scored as wrong.
Figure 3: System 1 Models, Published Results and Clinicians. Panel A shows text tasks and Panel B imaging tasks. For each task with a published result on the same dataset, label set and metric, the best of the four System 1 models (coloured marker) is shown with the best published result (open circle), which can be a task-specific model, a generative language model or a rule, and with clinicians or expert annotators (open triangle). Values of the System 1 models are computed on the questions that match the published result and can differ from the task-level AUROC. Acc denotes accuracy and BA balanced accuracy.
Figure 4: Calibration and Language. Panels A and B show the share of correct answers against the probability the model gave its answer, pooled over text tasks in Panel A and over imaging tasks in Panel B. The diagonal marks perfect calibration. Panel C shows, for each model, the accuracy of all answers (open marker) and of its most confident half (filled marker) on text and imaging tasks. Panel D shows accuracy on the same 1,089 medical examination questions in seven languages.
Figure 5: System 1 and Generative Models. Panel A shows, for each task, the balanced accuracy of the System 1 model minus that of the generative model of the same family on identical cases, with the mean and 95% CI (task bootstrap) for text and imaging tasks. The pairs are GPT-6 Luna Decisions and GPT-6 Luna, Clef and Qwen3.8-27B, and Clef-flash and Qwen3.5-9B. Panels B and C show the cost per 1,000 requests and the median latency per request on logarithmic scales, with filled markers for System 1 models and open markers for generative models.
Figure 6: Energy of Locally Run System 1 Models. Panel A shows the GPU energy per 1,000 requests and Panel B the GPU time per request of Clef (8-bit) and Clef-flash (bf16), run locally on one NVIDIA RTX 5090, for each primary task by capability tier. Energy is the whole-board GPU energy. Bars show the median over tasks, and n is the number of tasks per tier. No clinical-judgement task was run locally.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1: AUROC of every task (rows, numbered as in Table S1 ) and model (columns). Blue shades mark values above chance (0.5) and red shades values below chance.
Figure S2: Primary endpoint of each System 1 model minus that of its backbone on 20 tasks, with 95% bias-corrected and accelerated bootstrap CIs. Filled markers indicate Holm-adjusted P < 0.05 within the task.
Figure S3: Study-level AUROC of volumetric tasks with yes-or-no questions, averaged over questions, when the slice probabilities of a study were combined by their maximum (primary rule), the mean of the three highest, or the mean over all slices. Each panel shows one task and each line one model.
Spanish healthcare specialisation exam MCQ, Spanish original (HEAD-QA)
HEAD-QA 89
Medical knowledge
text
Spanish
2,675
1
005
Medical subsets of MMLU
MMLU 28
Medical knowledge
text
English
1,089
1
006
Health questions of MMLU-Pro
MMLU-Pro 92
Medical knowledge
text
English
818
1
Appendix
Table S1: Tasks. Tasks 004, 016, 017, 018, 019, 020, 021, 075 are language versions of another task. MCQ denotes multiple-choice questions, VQA visual question answering, and ED emergency department.
No.
Task
Clef
Clef-flash
GPT-6 Luna Decisions
Jev 1.14
001
MedQA (USMLE)
0.97 (0.96–0.97)
0.92 (0.90–0.93)
0.98 (0.97–0.98)
0.99 (0.98–0.99)
002
MedMCQA (India)
0.89 (0.89–0.90)
0.87 (0.86–0.88)
0.92 (0.92–0.93)
0.93 (0.93–0.94)
003
HEAD-QA (English)
0.96 (0.95–0.96)
0.95 (0.94–0.95)
0.96 (0.96–0.97)
0.97 (0.96–0.97)
004
HEAD-QA (Spanish)
0.96 (0.95–0.97)
0.95 (0.94–0.95)
0.97 (0.96–0.97)
0.97 (0.97–0.98)
005
MMLU medical
0.99 (0.98–0.99)
0.99 (0.98–0.99)
0.98 (0.98–0.99)
0.99 (0.98–0.99)
006
MMLU-Pro health
0.97 (0.96–0.97)
0.96 (0.95–0.97)
0.96 (0.94–0.96)
0.97 (0.96–0.98)
Appendix
Table S2: AUROC per task and model with 95% CI from 1,000 bootstrap resamples of cases. Refused or failed questions were scored as wrong with the label prevalence as probability.
Tier
Tasks
Metric
Clef
Clef-flash
GPT-6 Luna Decisions
Jev 1.14
Best
Medical knowledge
16
AUROC
0.90 (0.83–0.96)
0.88 (0.81–0.94)
0.94 (0.91–0.96)
0.93 (0.89–0.97)
0.94 (0.92–0.97)
Chance-corrected BA
0.65 (0.47–0.83)
0.60 (0.44–0.75)
0.73 (0.63–0.84)
0.72 (0.57–0.87)
0.74 (0.63–0.85)
Text understanding
13
AUROC
0.92 (0.90–0.93)
0.89 (0.87–0.92)
0.89 (0.87–0.92)
0.92 (0.90–0.94)
0.93 (0.91–0.94)
Chance-corrected BA
0.63 (0.54–0.73)
0.56 (0.43–0.69)
0.61 (0.51–0.71)
0.65 (0.55–0.75)
0.66 (0.57–0.75)
Clinical judgement from text
6
AUROC
0.79 (0.70–0.87)
0.73 (0.64–0.81)
0.80 (0.66–0.94)
0.82 (0.73–0.90)
0.83 (0.73–0.93)
Chance-corrected BA
0.34 (0.14–0.55)
0.24 (0.10–0.38)
0.41 (0.10–0.73)
0.42 (0.12–0.72)
0.44 (0.12–0.77)
Appendix
Table S3: Random-effects mean per capability tier with 95% CI. Best denotes the best of the four models on each task.
No.
Task
System 1
Generative
BA S1
BA gen.
USD/1k S1
USD/1k gen.
001
MedQA (USMLE)
Clef
Qwen3.8-27B
0.84
0.92
0.08
1.40
027
Lung cancer T from report
Clef
Qwen3.8-27B
0.65
0.83
0.11
1.62
028
BI-RADS from report
Clef
Qwen3.8-27B
0.54
0.35
0.12
1.64
029
pT stage from pathology report
Clef
Qwen3.8-27B
0.85
0.86
0.32
0.96
030
Trial report inference
Clef
Qwen3.8-27B
0.90
0.89
0.15
1.27
031
PubMedQA
Clef
Qwen3.8-27B
0.57
0.70
0.12
1.03
Appendix
Table S4: System 1 model against the generative model of its family on the same cases, 100 per task and 50 for Clef and Qwen3.8-27B. BA denotes balanced accuracy, USD/1k the cost in US dollars per 1,000 requests, S1 the System 1 model and gen. the generative model.
No.
Task
Type
Reference
Source
Metric
Ref.
Best S1
001
MedQA (USMLE)
Generative language model
o1-preview, zero-shot
Nori 2024
accuracy
0.96
0.89
001
MedQA (USMLE)
Task-specific model
Med-Gemini-L 1.0 (self-training, uncertainty-guided web search)
Saab 2024
accuracy
0.91
0.89
002
MedMCQA (India)
Generative language model
o1-preview, zero-shot
Nori 2024
accuracy
0.84
0.75
002
MedMCQA (India)
Task-specific model
Med-PaLM 2 (ensemble refinement)
Singhal 2025
accuracy
0.72
0.75
003
HEAD-QA (English)
Generative language model
GPT-5, zero-shot
Yang 2026
accuracy
0.90
0.90
005
MMLU medical
Generative language model
GPT-5
Wang 2025
accuracy
0.96
0.96
Appendix
Table S5: Published results with the same dataset, label set and metric, which can use other splits, and the best of the four System 1 models on the questions that match the published result (Best S1). Best S1 can therefore differ from the task-level AUROC in Table S2 .
Clinical care, health-system operations and biomedical discovery depend on extracting reliable answers from patient records, yet applying diverse clinical questions across entire populations remains computationally expensive. Here we present Clinical System One (CS1), a 4-billion-parameter decision model for fast, typed judgments from clinical records, along with two agentic harnesses in which CS1 works together with a generalist language model. CS1-4B is trained on 12.4 million questions and answers from 3.6 million patients across a multi-hospital health system, combining structured-data keys, observed outcomes and language-model supervision across clinical notes, diagnostic reports and patient timelines. It answers multiple typed questions in a single forward pass without task-specific retraining, and each answer carries a probability that can route uncertain cases to human review. On 963,408 held-out patients, CS1-4B outperformed open decision models up to 27 billion parameters on 13 of 14 tasks scored independently of its training teacher. Across 21 external data sets, its aggregate performance approached that of the strongest tested 27-billion-parameter model, with 6.4-fold faster inference in a matched workload. Integrating CS1-4B with a language model agent produced cited answers over complete multi-year patient histories and screened a patient population against the eligibility criteria of 17 registered clinical trials, placing four times as many judged-eligible patients among its top 20 candidates as keyword search while using 690-fold fewer language-model tokens than a language model reading every record. These findings support a division of labour in clinical AI, with small decision models performing repeated record reading and generative models formulating questions and synthesizing evidence.
Andrew Zhang, Trinh T. L. Vuong, Sophia J. Wagner +14
Mass General Brigham AI Institute, Boston, MA. · Harvard Medical School, Boston, MA. · Massachusetts Institute of Technology, Cambridge, MA.
Large language models (LLMs) often make discrete decisions, such as flagging phishing or escalating to a human. System-1 models, such as Jev and the OpenAI Decisions API, return probabilities over options in a single forward pass, while System-2 approaches generate reasoning steps. Which decisions can System-1 models handle, and which require generative reasoning? We introduce System1-Arena, a benchmark that gives all models the same decision requests and scores their probabilities. It contains 92k questions across 105 tasks from 268 public sources, in four groups: text classification and knowledge, applied decisions, evaluation and control, and agent and action. It also includes 138 closed-loop environments, from web shopping and SWE-bench coding control to robotics, games, and driving; in real-time mode, the world evolves while a model decides. We evaluate 30 models: 7 closed LLMs, 13 open LLMs read at the answer token, and 10 System-1 models. The largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency. A reasoning LLM is the most accurate, at about 70x the cost of a fast LLM. In closed loops, large open LLMs succeed most often (up to 32.9%); small System-1 models rarely succeed.
Jev has attracted considerable attention for its rapid, generalizable decision-making, yet its medical reliability remains underexplored. To address this gap, our study compares various types of system-one decision models across 15 medical datasets comprising 8,742 instances under a unified protocol. Beyond accuracy, we assess decision stability and the ability to recognize unreliable decisions. Our results show that Jev rivals leading models on primarily text-driven tasks but is less effective on numerically driven tasks, revealing a distinct capability boundary. Under question-format variations, Jev maintains strong decision consistency and small accuracy fluctuations. StartLux-27B exhibits comparable consistency, while D3-27B achieves higher overall accuracy and a lower mean decision reversal rate. But these strengths do not extend to reliable abstention awareness: most evaluated models show limitations in both recognizing missing correct answers and using optional abstention. Moreover, we find that Jev's probability-based answer selection can be improved by changing how decisions are scored: normalizing option-wise binary scores yields higher retained accuracy than choice scoring. Taken together, our findings provide early guidance for improving and applying decision models in medicine.