Authors: Xiufeng Yang, Konstantinos Vrettos, Zeineb Ben Chaaben, Fabian Drexel, Jannik Lübberstedt, Johannes Moll, Anirudh Narayanan, Frederike Stumpf, Cosmin I. Bercea, Jiawen Wang, Evangelia E. Vassalou, Krischan Braitsch, Aymen Meddeb, Sebastian Ziegelmayer, Markus Graf, Marcus R. Makowski, Lisa Adams, Nassir Navab, Michail E. Klontzas, Keno Bressem
Organizations: Department of Diagnostic and Interventional Radiology, School of Medicine and Health, Technical University of Munich, Munich, Germany · Artificial Intelligence and Translational Imaging Lab, Department of Radiology, School of Medicine, University of Crete, Heraklion, Greece · Department of Medical Imaging, University Hospital of Heraklion, Heraklion, Crete, Greece · Department of Medicine III, School of Medicine and Health, Technical University of Munich, Munich, Germany · Chair for Computer Aided Medical Procedures and Augmented Reality, School of Computation, Information and Technology, Technical University of Munich, Munich, Germany · Division of Radiology, Department for Clinical Science, Intervention and Technology (CLINTEC), Karolinska Institutet, Stockholm, Sweden · National Center for Tumor Diseases West, Essen, Germany · Institute of Artificial Intelligence in Medicine, University Hospital Essen, Essen, Germany · Department of Diagnostic and Interventional Radiology and Neuroradiology, University Hospital Essen, Essen, Germany
Many clinical decisions are a choice among a few fixed options, taken quickly and in large numbers. Such decisions also need an uncertainty estimate to know when to defer to a clinician. Generative language models answer slowly in free text and state their confidence poorly. In contrast, System 1 models answer zero-shot with a probability for each answer. However, their performance across clinical tasks is unknown. Four System 1 models, two proprietary and two with open weights, were evaluated zero-shot on 86 public tasks from 14 clinical domains in 20 languages, spanning examination questions and medical imaging. Tasks were assigned to six capability tiers before analysis. The primary outcome was the area under the receiver-operating-characteristic curve (AUROC). Secondary outcomes included calibration, language effects and comparisons with published results and generative models. Across 179,111 cases, the mean AUROC of the best model per task was 0.94 for medical knowledge, 0.93 for text understanding, 0.83 for clinical judgement from text, 0.89 for image perception, 0.84 for 2D image findings and 0.69 for volumetric findings. Imaging tasks scored 0.09 (95% CI, 0.06 to 0.11) lower than text tasks (P < 0.001). The best model matched or exceeded the best published result on 13 of 26 text tasks and on 8 of 35 imaging tasks. Triage nurses outperformed the best model (accuracy, 0.85 vs. 0.56). One decision took 0.34 to 0.44 seconds and 24 to 81 joules on open-weight models run locally. Accuracy on identical questions fell by up to 39 percentage points from English to Yoruba. Generative models of the same family were 11 to 76 times slower, with 3 to 5 percentage points higher balanced accuracy. System 1 models decide quickly, cheaply and with usable confidence when the answer is stated in a text or recalled from medical knowledge, but not when it must be judged or read from an image.
Figures & tables
Figure 1: Study Design. Panel A shows how a System 1 model answers. Text, one or more images, or both enters the model with typed questions, and the model returns a probability for each option in one forward pass. The generative model of the same family served as comparator on 32 tasks. Panel B shows the 86 tasks by capability tier and input type. Hatched tiles are language versions of another task and were analysed for language effects only.
Figure 2: Capability Map. Panel A shows the AUROC of each primary task (columns) for the four System 1 models, with tasks grouped by capability tier and sorted by the best model. Grey vertical lines span the four models. Horizontal bars and shaded bands show the random-effects mean AUROC of the best model per task with its 95% CI, and the weakest task of each tier is labelled. Panels B and C show the random-effects mean per tier and model with 95% CIs for AUROC and for chance-corrected balanced accuracy, where 0 is chance and 1 is perfect. Refused or failed questions were scored as wrong.
Figure 3: System 1 Models, Published Results and Clinicians. Panel A shows text tasks and Panel B imaging tasks. For each task with a published result on the same dataset, label set and metric, the best of the four System 1 models (coloured marker) is shown with the best published result (open circle), which can be a task-specific model, a generative language model or a rule, and with clinicians or expert annotators (open triangle). Values of the System 1 models are computed on the questions that match the published result and can differ from the task-level AUROC. Acc denotes accuracy and BA balanced accuracy.
Figure 4: Calibration and Language. Panels A and B show the share of correct answers against the probability the model gave its answer, pooled over text tasks in Panel A and over imaging tasks in Panel B. The diagonal marks perfect calibration. Panel C shows, for each model, the accuracy of all answers (open marker) and of its most confident half (filled marker) on text and imaging tasks. Panel D shows accuracy on the same 1,089 medical examination questions in seven languages.
Figure 5: System 1 and Generative Models. Panel A shows, for each task, the balanced accuracy of the System 1 model minus that of the generative model of the same family on identical cases, with the mean and 95% CI (task bootstrap) for text and imaging tasks. The pairs are GPT-6 Luna Decisions and GPT-6 Luna, Clef and Qwen3.8-27B, and Clef-flash and Qwen3.5-9B. Panels B and C show the cost per 1,000 requests and the median latency per request on logarithmic scales, with filled markers for System 1 models and open markers for generative models.
Figure 6: Energy of Locally Run System 1 Models. Panel A shows the GPU energy per 1,000 requests and Panel B the GPU time per request of Clef (8-bit) and Clef-flash (bf16), run locally on one NVIDIA RTX 5090, for each primary task by capability tier. Energy is the whole-board GPU energy. Bars show the median over tasks, and n is the number of tasks per tier. No clinical-judgement task was run locally.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1: AUROC of every task (rows, numbered as in Table S1 ) and model (columns). Blue shades mark values above chance (0.5) and red shades values below chance.
Figure S2: Primary endpoint of each System 1 model minus that of its backbone on 20 tasks, with 95% bias-corrected and accelerated bootstrap CIs. Filled markers indicate Holm-adjusted P < 0.05 within the task.
Figure S3: Study-level AUROC of volumetric tasks with yes-or-no questions, averaged over questions, when the slice probabilities of a study were combined by their maximum (primary rule), the mean of the three highest, or the mean over all slices. Each panel shows one task and each line one model.
Spanish healthcare specialisation exam MCQ, Spanish original (HEAD-QA)
HEAD-QA 89
Medical knowledge
text
Spanish
2,675
1
005
Medical subsets of MMLU
MMLU 28
Medical knowledge
text
English
1,089
1
006
Health questions of MMLU-Pro
MMLU-Pro 92
Medical knowledge
text
English
818
1
Appendix
Table S1: Tasks. Tasks 004, 016, 017, 018, 019, 020, 021, 075 are language versions of another task. MCQ denotes multiple-choice questions, VQA visual question answering, and ED emergency department.
No.
Task
Clef
Clef-flash
GPT-6 Luna Decisions
Jev 1.14
001
MedQA (USMLE)
0.97 (0.96–0.97)
0.92 (0.90–0.93)
0.98 (0.97–0.98)
0.99 (0.98–0.99)
002
MedMCQA (India)
0.89 (0.89–0.90)
0.87 (0.86–0.88)
0.92 (0.92–0.93)
0.93 (0.93–0.94)
003
HEAD-QA (English)
0.96 (0.95–0.96)
0.95 (0.94–0.95)
0.96 (0.96–0.97)
0.97 (0.96–0.97)
004
HEAD-QA (Spanish)
0.96 (0.95–0.97)
0.95 (0.94–0.95)
0.97 (0.96–0.97)
0.97 (0.97–0.98)
005
MMLU medical
0.99 (0.98–0.99)
0.99 (0.98–0.99)
0.98 (0.98–0.99)
0.99 (0.98–0.99)
006
MMLU-Pro health
0.97 (0.96–0.97)
0.96 (0.95–0.97)
0.96 (0.94–0.96)
0.97 (0.96–0.98)
Appendix
Table S2: AUROC per task and model with 95% CI from 1,000 bootstrap resamples of cases. Refused or failed questions were scored as wrong with the label prevalence as probability.
Tier
Tasks
Metric
Clef
Clef-flash
GPT-6 Luna Decisions
Jev 1.14
Best
Medical knowledge
16
AUROC
0.90 (0.83–0.96)
0.88 (0.81–0.94)
0.94 (0.91–0.96)
0.93 (0.89–0.97)
0.94 (0.92–0.97)
Chance-corrected BA
0.65 (0.47–0.83)
0.60 (0.44–0.75)
0.73 (0.63–0.84)
0.72 (0.57–0.87)
0.74 (0.63–0.85)
Text understanding
13
AUROC
0.92 (0.90–0.93)
0.89 (0.87–0.92)
0.89 (0.87–0.92)
0.92 (0.90–0.94)
0.93 (0.91–0.94)
Chance-corrected BA
0.63 (0.54–0.73)
0.56 (0.43–0.69)
0.61 (0.51–0.71)
0.65 (0.55–0.75)
0.66 (0.57–0.75)
Clinical judgement from text
6
AUROC
0.79 (0.70–0.87)
0.73 (0.64–0.81)
0.80 (0.66–0.94)
0.82 (0.73–0.90)
0.83 (0.73–0.93)
Chance-corrected BA
0.34 (0.14–0.55)
0.24 (0.10–0.38)
0.41 (0.10–0.73)
0.42 (0.12–0.72)
0.44 (0.12–0.77)
Appendix
Table S3: Random-effects mean per capability tier with 95% CI. Best denotes the best of the four models on each task.
No.
Task
System 1
Generative
BA S1
BA gen.
USD/1k S1
USD/1k gen.
001
MedQA (USMLE)
Clef
Qwen3.8-27B
0.84
0.92
0.08
1.40
027
Lung cancer T from report
Clef
Qwen3.8-27B
0.65
0.83
0.11
1.62
028
BI-RADS from report
Clef
Qwen3.8-27B
0.54
0.35
0.12
1.64
029
pT stage from pathology report
Clef
Qwen3.8-27B
0.85
0.86
0.32
0.96
030
Trial report inference
Clef
Qwen3.8-27B
0.90
0.89
0.15
1.27
031
PubMedQA
Clef
Qwen3.8-27B
0.57
0.70
0.12
1.03
Appendix
Table S4: System 1 model against the generative model of its family on the same cases, 100 per task and 50 for Clef and Qwen3.8-27B. BA denotes balanced accuracy, USD/1k the cost in US dollars per 1,000 requests, S1 the System 1 model and gen. the generative model.
No.
Task
Type
Reference
Source
Metric
Ref.
Best S1
001
MedQA (USMLE)
Generative language model
o1-preview, zero-shot
Nori 2024
accuracy
0.96
0.89
001
MedQA (USMLE)
Task-specific model
Med-Gemini-L 1.0 (self-training, uncertainty-guided web search)
Saab 2024
accuracy
0.91
0.89
002
MedMCQA (India)
Generative language model
o1-preview, zero-shot
Nori 2024
accuracy
0.84
0.75
002
MedMCQA (India)
Task-specific model
Med-PaLM 2 (ensemble refinement)
Singhal 2025
accuracy
0.72
0.75
003
HEAD-QA (English)
Generative language model
GPT-5, zero-shot
Yang 2026
accuracy
0.90
0.90
005
MMLU medical
Generative language model
GPT-5
Wang 2025
accuracy
0.96
0.96
Appendix
Table S5: Published results with the same dataset, label set and metric, which can use other splits, and the best of the four System 1 models on the questions that match the published result (Best S1). Best S1 can therefore differ from the task-level AUROC in Table S2 .