cs.CLAug 23, 2026

Teaching LLMs How ICU Physicians Approach Clinical Reasoning Through OMOP-Aligned Retrieval Improves Reasoning Across Clinical Domains

Authors: Miguel Contreras, Scott Siegel, Subhash Nerella, Jessica Sena, Jiaqing Zhang, Heng Sun, Hruday Tej Akkaladevi, Peiyu Lu, 

Organizations: Department of Biomedical Engineering, University of Florida, Gainesville, FL, USA. · Intelligent Clinical Care Center (IC3), University of Florida, Gainesville, FL, USA. · Department of Electrical and Computer Engineering, University of Florida, Gainesville, FL, USA. · Google, Seattle, WA, USA. · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Clinical decision-making relies on identifying relevant patient information to guide diagnosis and treatment, a challenge that is especially difficult in the data-dense and rapidly changing intensive care unit (ICU). Large language models (LLMs) could support this task. However, existing applications and datasets mostly emphasize surface-level retrieval or factual recall rather than the inductive and deductive reasoning clinicians practice to select and reason over decision-relevant evidence. We hypothesized that training LLMs on expert ICU reasoning could yield clinical reasoning skills that generalize beyond critical care. Here we introduce ICU-REACT, a reasoning dataset developed with 19 clinicians through a clinician-in-the-loop framework to teach LLMs to perform information retrieval and context-aware clinical reasoning in the ICU. Using ICU-REACT, we fine-tuned Clin-REACT models spanning 8B-70B parameters and three model families. Across five clinical reasoning benchmarks, Clin-REACT consistently outperformed its backbone models and open-source general-purpose and medical LLMs. Gains extended to different tasks including script concordance tests, and downstream diagnosis and treatment tasks. These findings suggest that expert reasoning supervision in critical care can improve broader clinical reasoning, although prospective evaluation is needed before real-world clinical use.

Figures & tables

Figure 1: Workflow overview of the ICU-REACT and Clin-REACT pipeline. (A) Seed dataset creation: GPT-4.1, selected for its state-of-the-art instruction-following performance at the time of initial generation, produces candidate samples that are stored in MongoDB and reviewed by clinicians through an online annotation tool. Approved samples are then refined with GPT-5.2, the state-of-the-art model available at the time of the experiments, using clinicians’ written feedback to produce the ICU-REACT seed set ( n=213n=213 ). (B) Dataset augmentation: the seed set is divided into a training seed ( n=142n=142 ) and held-out test set ( n=71n=71 ). Clinician-curated ICU topics and nn randomly sampled training examples are used as few-shot context for GPT-5.2 to generate new questions. GPT-4o-mini is intentionally used to produce imperfect initial answers requiring refinement, after which GPT-5.2 generates improved answers using the question and initial response. This process is repeated for kk rounds until the target training-set size is reached. (C) Model training: Llama 3.1 8B Instruct, Baichuan M1 14B Instruct, Gemma 4 31B, and Llama 3.3 70B Instruct are fine-tuned through a single supervised fine-tuning stage to generate refined answers and corresponding reasoning from the patient context, decision question, and initial answer. ICU-REACT-Train-Small contained 10,000 samples and was used for the 8B model; ICU-REACT-Train-Medium contained 5,307 hard reasoning-refinement samples and was used for the 14B and 31B models; and ICU-REACT-Train-Large contained 27,973 reasoning-refinement samples and was used for the 70B model. Fine-tuning produced the Clin-REACT 8B, 14B, 31B, and 70B models. (D) Evaluation: Clin-REACT models are compared with open-source general-purpose and medical LLMs on the held-out ICU-REACT test set and four external benchmarks (SCT-Bench, ER-Reason, MedRBench, and VivaBench) using automated and LLM-judge metrics.
Figure 2: Overall performance. (A) Relationship between model size and macro-average performance across all five clinical reasoning benchmarks evaluated (ICU-REACT, SCT-Bench, ER-Reason, MedRBench, and VivaBench). Models are colored by group: Clin-REACT models, open-source general-purpose LLMs, and open-source medical LLMs. (B) Benchmark-level performance of all models on ICU-REACT, SCT-Bench, ER-Reason, MedRBench, and VivaBench. Bars denote mean scores across each benchmark’s metrics, and error bars represent 95% confidence intervals. (C) Metric-level heatmap summarizing performance across the individual evaluation dimensions within each benchmark. Models are ordered by group and performance is displayed as percentage scores.
Figure 3: Clin-REACT performance gains. Each Clin-REACT model is compared against its untrained backbone (8B, 14B, 31B, and 70B) across five benchmarks. A "win" is a pairwise comparison in which a model significantly outperforms a competing model, and a "loss" is one in which it is significantly outperformed; significance was determined using two-sided Wilcoxon signed-rank tests ( p<0.05p<0.05 ). Comparing a backbone’s win/loss counts against its corresponding fine-tuned Clin-REACT model isolates the effect of ICU-REACT training. (A) Aggregate counts of significant wins (bars above the axis) and losses (bars below the axis) summed across all benchmark comparisons, shown for each backbone (gray) and its corresponding Clin-REACT model (teal). The red arrow marks the increase in wins and the blue arrow the decrease in losses attributable to training. Across all four scales, Clin-REACT increases wins and reduces losses. (B) Significant wins broken down by benchmark: backbone wins (left), Clin-REACT wins (right), and the net difference between them (middle). In the middle heatmap, positive values indicate additional wins gained through training and negative values indicate wins lost. (C) Significant losses broken down by benchmark: backbone losses (left), Clin-REACT losses (right), and the net change (middle). In the middle heatmap, negative values indicate fewer losses after training (an improvement in performance) and positive values indicate more losses (a decrease in performance).
Figure 4: Performance across different task types. (A) Task-level scores for information retrieval, diagnosis, and treatment across Clin-REACT models, open-source general-purpose LLMs, and open-source medical LLMs. Task scores were calculated by aggregating relevant benchmark metrics within each task domain. Bars denote mean performance across included metrics, and error bars denote the standard deviation across metrics. (B) Pairwise associations between model performance across information retrieval, diagnosis, and treatment categories. Points represent individual models, colored by model group, and fitted lines show the estimated linear relationships with shaded 95% confidence intervals. Associations were evaluated using Spearman rank correlation tests, with correlation coefficients and corresponding p-values shown in each panel.
Figure 5: Performance across topics in ICU-REACT test set. Heatmaps showing model performance across the nine topic areas in ICU-REACT test set for (A) parent variable F1, (B) variable F1, and (C) reasoning score. Rows denote ICU topics and columns denote evaluated models, ordered from left to right with open-source general-purpose LLMs followed by open-source medical LLMs and Clin-REACT models. Cell values represent percentage scores, with warmer colors indicating higher performance.
Figure 6: Performance across rubric dimensions and clinical data categories. (A) Cleveland dot plots summarizing model performance across the five ICU-REACT rubric dimensions: reasoning correctness, reasoning synthesis, task faithfulness, safety, and critical anchor. (B) Cleveland dot plots showing variable-level F1 performance stratified by the clinical data categories represented in ICU-REACT questions, including physiology, laboratory measurements, medications, diagnoses, interventions, scores and assessments, imaging, and administrative information. Each dot represents the score for a given model on a specific dimension or category. Models are grouped by open-source general-purpose LLMs, open-source medical LLMs, and Clin-REACT models.
Figure 7: Performance correlations between clinical reasoning benchmarks. (A) Associations between models’ ICU-REACT average scores and their average scores on SCT-Bench, ER-Reason, MedRBench, and VivaBench. Points represent individual models and are colored by model group, with solid lines indicating fitted linear trends with shaded 95% confidence intervals. Associations were evaluated using Spearman rank correlation tests, with correlation coefficients and corresponding p-values shown in each panel. (B) Heatmap of pairwise Spearman rank correlations among individual metrics across the five clinical reasoning benchmarks evaluated. Cell values denote correlation coefficients, with positive and negative associations represented by teal and red shading, respectively.
Figure 8: Seed dataset creation overview. (A) A brainstorming agent generates candidate ICU decision-making questions that can be answered with data from an EHR database according to predefined clinical criteria and topic categories. A reasoning agent then identifies the EHR retrieval tasks and initial clinical reasoning needed to answer each question, with variables standardized using the OMOP dictionary. (B) Clinical experts review generated questions for relevance, evaluate the completeness and appropriateness of proposed retrieval tasks, add missing variables when necessary, and assess the validity of the initial reasoning while providing written feedback. (C) Written clinician feedback is incorporated by a refinement agent through distinct pipelines for the training and test seed sets. For ICU-REACT-Train, feedback is first categorized as variable-selection, reasoning improvement, context–question improvement, or unclassified. Feedback concerning variable selection and reasoning improvement is then combined with the original patient context–question pair and initial reasoning to generate both an explanation of required revisions and an improved reasoning response. The resulting training samples contain the context, question, initial reasoning, refinement explanation, and improved reasoning. For ICU-REACT-Test, approved questions, retrieval tasks, and written clinician feedback are directly provided to the refinement agent to generate complete reasoning responses incorporating all required EHR variables.
Figure 9: Dataset augmentation overview. Beginning with the clinician-validated ICU-REACT-Train seed set, the pipeline iteratively sampled ICU topics together with topic-specific scenarios, patient presentations, clinical modifiers, and decision types to guide generation of diverse context–question pairs. At each generation stage, few-shot examples were randomly sampled from the train seed dataset and incorporated into the corresponding prompts to preserve the expert-aligned structure, terminology, and reasoning schema of the seed examples. A GPT-5.2 augmentation agent generated new patient contexts and ICU decision-making questions, after which a GPT-4o-mini agent produced initial reasoning responses. A GPT-5.2 refinement agent then generated a refinement explanation and an improved reasoning response from the initial reasoning, using additional seed-derived few-shot examples to maintain consistency with clinician-validated reasoning patterns. To reduce redundancy, augmented samples were vectorized, grouped through nearest-neighbor search, and compared within groups using ROUGE-L; samples with ROUGE-L scores of at least 0.7 were removed. Finally, a GPT-5.2 difficulty-judging agent categorized retained samples as easy, medium, or hard, and samples were filtered according to difficulty level to construct training sets tailored to each model architecture and scale, producing the final ICU-REACT-Train datasets.
Figure 10: Model Training and Evaluation. (A) Supervised fine-tuning of four open-source backbone models using the ICU-REACT-Train datasets. Low-rank adaptation (LoRA) adapters were trained through reasoning-refinement supervision, with the patient context, decision-making question, and initial response provided as model inputs, and the refinement explanation and improved reasoning response used as target outputs. This process yielded Clin-REACT 8B, 14B, 31B, and 70B models. (B) Evaluation of Clin-REACT models alongside open-source general-purpose and medical LLM baselines. In-domain performance was assessed on the held-out ICU-REACT-Test set using automated variable-retrieval precision, recall, and F1 metrics, together with GPT-5.2 rubric-based reasoning scores. External evaluation included SCT-Bench, which measured agreement with expert response distributions under clinical uncertainty; ER-Reason, which evaluated precision and recall of clinically relevant concepts in acute-care rationales; MedRBench, which assessed examination recommendation, diagnostic, and treatment performance; and VivaBench, which evaluated information-seeking and final diagnostic accuracy in multi-turn clinical cases.
Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: Clinician annotation workflow and interface for the ICU-REACT seed dataset. The panels show screen captures of the web-based annotation tool for a single example being annotated, with arrows indicating the order of the workflow. A clinician team logs in and is presented with the Clinical Context and Question, tagged by relevant clinical domains, and chooses to annotate or skip the item. The annotation then proceeds through sequential sections: (i) Question Validity, confirming whether the question is one a clinician would ask for such a patient; (ii) Required Data Elements, toggling the variables needed to answer the question, organized by category; (iii) Missing Data Elements, indicating whether the retrieved data are complete and, if not, adding further variables by name or concept ID or as custom entries; (iv) Reasoning, judging whether the provided rationale is valid; and (v) Critical Feedback, an open-text field for additional suggestions. Completed annotations are written to a MongoDB store and passed to the research team for use in the construction of the ICU-REACT seed dataset. Note: the example shown is for illustration of the interface only and is not a real nor validated annotated sample.
Figure S2: Topic distribution of the ICU-REACT-Train datasets. t-distributed stochastic neighbor embedding (t-SNE) projections of the ICU-REACT-Train Small (A), Medium (B), and Large (C) datasets, with samples colored according to the primary ICU topic represented in each question. Dataset names (Small, Medium, Large) refer to the parameter scale of the target model each set was used to train. For each dataset scale, the left panel depicts the overall distribution of input-output training examples, whereas the middle and right panels separately visualize examples corresponding to the initial-reasoning and improved-reasoning stages.
Figure S3: ICU-REACT-Test dataset description. (A) Summary of dataset composition, including the number of samples and clinical topics, and the median number of categories, sub-categories, and variables represented per question. (B) t-distributed stochastic neighbor embedding (t-SNE) visualization of questions according to their clinical content, colored by the nine ICU topic areas, alongside the number of samples on each topic. (C) Proportion of test-set questions containing each information category, grouped into physiologic variables, laboratory measurements, medications, scores and assessments, imaging, interventions, diagnoses, and administrative information.
Figure S4: Rubric statistics and item distribution for the LLM-Judge evaluation on the ICU-REACT test set. (A) Summary statistics of the rubric set, comprising 71 samples and 845 rubric items across 5 evaluation dimensions, together with the per-sample median (IQR) of rubric items, maximum score points, and rubric-description length in words. (B) Distribution of rubric items across the five evaluation dimensions (Reasoning correctness, Safety, Reasoning synthesis, Task faithfulness, and Critical anchor), expressed as a percentage of all 845 rubric items with item counts in parentheses. (C) Two-dimensional t-distributed stochastic neighbor embedding (t-SNE) projection at the sample level, in which each marker is one of the 71 samples, sized by its number of rubric items and colored by its ICU-REACT clinical topic; legend counts give the number of samples per topic. (D) Two-dimensional t-SNE projection at the rubric-item level, in which each point is one of the 845 rubric items, colored by its evaluation dimension; legend counts give the number of items per dimension.
HyperparameterClin-REACT 8BClin-REACT 14BClin-REACT 31BClin-REACT 70B
LoRA rank ( rr )16161616
LoRA alpha ( α\alpha )32323232
LoRA dropout0.050.050.050.05
Learning rate1×10−51\times 10^{-5}1×10−51\times 10^{-5}1×10−51\times 10^{-5}1×10−51\times 10^{-5}
Weight decay0.010.010.010.01
Warmup steps100100100100
Appendix
Table S1: Training hyperparameters for each Clin-REACT variant.
Figure S5: Per-benchmark score differences between each Clin-REACT model and every baseline model across five clinical-reasoning benchmarks. Panels correspond to the four Clin-REACT models (top to bottom: Clin-REACT 8B, 14B, 31B, and 70B). Each cell reports the difference in score ( Δ\Delta , percentage points) between the Clin-REACT model and a baseline model on one benchmark (rows), computed as the Clin-REACT model minus the baseline using each benchmark’s average-across-metrics score; green indicates the Clin-REACT model scored higher, orange indicates the baseline scored higher, and color intensity represents the magnitude of the difference. Columns are the 15 open-source baseline models (medical and general purpose), ordered by parameter count; rows are the five clinical-reasoning benchmarks (ICU-REACT, SCT-Bench, ER-Reason, MedRBench, and VivaBench). Numeric labels give each cell’s Δ\Delta , and asterisks denote the statistical significance of the paired difference (two-sided Wilcoxon signed-rank test; *, p<0.05p<0.05 ; **, p<0.01p<0.01 ; ***, p<0.001p<0.001 ). Each panel uses an independent color scale.
Figure S6: Per-benchmark score differences of each Clin-REACT model and its backbone against baseline models across five clinical-reasoning benchmarks. Each panel pairs a backbone (top heatmap) with its fine-tuned Clin-REACT counterpart (bottom heatmap) at a single scale: (A) Llama 3.1 8B Instruct and Clin-REACT 8B, (B) Baichuan M1 14B Instruct and Clin-REACT 14B, (C) Gemma 4 31B and Clin-REACT 31B, and (D) Llama 3.3 70B Instruct and Clin-REACT 70B. Each cell reports the difference in score ( Δ\Delta , percentage points) between the featured model (backbone or Clin-REACT) and a baseline model on one benchmark (rows), computed as the featured model minus the baseline using each benchmark’s average-across-metrics score; green indicates the featured model scored higher, orange indicates the baseline scored higher, and color intensity represents the magnitude of the difference. Columns are the baseline models (all evaluated baselines except the panel’s backbone), ordered by parameter count; rows are the five clinical-reasoning benchmarks (ICU-REACT, SCT-Bench, ER-Reason, MedRBench, and VivaBench), and the top and bottom heatmaps of each panel share the same columns and rows. Asterisks denote the statistical significance of the paired difference (two-sided Wilcoxon signed-rank test; *, p<0.05p<0.05 ; **, p<0.01p<0.01 ; ***, p<0.001p<0.001 ). Each panel uses an independent color scale.
Figure S7: Training task ablation. ICU-REACT average score for models trained on each combination of the five self-supervised augmentation tasks (Variable Selection Reasoning, Reasoning Refinement, Question Generation, Context Generation, and Context/Question Refinement) for (A) Llama-3.1-8B-Instruct and (B) Llama-3.3-70B-Instruct. Filled dots in the lower matrix denote the tasks included in a given configuration, connected vertically when combined; bars are sorted in descending order of average score, and the best-performing configuration in each panel is highlighted in dark teal.
Figure S8: Training task ablation decomposed by evaluation metric. ICU-REACT performance decomposed into parent variable F1 (top row), variable F1 (middle row), and reasoning score (bottom row) across training-task combinations for (A) Llama-3.1-8B-Instruct and (B) Llama-3.3-70B-Instruct. Filled dots in the lower matrix denote the tasks included in each configuration; within each metric row, the best-performing configuration is highlighted in dark teal.
Figure S9: Training dataset size ablation. Average ICU-REACT test score as a function of training dataset size for Llama-3.1-8B-Instruct (light teal) and Llama-3.3-70B-Instruct (dark teal). Each point is a model fine-tuned on the indicated number of examples; the leftmost point of each series (size 0) denotes the corresponding zero-shot instruct baseline. Highest performance for each model are highlighted with boxed scores.
Figure S10: Comparison of Clin-REACT models against proprietary frontier LLMs on clinical reasoning. (A) Clinical-reasoning score across five benchmarks for both model classes, plotted against parameter count (left region, in billions) for Clin-REACT models (circles, n=4n=4 ) and against output cost (right region, per1Mtokens)forfrontierLLMs(diamonds,per 1M tokens) for frontier LLMs (diamonds,n=4),withaverticaldividerseparatingthetwoscales.Eachclinical−reasoningscoreisthemeanacrossfiveclinical−reasoningbenchmarks(ICU−REACT,SCT−Bench,ER−Reason,MedRBench,andVivaBench).(B)Per−benchmarkcomparisonofthefourClin−REACTmodels(teal)againstthefourfrontierLLMs(red)acrossthefiveclinical−reasoningbenchmarks.Barsareorderedbyscorewithineachpanel,andthehighest−scoringClin−REACTmodelperbenchmarkisoutlined.Errorbarsdenote95), with a vertical divider separating the two scales. Each clinical-reasoning score is the mean across five clinical-reasoning benchmarks (ICU-REACT, SCT-Bench, ER-Reason, MedRBench, and VivaBench). (B) Per-benchmark comparison of the four Clin-REACT models (teal) against the four frontier LLMs (red) across the five clinical-reasoning benchmarks. Bars are ordered by score within each panel, and the highest-scoring Clin-REACT model per benchmark is outlined. Error bars denote 95% CIs; brackets indicate pairwise comparisons between the best Clin-REACT model on each benchmark and frontier LLMs (two-sided Wilcoxon signed-rank test; NS, not significant; *** ,p<0.001$ ). (C) Heatmap of component sub-metrics underlying each aggregate benchmark score in (B), grouped by benchmark: ICU-REACT (Parent Variable F1, Variable F1, Reasoning Score), SCT-Bench (SCT Score), ER-Reason (Decision Factors, Differential, Treatment), MedRBench (Assessment Recall, Assessment Precision, Diagnosis Accuracy, Treatment Accuracy), and VivaBench (Overall Key Recall, Overall Key Precision, Final Diagnosis Accuracy). Cell color encodes score (0–100%), with warmer colors indicating higher scores.
Figure S11: Per-clinical content training gains of Clin-REACT models over backbones across clinical-reasoning benchmarks. Gains are shown for (A) Clin-REACT 8B (backbone: Llama 3.1 8B Instruct), (B) Clin-REACT 14B (backbone: Baichuan M1 14B Instruct), (C) Clin-REACT 31B (backbone: Gemma 4 31B), and (D) Clin-REACT 70B (backbone: Llama 3.3 70B Instruct). Each bar reports the change in score ( Δ\Delta , percentage points) for a single clinical category, computed as the Clin-REACT model minus its corresponding backbone using each benchmark’s average-across-metrics score; bar color encodes the sign and magnitude of the change, with teal representing improvement over the backbone, red representing regression, and darker shades denoting larger magnitudes. Within each benchmark section, bars are sorted in descending order of Δ\Delta , so category order varies across panels; numeric labels show each category’s Δ\Delta and axis labels show the number of test samples ( nn ). The three sections per panel, separated by dashed vertical lines, correspond to ICU-REACT topics (7 categories), MedRBench disorder groups (6 categories), and VivaBench specialty groups (8 categories).
Figure S12: Per-clinical content comparison of Clin-REACT and baseline models across clinical-reasoning benchmarks. Each panel ranks all evaluated models by score within a single clinical category, drawn from one of the three benchmarks that carry per-sample categorical labels: ICU-REACT topics (green panels, 7 categories), MedRBench disorder groups (pink panels, 6 categories), and VivaBench specialty groups (blue panels, 8 categories). Panel titles give the category name and its number of test samples ( nn ). Within each panel, points are ordered by score (highest at top, ranked independently per panel) and colored by model class: Clin-REACT (teal, n=4n=4 ), open-source general purpose LLMs (orange, n=9n=9 ), and open-source medical LLMs (blue, n=6n=6 ); numeric labels give each model’s score within the category. Scores are calculated as the average score across metrics within each benchmark.
Figure S13: Per-clinical content, per-metric heatmap of Clin-REACT and baseline models’ performance on MedRBench and VivaBench. Each cell reports a model’s score (%) within a single clinical category (rows) for one evaluation metric (heatmap blocks). Columns comprise all evaluated models, grouped by class and separated by dashed vertical lines: Clin-REACT (left, n=4n=4 ), open-source medical LLMs (center, n=8n=8 ), and open-source general purpose LLMs (right, n=7n=7 ). Row labels give the category name and its number of test samples ( nn ); numeric labels give each cell’s score. Warmer colors indicate higher scores, with each metric block using an independent color scale such shading is comparable within a block. (A) MedRBench disorder groups (6 categories) across four metrics: Assessment Recommendation Precision, Assessment Recommendation Recall, Diagnosis Accuracy, and Treatment Accuracy. (B) VivaBench specialty groups (8 categories) across three metrics: Overall Key Recall, Overall Key Precision, and Final Diagnosis Accuracy.
Figure S14: Per-category information-retrieval gains of Clin-REACT models over backbones on clinical-reasoning benchmarks. Gains are shown for (A) Clin-REACT 8B (backbone: Llama 3.1 8B Instruct), (B) Clin-REACT 14B (backbone: Baichuan M1 14B Instruct), (C) Clin-REACT 31B (backbone: Gemma 4 31B), and (D) Clin-REACT 70B (backbone: Llama 3.3 70B Instruct) across ICU-REACT and VivaBench. Each bar reports the change in information-retrieval F1 ( Δ\Delta , percentage points) for a single content category, computed as the Clin-REACT model’s F1 minus its corresponding backbone’s F1; bar color encodes the sign and magnitude of the change, with teal representing improvement over the backbone, red representing regression, and darker shades denoting larger magnitudes. Categories appear in a fixed order that is consistent across panels, and numeric labels give each category’s Δ\Delta . The two sections per panel, separated by a dashed vertical line, correspond to ICU-REACT variable categories (8 categories) and VivaBench finding categories (4 categories).
Figure S15: Per-category heatmaps of information-retrieval of Clin-REACT and baseline models on ICU-REACT and VivaBench. Information-retrieval is shown as (A) precision and (B) recall. Each cell reports a model’s value (%) for a single content category (rows). Columns comprise all evaluated models, grouped by class and separated by dashed vertical lines: Clin-REACT (left, n=4n=4 ), open-source medical LLMs (center, n=8n=8 ), and open-source general purpose LLMs (right, n=7n=7 ). Rows are grouped and separated by a horizontal line into ICU-REACT variable categories (8 categories) and VivaBench finding categories (4 categories); numeric labels give each cell’s value. Warmer colors indicate higher values, with each panel using an independent color scale such that shading is comparable within a panel.
ModelAverageMMLU-MedMedMCQAMedQAMedXpertQA
Llama 3.1 8B Instruct43.364.047.547.114.8
Clin-REACT 8B34.6 ▼\blacktriangledown ( −- 8.7 )55.2 ▼\blacktriangledown ( −- 8.8 )42.0 ▼\blacktriangledown ( −- 5.5 )27.3 ▼\blacktriangledown ( −- 19.8 )13.8 ▼\blacktriangledown ( −- 1.0 )
Baichuan M1 14B Instruct56.581.462.561.221.0
Clin-REACT 14B55.3 ▼\blacktriangledown ( −- 1.2 )78.8 ▼\blacktriangledown ( −- 2.6 )61.3 ▼\blacktriangledown ( −- 1.2 )59.5 ▼\blacktriangledown ( −- 1.7 )21.7 ▲\blacktriangle ( ++ 0.7 )
Gemma-4-31B-It80.294.278.592.455.8
Clin-REACT 31B78.8 ▼\blacktriangledown ( −- 1.4 )94.9 ▲\blacktriangle ( ++ 0.7 )78.4 ▼\blacktriangledown ( −- 0.1 )89.8 ▼\blacktriangledown ( −- 2.6 )52.2 ▼\blacktriangledown ( −- 3.6 )
Appendix
Table S4: Performance of Clin-REACT models against their backbones on general medical multiple-choice benchmarks.
Figure S16: Relationship between medical multiple-choice and clinical reasoning benchmarks performance. (A) Clinical-reasoning average score versus multiple-choice average score for all evaluated models. Each benchmark score is first computed by averaging its constituent sub-metrics, and each axis is then the mean across benchmarks (four multiple-choice benchmarks: MMLU-Med, MedMCQA, MedQA, MedXpertQA; five clinical-reasoning benchmarks: ER-Reason, ICU-REACT, MedRBench, SCT-Bench, VivaBench). Marker shape denotes model class: general-purpose (triangles, n=7n=7 ), open-source medical (circles, n=8n=8 ), and Clin-REACT (diamonds, n=4n=4 ). The solid line is an ordinary least-squares fit to the baseline models only (general-purpose and medical, n=15n=15 ; shaded band, 95% confidence interval); Clin-REACT models are overlaid but excluded from the fit so that the line describes the multiple-choice-to-reasoning relationship among existing models. Goodness of fit for the baseline relationship is inset ( R2=0.67R^{2}=0.67 , Spearman ρ=0.78\rho=0.78 ). (B) The same models after standardizing each axis to zz -scores across the full cohort ( n=19n=19 ). The dashed diagonal ( y=xy=x ) marks equal relative standing on the two benchmark families: models above the line rank higher on clinical reasoning than on multiple choice, and models below the line rank lower.
Figure S17: Benchmark-level relationships between multiple-choice and clinical-reasoning performance. Pairwise scatter plots for all 20 combinations of the five clinical-reasoning benchmarks (rows) and four multiple-choice benchmarks (columns). Each point is one model: general-purpose (triangles), open-source medical (circles), and Clin-REACT (diamonds). Clinical-reasoning benchmark scores are the average of the benchmark’s sub-metrics. Within each panel, the solid line is an ordinary least-squares fit to the baseline models only (general-purpose and medical, n=15n=15 ), with the corresponding coefficient of determination ( R2R^{2} ) and Spearman rank correlation ( ρ\rho ) inset; Clin-REACT models are overlaid but excluded from the fit and from the reported statistics. Axes are shared within each row and column.
Figure S18: Data-contamination analysis between ICU-REACT training sets and clinical reasoning evaluation benchmarks. (A) Distributions of nearest-neighbor cosine similarity between samples from five clinical reasoning benchmarks (rows: ICU-REACT Test, SCT-Bench, ER-Reason, MedRBench, and VivaBench) and the ICU-REACT Train Small, Medium, and Large datasets (columns). Bars show the number of evaluation samples within each cosine-similarity interval, with color intensity increasing with similarity; vertical dotted lines indicate the median similarity for each comparison, with the median value reported above each distribution. (B) Potential leakage rates for Clin-REACT 8B and 70B and their corresponding Llama 3.1 8B Instruct and Llama 3.3 70B Instruct backbones across the five clinical reasoning benchmarks. Each cell reports the percentage of samples satisfying the potential-leakage criterion, with darker teal indicating a higher rate. (C) Exact 5-token match rates at five prediction positions (tokens 10, 15, 20, 25, and 30) for the same four models across each benchmark. Rows are grouped by benchmark and separated by dashed horizontal lines; each cell reports the percentage of samples with an exact 5-token match at the corresponding prediction position, with darker teal indicating a higher match rate.
   
DomainSubcategoryVariables
DiagnosisCardiacacute coronary syndrome; atrial fibrillation; bradycardia; cardiac arrest; cardiogenic shock; complete heart block; endocarditis; heart failure; myocardial infarction; pericardial effusion; supraventricular tachycardia; ventricular fibrillation; ventricular tachycardia; Aortic dissection; Cardiomyopathy; Myocarditis; Valvular heart disease; DVT; Pulmonary Embolism
DiagnosisPulmonaryrespiratory acidosis; respiratory alkalosis; ventilator-associated pneumonia; ARDS; COPD exacerbation; aspiration pneumonitis; asthma exacerbation; hemothorax; pleural effusion; pneumonia; pneumothorax; pulmonary contusion; pulmonary edema; pulmonary embolism; pulmonary hypertension; respiratory failure; air embolism; fat embolism syndrome; Atelectasis; Pulmonary fibrosis; Bronchiectasis; Lung cancer
DiagnosisNeurologicalhemorrhage; encephalitis; myxedema coma; Guillain-Barré syndrome; brain death; cerebral edema; encephalopathy; epidural hematoma; intracranial hemorrhage; ischemic stroke; myasthenia gravis; status epilepticus; subarachnoid hemorrhage; subdural hematoma; traumatic brain injury; Spinal cord injury; Brain tumor; Hydrocephalus; ICU-acquired weakness; Delirium (Hypoactive)
DiagnosisInfectiousbacteremia; catheter-related bloodstream infection; fungemia; necrotizing fasciitis; anaphylaxis; sepsis; septic shock; COVID-19; Influenza; Cellulitis; Abscess; Ventilator-associated pneumonia; CLABSI; CAUTI
DiagnosisRenalurinary tract infection; adrenal insufficiency; acute kidney injury; acute tubular necrosis; chronic kidney disease; nephrotic syndrome; rhabdomyolysis; urinary retention; Nephrolithiasis; Glomerulonephritis; Hydronephrosis; Pyelonephritis; Hepatorenal Syndrome
Appendix
Table S5: ICU variable taxonomy organized by clinical domain and sub-domain.
   
TopicDimensionItems
Hemodynamic Instability / ShockClinical scenariosSeptic shock from pneumonia; Septic shock from intra-abdominal source; Postoperative hypotension after major surgery; Acute upper GI hemorrhage with volume loss; Traumatic hemorrhagic shock; Cardiogenic shock after acute MI; Acute decompensated heart failure with low output; Acute right-ventricular failure from massive PE; Cardiac tamponade with obstructive physiology; Anaphylactic distributive shock; Neurogenic/spinal shock after trauma; Adrenal/relative cortisol insufficiency in shock; Tension pneumothorax causing obstructive shock; Dialysis/CRRT-associated hypotension in ICU
Hemodynamic Instability / ShockPresentationsSustained low MAP despite initial fluids; Rising lactate and signs of poor perfusion; Cool, mottled extremities and delayed cap refill; Warm, vasodilatory shock with wide pulse pressure; Oliguria or declining urine output; Acute change in mental status from hypoperfusion; Tachycardia with narrow pulse pressure; Escalating vasopressor requirement over hours; Hypotension with new arrhythmia; Need for closer hemodynamic monitoring; Hypotension in setting of limited fluid tolerance (ARDS/CHF)
Hemodynamic Instability / ShockManagement decisionsInitiate norepinephrine as first-line vasopressor; Add a second vasopressor such as vasopressin; Start inotropic support for low cardiac output; Administer another fluid bolus vs hold further fluids; Transfuse packed RBCs for hemorrhagic component; Obtain bedside echocardiography to define etiology; Place arterial and/or central line for monitoring; Initiate stress-dose corticosteroids for refractory shock; Arrange source-control procedure for shock driver; Escalate to mechanical/advanced circulatory support; Begin conservative fluid strategy after stabilization
Hemodynamic Instability / ShockContextual modifiersPatient has significant cardiopulmonary comorbidities; High risk of fluid overload/ARDS so resuscitate cautiously; Limited vascular access or line-related infection concern; Post-cardiac surgery physiology alters pressor choice; Immunocompromised host with atypical infection; Goals-of-care or DNR/DNI status limits escalation
Respiratory FailureClinical scenariosARDS secondary to severe pneumonia; Community-acquired pneumonia with hypoxemic failure; COPD exacerbation with hypercapnic respiratory failure; Obesity hypoventilation/OSA causing ventilatory failure; Postoperative atelectasis with impaired gas exchange; Aspiration event in intubated patient; Ventilator-associated pneumonia in ICU; Neuromuscular weakness (GBS/MG) causing hypoventilation; Cardiogenic pulmonary edema with respiratory compromise; Prolonged mechanical ventilation needing weaning; Post-extubation stridor or upper-airway compromise; Trauma/chest wall injury with poor ventilation; Severe asthma in ICU not responding to usual therapy
Appendix
Table S6: ICU clinical topics and dimensions used to generate ICU-REACT training sets
More in this topic

Medical LLMs

Medical large language models are AI systems adapted to clinical and biomedical data for health reasoning, information extraction, and communication.

See all 402 papers

Explore similar work

CardsList
May 13, 2026

RealICU: Do LLM Agents Understand Long-Context ICU Data? A Benchmark Beyond Behavior Imitation

Intensive care units (ICU) generate long, dense and evolving streams of clinical information, where physicians must repeatedly reassess patient states under time pressure, underscoring a clear need for reliable AI decision support. Existing ICU benchmarks typically treat historical clinician actions as ground truth. However, these actions are made under incomplete information and limited temporal context of the underlying patient state, and may therefore be suboptimal, making it difficult to assess the true reasoning capabilities of AI systems. We introduce RealICU, a hindsight-annotated benchmark for evaluating large language models (LLMs) under realistic ICU conditions, where labels are created after senior physicians review the full patient trajectory. We formulate four physician-motivated tasks: assess Patient Status, Acute Problems, Recommended Actions, and Red Flag actions that risk unsafe outcomes. We partition each trajectory with 30-min windows and release two datasets: RealICU-Gold with 930-window annotations from 94 MIMIC-IV patients, and RealICU-Scale with 11,862 windows extended by Oracle, a physician-validated LLM hindsight labeler. Existing LLMs including memory-augmented ones performed poorly on RealICU, exposing two failure modes: a recall-safety tradeoff for clinical recommendations, and an anchoring bias to early interpretations of the patient. We further introduce ICU-Evo to study structured-memory agents that improves long-horizon reasoning but does not fully eliminate safety failures. Together, RealICU provides a clinically grounded testbed for measuring and improving AI sequential decision-support in high-stakes care. Project page: https://chengzhi-leo.github.io/RealICU-Bench/

Chengzhi Shen, Weixiang Shen, Tobias Susetzky +8

Cherise · Technical University of Munich (TUM) · TUM University Hospital +8

HealthcareClinical Decision-MakingClinical ReasoningClinical Language Model EvaluationMemory-Augmented Language Model Agents
AskPDF
Mar 17, 2026

Medical Reasoning with Large Language Models: A Survey and MR-Bench

Large language models (LLMs) have achieved strong performance on medical exam-style tasks, motivating growing interest in their deployment in real-world clinical settings. However, clinical decision-making is inherently safety-critical, context-dependent, and conducted under evolving evidence. In such situations, reliable LLM performance depends not on factual recall alone, but on robust medical reasoning. In this work, we present a comprehensive review of medical reasoning with LLMs. Grounded in cognitive theories of clinical reasoning, we conceptualize medical reasoning as an iterative process of abduction, deduction, and induction, and organize existing methods into seven major technical routes spanning training-based and training-free approaches. We further conduct a unified cross-benchmark evaluation of representative medical reasoning models under a consistent experimental setting, enabling a more systematic and comparable assessment of the empirical impact of existing methods. To better assess clinically grounded reasoning, we introduce MR-Bench, a benchmark derived from real-world hospital data. Evaluations on MR-Bench expose a pronounced gap between exam-level performance and accuracy on authentic clinical decision tasks. Overall, this survey provides a unified view of existing medical reasoning methods, benchmarks, and evaluation practices, and highlights key gaps between current model performance and the requirements of real-world clinical reasoning.

Xiaohan Ren, Chenxiao Fan, Wenyin Ma +4

School of Cyber Science and Technology, University of Science and Technology of China, Anhui, China · School of Artificial Intelligence and Data Science, University of Science and Technology of China, Anhui, China · Department of Infectious Disease, The First Affiliated Hospital of USTC, Anhui, China +2

Medical ReasoningClinical ReasoningClinical Language Model EvaluationClinical Decision-Making
AskPDF
Aug 26, 2026

Didactic knowledge or Clinical Cases? How Data Types Shape Medical Large Language Models

Medical large language models are commonly trained on mixtures of didactic data (e.g., textbooks) and clinical data (e.g., patient records), yet how these data types differentially shape model capabilities remains unclear. We address this issue with token-matched experiments that vary the didactic-to-clinical ratio and analyze how data composition affects performance, capability profiles, and error patterns across knowledge-intensive and clinic-oriented tasks. We uncover an asymmetric transfer across task types: clinical data improves clinic-oriented tasks while remaining competitive on knowledge-intensive ones, whereas didactic data mainly improves knowledge-intensive tasks. Error analysis suggests a knowing-doing gap, where improvements in knowledge recall do not reliably generalize to clinical reasoning. We further observe that modest amounts of clinical data yield most of the gains on EHR-grounded tasks, while the optimal mixture ratio varies with the knowledge and clinical reasoning demands of downstream tasks. These findings suggest that medical LLM data curation should be application-driven, with higher proportions of clinical data preferred for reasoning-intensive use cases.

Yuzheng Fan, Haochun Wang, Sendong Zhao +3

Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, Harbin 150001, China

Medical LLMsLLM TrainingTraining Data CurationClinical Reasoning
AskPDF