Large language models (LLMs) can reason over both structured measurements and free-text records, which makes them attractive for emergency department (ED) prediction. However, most existing LLMs still fall short of conventional ML methods like trees on EHR-based prediction tasks. Our controlled analysis traces this gap to two weaknesses: LLMs do not select informative features, and they misjudge what numerical values imply for risk even when given the right features. We propose TIR-EC, a tool-integrated reinforcement learning framework that addresses both. TIR-EC exposes EHR feature groups as tools the LLM can call. It trains the LLM with rewards from a gradient-boosted tree teacher, which provides a reference set of informative tools and a graded risk target. TIR-EC also distills task-specific clinical skills into the model weights through a reverse curriculum that gradually removes the skills from the prompt during RL process. Evaluation on three clinical prediction tasks shows that our 4B agent achieves the highest macro F1 of all compared models, outperforms conventional ML baselines, and is competitive with much larger frontier LLMs. Further analysis shows that TIR-EC selects more useful evidence with fewer tool calls and combines measurements more effectively.
Figures & tables
Figure 1: Preliminary analysis on 2000 MC-MED patients ( Kansal et al., 2025 ) . (a) Disposition AUROC given k of 12 numerical feature groups, averaged over 10 runs. (b) 46 features’ dataset level vs. model-stated change in risk. (c) One decompensation patient. (d) Error decomposition on 100 randomly sampled patients for whom the GBM predicts correctly but Qwen3-4B free selection fails, human-labeled.
Figure 2: Overview of TIR-EC. The agent retrieves patient evidence through clinical tools and predicts an outcome. Tree models provide tool targets and risk signals, while clinical skill guidance fades during training. Alternative trajectories for the same patient share retained guidance and receive rewards for agentic RL updates.
Disposition
Decompensation
Revisit
Macro avg.
Model
P
R
F1
AUPRC
AUROC
P
R
F1
AUPRC
AUROC
P
R
F1
AUPRC
AUROC
F1
AUPRC
AUROC
ML & DL Baselines
XGBoost
62.3
84.3
71.6
75.9
80.2
27.9
41.7
33.5
26.9
65.2
30.3
60.6
40.4
26.6
60.9
48.5
43.1
68.8
BEHRT
70.2
76.7
73.3
82.3
82.8
21.0
61.1
31.2
28.8
65.8
24.3
55.1
33.8
23.7
57.5
46.1
44.9
68.7
MC-BEC
70.5
77.5
73.8
82.6
83.0
20.4
55.6
29.9
33.0
64.0
22.3
51.0
31.1
27.8
59.3
44.9
47.8
68.8
TransformerEHR
66.1
68.3
67.2
68.1
72.2
24.3
25.0
24.7
19.6
53.6
21.4
63.3
32.0
21.2
52.6
41.3
36.3
59.5
Table 1: Results on three emergency-care prediction tasks. All LLMs are evaluated in the same agentic feature selection setup as TIR-EC, except Qwen’s "Full Features" rows intakes all features for comparison.
Figure 3: Selective evidence acquisition. (a) Disposition AUROC when the agent may request up to k feature groups (solid) or is given k random groups (dotted). (b) Macro AUROC versus mean tool calls per patient.
Figure 4: Feature interpretation helps selection. Each arrow is one feature group, from Qwen3-4B (tail) to TIR-EC (head). Only commonly selected features are shown. Lines: least-squares fit across groups.
Figure 5: Feature interpretation . Every model receives the same k randomly drawn feature groups for each patient. Lines show macro AUROC over the three tasks, averaged over 10 random draws at each k .
Figure 6: Barcode plot of tool call (top features for GBM) patterns for (Left) TIR-EC, (Middle) GBM, (Right) Qwen-4B base.
Case 1: Risk assessment beyond isolated abnormalities
Patient
Woman aged 69; ankle fracture. Predict admission at 2 h. Stable vital signs; Hb 11.4 g/dL (low); diagnostic testing and pain relief.
GPT Sol
“The fracture and low hemoglobin could raise concern in an older patient. Weighting these findings above stable vital signs and supportive treatment could favor admission.” <answer>0.85</answer> ×
Our Model
“While the patient has a fracture, the lab values are mostly normal […] The orders are focused on diagnostic and supportive care rather than admission-level interventions.” <answer>0.29</answer> ✓
Outcome
Discharged home.
Case 2: More efficient retrieval of task-relevant evidence
Patient
Man aged 46; motor vehicle crash. Stable HR, oxygenation, and MAP; predict deterioration in 60 min.
Table 2: Case study of TIR-EC versus GPT.
Figure 7: General prompt template and interaction protocol for TIR rollout. The figure summarizes the decision process rather than reproducing the runtime prompt verbatim. Blue denotes model reasoning, green a tool call, brown an environment observation, and purple the final prediction.
Predicting future clinical outcomes from electronic health records (EHR) remains challenging due to the complexity and heterogeneity of patient data. LLMs have shown strong potential for such predictive tasks, yet existing approaches mainly focus on enhancing medical knowledge through distillation or RAG while relying on the model's internal ability to interpret contextual information. In this work, we present ReMedi (Reasoner for Medical Clinical Prediction), a framework for improving clinical outcome prediction from EHR. ReMedi generates rationale-answer pairs using a challenging sample regeneration mechanism for complex clinical questions, which leverages ground-truth answers as hints to enhance reasoning for further fine-tuning and preference tuning. ReMedi integrates ground-truth outcome guidance into the preference data construction loop, regenerating rationale-answer variants. By tuning on these rationale-answer pairs, the model improves its predictive performance. Experiments on multiple EHR prediction tasks demonstrate substantial gains of up to 19.9 percent over state-of-the-art baselines in terms of F1 score, underscoring ReMedi's effectiveness in real-world clinical prediction.
Yushi Cao, Yiming Chen, Hongchao Jiang +2
ASUS Intelligent Cloud Services (AICS), Singapore · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), Taiwan · National University of Singapore, Singapore
Large language models (LLMs) exhibit strong natural-language reasoning abilities for clinical decision support, but struggle to effectively model structured longitudinal electronic health records (EHRs). In contrast, EHR foundation models can learn predictive patient representations, yet lack interpretable language-based reasoning. To bridge this gap, we propose ChatHealthAI, a multimodal reasoning framework that aligns structured EHR representations from a pretrained EHR foundation model with the semantic space of a frozen LLM through a task-aware resampler. By integrating longitudinal patient representations with refined clinical event descriptions, ChatHealthAI enables clinically grounded natural-language reasoning while maintaining accurate patient prediction. We evaluated ChatHealthAI on three clinical predictive tasks from the EHRSHOT benchmark. Results show that ChatHealthAI improves reasoning quality and interpretability while preserving competitive predictive performance. These findings highlight the potential of integrating EHR foundation models with pretrained LLMs for interpretable clinical prediction.
Bo-Hong Wang, Baicheng Peng, Ruilin Wang +3
School of Computer Science, McGill University, Montreal, QC, Canada · Mila - Quebec AI Institute, Montreal, QC, Canada
Clinical early warning systems built on irregularly sampled medical time series (ISMTS) from electronic health records must deliver continuous risk scores for patient triage as well as interpretable rationales that clinicians can verify. Large language models (LLMs) are uniquely positioned for both, deriving risk from their output probabilities and rationales from their medical knowledge. However, we find that conventional LLM reasoning collapses graded risk into overconfident predictions and thereby undermines the cross-patient comparability on which triage depends. We refer to this failure mode as risk polarization and identify two underlying behaviors: early commitment to a single outcome, and one-sided reasoning that focuses only on the evidence for that outcome. To address this, we propose TRIAGE, a framework that trains an LLM to reason dialectically over competing clinical outcomes by eliciting outcome-specific rationales. This dialectical formulation mitigates risk polarization, enabling a single LLM to jointly provide explicit clinical rationales and risk scores comparable across patients. Across five ISMTS benchmarks, TRIAGE improves mean AUPRC by 17.0% and reduces mean calibration error by 82.8% relative to the competitive LLM-based baseline, while surpassing the strongest ISMTS baseline by 3.5% in mean AUPRC.