Recent advances in multimodal foundation models have demonstrated strong performance across diverse clinical tasks. However, existing approaches predominantly operate at the individual sample level and fail to model the interplay between population-level cohort patterns and patient-specific variation that underlie real-world clinical reasoning. In practice, clinicians routinely reason through patient cohorts sharing similar demographic, physiological, or pathological characteristics, while accounting for patient-specific deviations from the group prototype. Motivated by this gap, we propose CODA, a cohort- and drift-aware foundation model for multimodal clinical reasoning. CODA explicitly integrates cohort structure into the modeling pipeline through two complementary mechanisms: cohort-aware population stratification and drift-aware individual adaptation. In particular, we leverage reinforcement learning to discover clinically meaningful patient cohorts, and further model each patient's deviation from cohort prototypes as a structured drift signal encoding fine-grained individual heterogeneity. These cohort- and drift-aware representations are incorporated into the foundation model to enable reasoning that is simultaneously informed by population-level patterns and sensitive to individual-level variation. Extensive experiments on real-world electronic health record (EHR) benchmarks demonstrate that CODA achieves consistent effectiveness and strong generalizability across diverse clinical tasks, including closed-QA, open-QA, and report generation.
Figures & tables
Figure 1: Rethinking Clinical Reasoning: From instance-level predictions to cohort-aware modeling.
Figure 2: Overview of CODA for cohort- and drift-aware multimodal clinical reasoning.
Domain
Model
ID
OOD
Avg.
Admissions
IE
POE
Services
Transfers
Obesity
ACC
MCC
ACC
MCC
ACC
MCC
ACC
MCC
ACC
MCC
ACC
MCC
ACC
MCC
General Domain
LLaVA 1.5-7B
40.00
20.06
19.44
-7.48
57.00
42.82
63.59
51.45
60.60
47.61
61.90
47.97
50.42
33.74
LLaMA 3-11B
32.80
10.74
26.52
4.85
35.40
14.55
43.64
25.32
37.00
16.29
54.12
39.58
38.25
18.56
InternVL 3-8B
28.20
5.06
18.94
-6.52
46.80
29.13
42.39
23.35
27.00
2.58
70.92
59.61
39.04
18.87
Qwen 3-VL-8B
36.40
15.35
38.38
17.64
40.40
20.49
60.60
48.07
53.80
38.63
69.67
58.10
49.88
33.05
Table 1: Comparison of CODA with baselines under Closed-QA settings.
Domain
Model
ID
OOD
Avg.
Admissions
IE
POE
Services
MIMIC-CXR
Transfers
Obesity
PadChest
Open-QA
Report
F1-Bio
BLEU-1
F1-Bio
BLEU-1
F1-Bio
BLEU-1
F1-Bio
BLEU-1
F1-Rad
F1-Bio
F1-Bio
BLEU-1
F1-Bio
BLEU-1
F1-Rad
F1-Bio
F1-Bio
BLEU-1
F1-Rad
F1-Bio
General Domain
LLaVA 1.5-7B
75.95
36.91
75.16
29.21
75.66
33.51
75.08
34.12
1.79
68.12
75.79
41.27
72.23
9.99
11.75
68.78
74.98
30.83
6.77
68.45
LLaMA 3-11B
74.68
35.65
72.73
24.38
74.83
36.18
74.97
34.34
4.38
65.01
74.88
38.01
70.51
8.68
7.54
62.28
73.77
29.54
5.96
63.64
InternVL 3-8B
72.20
24.43
73.91
25.01
72.61
23.90
72.39
23.83
6.38
61.62
73.96
26.16
67.90
5.34
9.32
61.45
72.16
21.45
7.85
61.54
Qwen 3-VL-8B
68.82
18.31
68.77
17.14
68.47
16.82
68.07
17.09
8.23
63.63
72.23
26.96
68.95
6.40
8.15
66.42
69.22
17.12
8.19
65.03
Table 2: Comparison of CODA with baselines under Open-QA and report generation settings.
Model
ID
OOD
Closed-QA
Open-QA
Report
Closed-QA
Open-QA
Report
ACC
MCC
F1-Bio
BLEU-1
F1-Rad
F1-Bio
ACC
MCC
F1-Bio
BLEU-1
F1-Rad
F1-Bio
Vanilla Backbone
43.95
25.39
68.53
17.34
8.23
63.63
61.74
48.36
70.59
16.68
8.15
66.42
w/o Cohort
85.58
80.69
80.11
49.18
15.73
78.58
72.56
62.19
76.82
28.83
24.60
72.51
w/o Deviation
85.56
80.73
80.10
48.86
16.33
78.94
73.05
62.95
76.65
28.82
25.19
72.41
CODA
86.34
81.78
81.06
51.32
17.49
79.03
73.76
64.08
77.47
31.03
25.52
73.93
Table 3: Ablation study results of CODA. "w/o Cohort" removes the Cohort Context Encoding, while "w/o Deviation" removes Drift-aware Deviation Modeling.
Figure 3: Comparisons with fine-tuned baselines.
Figure 4: Visualization of embeddings within different cohorts.
Model
Tokens
Latency
Token Throughput
Peak GPU Mem.
FLOPs
(tokens/sample)
(s/sample)
(tokens/s)
(GB)
(G/token)
LLaVA 1.5-7B
187.31
8.62
21.73
13.27
13.60
LLaVA-Med v1.5-7B
69.56
3.21
21.67
14.21
14.60
Hulu-Med-7B
72.75
5.79
28.95
15.65
14.40
Lingshu-7B
167.62
5.79
25.18
15.65
14.40
InternVL 3-8B
417.55
19.53
21.38
15.00
14.80
Table 4: Efficiency comparison of CODA against different baselines.
Figure 5: Expert review results on the first preference.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Notation
Description
x
Input instance, consisting of clinical records, a question and an image.
y
The ground truth output sequence.
xi
The i-th clinical records in x .
q
The question in x .
I
Input image of height H, width W, and three RGB channels.
X
The textual input, including clinical records and a question.
Appendix
Table 5: Notations.
Dataset
Task Description
Admissions
Predict the next admission type from EHR records (Closed-QA) and provide clinical reasoning (Open-QA).
Services
Predict the next clinical department from EHR records (Closed-QA) and provide clinical reasoning (Open-QA).
Ingredient Events (IE)
Predict the next administered medication or substance from EHR records (Closed-QA) and provide clinical reasoning (Open-QA).
Provider Order Entry (POE)
Predict the next clinical order type from EHR records (Closed-QA) and provide clinical reasoning (Open-QA).
Transfers
Predict the next care location from EHR records (Closed-QA) and provide clinical reasoning (Open-QA).
Obesity
Assess obesity risk from self-reported EHR records (Closed-QA) and provide supporting analysis (Open-QA).
Appendix
Table 6: Task descriptions across datasets.
Configuration
Stage (i)
Stage (ii)
Stage (iii)
Optimizer
AdamW
AdamW
AdamW
Adapter LR
/
5e-5
1e-4
Learning Rate
2e-5
1e-4
2e-5
Weight Decay
/
0.01
0.01
Dropout Rate
0.05
0.1
0.1
LR Scheduler
Warmup
Constant
Cosine w/ Warmup
Appendix
Table 7: Summary of configurations for the training process.
Foundation models have achieved remarkable performance across medical question answering, imaging, and electronic health record (EHR) tasks, yet reliable clinical deployment remains challenging due to limited interpretability, vulnerability to distribution shift, and weak alignment with clinician reasoning. We argue that these limitations arise because existing approaches prioritize representation learning while treating patient comparison as an emergent property rather than a primary source of clinical evidence. To address this gap, we propose CAFM, a Cohort-Anchored Foundation Model framework that elevates patient cohorts to a first-class object throughout the learning pipeline. The framework consists of four stages: deviation-aware data curation, cohort-conditioned pretraining, multimodal cohort alignment, and clinician-in-the-loop refinement. Together, these stages improve data quality, organize representations around clinically meaningful cohort structure, preserve modality-specific relationships, and support auditable clinical decision-making. The framework is compositional and can augment existing EHR foundation models without modifying their underlying encoders. We illustrate CAFM through four clinical case studies spanning acute kidney injury prediction, cardiovascular risk stratification from electrocardiograms, optic neuropathy triage from orbital imaging, and electroretinogram-grounded report generation. We further present five empirically testable hypotheses and identify open challenges in data quality, irregular temporality, multimodal learning, distribution shift, and evaluation beyond predictive accuracy. We argue that explicitly anchoring foundation models to patient cohorts provides a principled path toward trustworthy clinical AI.
Healthcare foundation models have largely followed paradigms from natural language processing and computer vision, emphasizing large scale pretraining and deterministic representations over heterogeneous clinical data. However, clinical observations are inherently incomplete, reflecting sparse, irregular, and modality dependent measurements of an underlying physiologic state. In this work, we propose a framework for uncertainty aware foundation modeling that represents each patient not as a point embedding, but as a distribution over plausible latent states. By learning set valued representations and enforcing consistency across partial views of the same patient, the model captures what is invariantly inferable while explicitly encoding epistemic uncertainty. We integrate this formulation with multimodal encoders and scalable self supervised objectives, combining reconstruction, contrastive alignment, and distributional regularization. Across diverse clinical tasks, our approach improves predictive performance, robustness under missing data, and uncertainty calibration relative to strong baselines. These results suggest that modeling what is not observed rather than only what is constitutes a critical inductive bias for healthcare foundation models.
Qian Zhou, Yuanyun Zhang, Shi Li
University of the Chinese Academy of Sciences · Columbia University
Electronic health record (EHR) foundation models trained on longitudinal patient trajectories have demonstrated strong performance across diverse clinical prediction tasks. However, their clinical reasoning capabilities remain constrained by next-token prediction on limited and incomplete EHR data. To address this, we propose a reinforcement learning (RL) fine-tuning framework that treats EHR foundation models as generative policies over patient trajectories. We formulate common clinical prediction problems (e.g., hospital readmission) as event-conditioned, time-windowed reasoning tasks. We then design time-aware, rollout-sensitive rewards to account for finite rollout lengths and temporally inconclusive outcomes. We find that RL fine-tuning consistently improves over pre-trained backbones and strong baselines. Notably, it enables smaller models to surpass larger pre-trained models in data-limited regimes and induces positive transfer across tasks. Further analysis shows that RL fine-tuned models generate trajectories with stronger structural and semantic alignment to ground truth and greater downstream utility.