CODA: Cohort- and Drift-aware Foundation Model for Multimodal Clinical Reasoning

Changshuo Liu; Jiaqi Zhu; Wenqiao Zhang; Xiaokui Xiao; Beng Chin Ooi

cs.AIOct 19, 2026

CODA: Cohort- and Drift-aware Foundation Model for Multimodal Clinical Reasoning

Authors: Changshuo Liu, Jiaqi Zhu, Wenqiao Zhang, Xiaokui Xiao, Beng Chin Ooi

Organizations: School of Computing National University of Singapore · School of Software Technology Zhejiang University

Abstract

Recent advances in multimodal foundation models have demonstrated strong performance across diverse clinical tasks. However, existing approaches predominantly operate at the individual sample level and fail to model the interplay between population-level cohort patterns and patient-specific variation that underlie real-world clinical reasoning. In practice, clinicians routinely reason through patient cohorts sharing similar demographic, physiological, or pathological characteristics, while accounting for patient-specific deviations from the group prototype. Motivated by this gap, we propose CODA, a cohort- and drift-aware foundation model for multimodal clinical reasoning. CODA explicitly integrates cohort structure into the modeling pipeline through two complementary mechanisms: cohort-aware population stratification and drift-aware individual adaptation. In particular, we leverage reinforcement learning to discover clinically meaningful patient cohorts, and further model each patient's deviation from cohort prototypes as a structured drift signal encoding fine-grained individual heterogeneity. These cohort- and drift-aware representations are incorporated into the foundation model to enable reasoning that is simultaneously informed by population-level patterns and sensitive to individual-level variation. Extensive experiments on real-world electronic health record (EHR) benchmarks demonstrate that CODA achieves consistent effectiveness and strong generalizability across diverse clinical tasks, including closed-QA, open-QA, and report generation.

Figures & tables

Figure 1: Rethinking Clinical Reasoning: From instance-level predictions to cohort-aware modeling.

Figure 2: Overview of CODA for cohort- and drift-aware multimodal clinical reasoning.

Domain	Model	ID								OOD				Avg.
		Admissions		IE		POE		Services		Transfers		Obesity		Avg.
		ACC	MCC	ACC	MCC	ACC	MCC	ACC	MCC	ACC	MCC	ACC	MCC	ACC	MCC
General Domain	LLaVA 1.5-7B	40.00	20.06	19.44	-7.48	57.00	42.82	63.59	51.45	60.60	47.61	61.90	47.97	50.42	33.74
	LLaMA 3-11B	32.80	10.74	26.52	4.85	35.40	14.55	43.64	25.32	37.00	16.29	54.12	39.58	38.25	18.56
	InternVL 3-8B	28.20	5.06	18.94	-6.52	46.80	29.13	42.39	23.35	27.00	2.58	70.92	59.61	39.04	18.87
	Qwen 3-VL-8B	36.40	15.35	38.38	17.64	40.40	20.49	60.60	48.07	53.80	38.63	69.67	58.10	49.88	33.05

Table 1: Comparison of CODA with baselines under Closed-QA settings.

Domain	Model	ID										OOD						Avg.
		Admissions		IE		POE		Services		MIMIC-CXR		Transfers		Obesity		PadChest		Open-QA		Report
		F1-Bio	BLEU-1	F1-Bio	BLEU-1	F1-Bio	BLEU-1	F1-Bio	BLEU-1	F1-Rad	F1-Bio	F1-Bio	BLEU-1	F1-Bio	BLEU-1	F1-Rad	F1-Bio	F1-Bio	BLEU-1	F1-Rad	F1-Bio
General Domain	LLaVA 1.5-7B	75.95	36.91	75.16	29.21	75.66	33.51	75.08	34.12	1.79	68.12	75.79	41.27	72.23	9.99	11.75	68.78	74.98	30.83	6.77	68.45
	LLaMA 3-11B	74.68	35.65	72.73	24.38	74.83	36.18	74.97	34.34	4.38	65.01	74.88	38.01	70.51	8.68	7.54	62.28	73.77	29.54	5.96	63.64
	InternVL 3-8B	72.20	24.43	73.91	25.01	72.61	23.90	72.39	23.83	6.38	61.62	73.96	26.16	67.90	5.34	9.32	61.45	72.16	21.45	7.85	61.54
	Qwen 3-VL-8B	68.82	18.31	68.77	17.14	68.47	16.82	68.07	17.09	8.23	63.63	72.23	26.96	68.95	6.40	8.15	66.42	69.22	17.12	8.19	65.03

Table 2: Comparison of CODA with baselines under Open-QA and report generation settings.

Model	ID						OOD
	Closed-QA		Open-QA		Report		Closed-QA		Open-QA		Report
	ACC	MCC	F1-Bio	BLEU-1	F1-Rad	F1-Bio	ACC	MCC	F1-Bio	BLEU-1	F1-Rad	F1-Bio
Vanilla Backbone	43.95	25.39	68.53	17.34	8.23	63.63	61.74	48.36	70.59	16.68	8.15	66.42
w/o Cohort	85.58	80.69	80.11	49.18	15.73	78.58	72.56	62.19	76.82	28.83	24.60	72.51
w/o Deviation	85.56	80.73	80.10	48.86	16.33	78.94	73.05	62.95	76.65	28.82	25.19	72.41
CODA	86.34	81.78	81.06	51.32	17.49	79.03	73.76	64.08	77.47	31.03	25.52	73.93

Table 3: Ablation study results of CODA. "w/o Cohort" removes the Cohort Context Encoding, while "w/o Deviation" removes Drift-aware Deviation Modeling.

Panel of Figure 3 — Figure 3: Comparisons with fine-tuned baselines.

Figure 4: Visualization of embeddings within different cohorts.

Model	Tokens	Latency	Token Throughput	Peak GPU Mem.	FLOPs
	(tokens/sample)	(s/sample)	(tokens/s)	(GB)	(G/token)
LLaVA 1.5-7B	187.31	8.62	21.73	13.27	13.60
LLaVA-Med v1.5-7B	69.56	3.21	21.67	14.21	14.60
Hulu-Med-7B	72.75	5.79	28.95	15.65	14.40
Lingshu-7B	167.62	5.79	25.18	15.65	14.40
InternVL 3-8B	417.55	19.53	21.38	15.00	14.80

Table 4: Efficiency comparison of CODA against different baselines.

Panel of Figure 5 — Figure 5: Expert review results on the first preference.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Description
$\mathbf{x}$	Input instance, consisting of clinical records, a question and an image.
$\mathbf{y}$	The ground truth output sequence.
$x_{i}$	The i-th clinical records in $\mathbf{x}$ .
$\mathbf{q}$	The question in $\mathbf{x}$ .
$\mathcal{I}$	Input image of height H, width W, and three RGB channels.
$\mathcal{X}$	The textual input, including clinical records and a question.

Appendix

Table 5: Notations.

Dataset	Task Description
Admissions	Predict the next admission type from EHR records (Closed-QA) and provide clinical reasoning (Open-QA).
Services	Predict the next clinical department from EHR records (Closed-QA) and provide clinical reasoning (Open-QA).
Ingredient Events (IE)	Predict the next administered medication or substance from EHR records (Closed-QA) and provide clinical reasoning (Open-QA).
Provider Order Entry (POE)	Predict the next clinical order type from EHR records (Closed-QA) and provide clinical reasoning (Open-QA).
Transfers	Predict the next care location from EHR records (Closed-QA) and provide clinical reasoning (Open-QA).
Obesity	Assess obesity risk from self-reported EHR records (Closed-QA) and provide supporting analysis (Open-QA).

Appendix

Table 6: Task descriptions across datasets.

Configuration	Stage (i)	Stage (ii)	Stage (iii)
Optimizer	AdamW	AdamW	AdamW
Adapter LR	/	5e-5	1e-4
Learning Rate	2e-5	1e-4	2e-5
Weight Decay	/	0.01	0.01
Dropout Rate	0.05	0.1	0.1
LR Scheduler	Warmup	Constant	Cosine w/ Warmup

Appendix

Table 7: Summary of configurations for the training process.

Module	Parameter	Value / Shape
Tokenizer	Vocabulary size	151,670
Backbone (Qwen3-VL)	Hidden size	4096
	# Transformer layers	36
	Embedding matrix	(151670, 4096)
	Self-attn projections	q: (4096, 4096), k/v: (4096, 1024), o: (4096, 4096)
	MLP	gate/up: (4096, 12288), down: (12288, 4096)

Appendix

Table 8: Architectural parameters of CODA.

Figure 6: Illustration of Cohort Structure.

Model	ID								OOD				Avg.
	Admissions		IE		POE		Services		Transfers		Obesity		Avg.
	ACC	MCC	ACC	MCC	ACC	MCC	ACC	MCC	ACC	MCC	ACC	MCC	ACC	MCC
LoRA Only	74.60	65.78	91.92	89.04	81.20	74.83	92.02	89.42	73.60	64.65	70.30	58.48	80.61	73.70
w/o Cohort	75.20	66.57	92.17	89.57	82.20	76.25	92.77	90.38	74.20	65.16	70.92	59.21	81.24	74.52
w/o Deviation	75.80	67.37	92.17	89.51	82.00	76.00	92.27	90.06	74.40	65.70	71.70	60.20	81.39	74.81
CODA	76.60	68.77	92.42	89.93	82.80	77.05	93.52	91.39	75.20	66.89	72.32	61.28	82.14	75.88

Appendix

Table 9: Ablation Study of CODA under Closed-QA settings.

Model	ID										OOD						Avg.
	Admissions		IE		POE		Services		MIMIC-CXR		Transfers		Obesity		PadChest		Open-QA		Report
	F1-Bio	BLEU-1	F1-Bio	BLEU-1	F1-Bio	BLEU-1	F1-Bio	BLEU-1	F1-Rad	F1-Bio	F1-Bio	BLEU-1	F1-Bio	BLEU-1	F1-Rad	F1-Bio	F1-Bio	BLEU-1	F1-Rad	F1-Bio
LoRA Only	79.06	46.09	80.94	47.20	80.16	50.27	78.87	46.24	15.46	78.74	78.61	44.18	74.61	13.44	24.68	72.24	78.71	41.24	20.07	75.49
w/o Cohort	79.71	48.02	81.10	49.08	80.44	52.03	79.20	47.60	15.73	78.58	78.58	43.66	75.07	14.01	24.60	72.51	79.02	42.40	20.17	75.55
w/o Deviation	79.80	48.64	81.18	48.99	80.06	50.50	79.34	47.32	16.33	78.94	78.55	43.68	74.76	13.96	25.19	72.41	78.95	42.18	20.76	75.67
CODA	80.82	52.45	82.21	51.69	81.02	51.97	80.21	49.19	17.49	79.03	79.57	46.42	75.37	15.65	25.52	73.93	79.87	44.56	21.51	76.48

Appendix

Table 10: Ablation Study of CODA under Open-QA and report generation settings.

Model	ID								OOD				Avg.
	Admissions		IE		POE		Services		Transfers		Obesity		Avg.
	ACC	MCC	ACC	MCC	ACC	MCC	ACC	MCC	ACC	MCC	ACC	MCC	ACC	MCC
InternVL 3-8B	76.00	67.98	91.18	88.51	81.20	74.93	93.27	91.02	74.60	65.75	71.23	59.52	81.25	74.62
Lingshu-7B	76.20	68.24	91.67	88.85	81.20	74.96	93.27	91.07	74.40	65.52	71.38	60.24	81.35	74.81
CODA-Qwen3-8B	76.60	68.77	92.42	89.93	82.80	77.05	93.52	91.39	75.20	66.89	72.32	61.28	82.14	75.88

Appendix

Table 11: Comparison of CODA with LoRA fine-tuned baselines under Closed-QA settings.

Model	ID										OOD						Avg.
	Admissions		IE		POE		Services		MIMIC-CXR		Transfers		Obesity		PadChest		Open-QA		Report
	F1-Bio	BLEU-1	F1-Bio	BLEU-1	F1-Bio	BLEU-1	F1-Bio	BLEU-1	F1-Rad	F1-Bio	F1-Bio	BLEU-1	F1-Bio	BLEU-1	F1-Rad	F1-Bio	F1-Bio	BLEU-1	F1-Rad	F1-Bio
InternVL 3-8B	80.73	52.10	81.54	50.22	80.91	51.51	80.18	49.39	16.59	78.69	79.14	46.00	72.17	11.21	23.85	72.05	79.11	43.74	20.22	75.37
Lingshu-7B	80.60	48.92	82.16	50.76	80.99	51.65	80.01	49.03	16.89	78.88	78.82	46.64	73.72	12.07	22.58	71.90	79.38	43.44	19.74	75.39
CODA	80.82	52.45	82.21	51.69	81.02	51.97	80.21	49.19	17.49	79.03	79.57	46.42	75.37	15.65	25.52	73.93	79.87	44.56	21.51	76.48

Appendix

Table 12: Comparison of CODA with LoRA fine-tuned baselines under Open-QA and report generation settings.

Model	ID								OOD				Avg.
	Admissions		IE		POE		Services		Transfers		Obesity		Avg.
	ACC	MCC	ACC	MCC	ACC	MCC	ACC	MCC	ACC	MCC	ACC	MCC	ACC	MCC
LLaVA 1.5-7B 1-shot	35.60	17.73	20.20	0.94	41.80	26.70	55.36	41.83	54.60	39.33	58.79	44.27	44.39	28.46
LLaVA 1.5-7B 2-shot	10.60	5.68	8.59	5.72	14.20	11.59	18.20	16.18	46.20	34.57	49.61	35.99	24.57	18.29
InternVL 3-8B 1-shot	38.00	17.64	29.04	5.55	45.40	27.36	55.36	40.53	38.20	17.96	53.65	37.40	43.28	24.41
InternVL 3-8B 2-shot	42.60	23.74	30.30	6.82	43.40	24.72	56.86	42.46	42.60	23.92	53.50	36.94	44.88	26.43

Appendix

Table 13: Comparison of CODA with few-shot baselines under Closed-QA settings.

Model	In-distribution Task										Out-of-distribution Task						Avg.
	Admissions		IE		POE		Services		MIMIC-CXR		Transfers		Obesity		PadChest		Open-QA		Report
	F1-Bio	BLEU-1	F1-Bio	BLEU-1	F1-Bio	BLEU-1	F1-Bio	BLEU-1	F1-Rad	F1-Bio	F1-Bio	BLEU-1	F1-Bio	BLEU-1	F1-Rad	F1-Bio	F1-Bio	BLEU-1	F1-Rad	F1-Bio
LLaVA 1.5-7B 1-shot	72.42	23.91	68.45	17.87	71.84	22.52	72.93	22.99	4.17	68.19	76.07	28.96	73.11	11.48	9.25	67.83	72.47	21.29	6.71	68.01
LLaVA 1.5-7B 2-shot	51.98	6.14	50.39	3.98	51.28	5.24	52.83	6.56	2.44	54.40	68.85	23.58	67.65	9.44	4.69	45.58	57.16	9.16	3.57	49.99
InternVL 3-8B 1-shot	74.40	27.94	75.70	28.50	74.89	28.06	74.92	28.32	9.18	67.77	75.44	30.32	69.46	7.04	8.63	63.21	74.13	25.03	8.91	65.49
InternVL 3-8B 2-shot	75.48	30.00	76.24	30.14	75.69	30.34	75.61	30.27	9.32	68.25	76.11	31.96	69.45	6.98	8.31	63.76	74.76	26.62	8.82	66.01

Appendix

Table 14: Comparison of CODA with few-shot baselines under Open-QA and report generation settings.

Panel of Figure 7 — Figure 7: Hyperparameter study on $\lambda_{g}$ and $\lambda_{d}$ .

Figure 8: A case study of Transfers task under Closed-QA setting.

Figure 9: A case study of PadChest dataset under Report Generation setting.

EHR Foundation Models

See all 19 papers

Cohort-Anchored Foundation Models for Electronic Health Records: From Risk Scores to Auditable Peer Cohorts
Jun 20, 2026Kaiping ZhengClinical Decision-Making Multimodal Model Interpretability
National University of Singapore
Uncertainty-Aware Foundation Models for Clinical Data
Apr 5, 2026Qian Zhou, Yuanyun Zhang, Shi LiEpistemic Uncertainty Multimodal Foundation Models
University of the Chinese Academy of Sciences · Columbia University
Reinforcement Learning over Patient Trajectories for Clinical Reasoning in EHR Foundation Models
Sep 10, 2026Yuxin Xiao, Sheng Zhang, Chandan Singh +4EHR Foundation Models Reinforcement Learning
MIT · Microsoft Research

Jun 20, 2026J/K move · Enter open · S save

Cohort-Anchored Foundation Models for Electronic Health Records: From Risk Scores to Auditable Peer Cohorts

Kaiping Zheng

National University of Singapore

Clinical Decision-Making Multimodal Model Interpretability Longitudinal EHR Modeling Clinical Risk Prediction EHR Foundation Models

Open paper AskPDF arXiv

Foundation models have achieved remarkable performance across medical question answering, imaging, and electronic health record (EHR) tasks, yet reliable clinical deployment remains challenging due to limited interpretability, vulnerability to distribution shift, and weak alignment with clinician reasoning. We argue that these limitations arise because existing approaches prioritize representation learning while treating patient comparison as an emergent property rather than a primary source of clinical evidence. To address this gap, we propose CAFM, a Cohort-Anchored Foundation Model framework that elevates patient cohorts to a first-class object throughout the learning pipeline. The framework consists of four stages: deviation-aware data curation, cohort-conditioned pretraining, multimodal cohort alignment, and clinician-in-the-loop refinement. Together, these stages improve data quality, organize representations around clinically meaningful cohort structure, preserve modality-specific relationships, and support auditable clinical decision-making. The framework is compositional and can augment existing EHR foundation models without modifying their underlying encoders. We illustrate CAFM through four clinical case studies spanning acute kidney injury prediction, cardiovascular risk stratification from electrocardiograms, optic neuropathy triage from orbital imaging, and electroretinogram-grounded report generation. We further present five empirically testable hypotheses and identify open challenges in data quality, irregular temporality, multimodal learning, distribution shift, and evaluation beyond predictive accuracy. We argue that explicitly anchoring foundation models to patient cohorts provides a principled path toward trustworthy clinical AI.

Abstract

EHR Foundation Models

Explore similar work

Cohort-Anchored Foundation Models for Electronic Health Records: From Risk Scores to Auditable Peer Cohorts

Uncertainty-Aware Foundation Models for Clinical Data

Reinforcement Learning over Patient Trajectories for Clinical Reasoning in EHR Foundation Models