Didactic knowledge or Clinical Cases? How Data Types Shape Medical Large Language Models

Yuzheng Fan; Haochun Wang; Sendong Zhao; Xiao Han; Ming Ma; Bing Qin

cs.AIAug 26, 2026

Didactic knowledge or Clinical Cases? How Data Types Shape Medical Large Language Models

Authors: Yuzheng Fan, Haochun Wang, Sendong Zhao, Xiao Han, Ming Ma, Bing Qin

Organizations: Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, Harbin 150001, China

Abstract

Medical large language models are commonly trained on mixtures of didactic data (e.g., textbooks) and clinical data (e.g., patient records), yet how these data types differentially shape model capabilities remains unclear. We address this issue with token-matched experiments that vary the didactic-to-clinical ratio and analyze how data composition affects performance, capability profiles, and error patterns across knowledge-intensive and clinic-oriented tasks. We uncover an asymmetric transfer across task types: clinical data improves clinic-oriented tasks while remaining competitive on knowledge-intensive ones, whereas didactic data mainly improves knowledge-intensive tasks. Error analysis suggests a knowing-doing gap, where improvements in knowledge recall do not reliably generalize to clinical reasoning. We further observe that modest amounts of clinical data yield most of the gains on EHR-grounded tasks, while the optimal mixture ratio varies with the knowledge and clinical reasoning demands of downstream tasks. These findings suggest that medical LLM data curation should be application-driven, with higher proportions of clinical data preferred for reasoning-intensive use cases.

Figures & tables

Figure 1: Examples of didactic and clinical case-based data across two QA formats.

Figure 2: Overview of the study setup, including (a) the training corpora, (b) model variants fine-tuned on different data compositions, (c) evaluation benchmarks, and (d) comparative performance results for Qwen2.5-7B.

Base Model	Model Variant	MedQA			MedMCQA		PubMedQA	MMLU	Medbullets-5
Base Model	Model Variant	step1	step2&3	All	Didactic	Clinical
Qwen2.5-7B	Base	50.22	50.67	50.43	55.80	59.60	53.00	70.98	43.18
	Instruct- $\mathcal{D}$	55.67	57.07	56.32	68.60	67.20	64.80	76.85	46.10
	Instruct- $\mathcal{C}$	55.38	57.58	56.40	67.60	71.60	70.40	76.68	46.75
	Instruct- $\mathcal{M}_{1:3}$	55.96	57.07	56.48	66.80	70.00	72.80	76.21	48.70
	Instruct- $\mathcal{M}_{1:1}$	57.44	56.90	57.19	69.20	69.00	73.00	76.12	47.40

Table 1: Performance on knowledge-intensive and reasoning-intensive benchmarks across model variants. Bold indicates the best performance.

Base Model	Model Variant	Service	Primary Diagnosis	Medication		Discharge Disposition	Discharge Instruction
Base Model	Model Variant	Service	Primary Diagnosis	F1	Jaccard	Discharge Disposition	Discharge Instruction
Qwen2.5-7B	Base	34.00	72.00	19.37	10.72	15.00	2.14
	Instruct ∗	50.00	75.00	16.01	8.70	43.00	2.51
	Instruct- $\mathcal{D}$	59.00	76.00	16.15	8.78	26.00	2.08
	Instruct- $\mathcal{C}$	73.00	83.00	30.16	17.76	52.00	2.53
	Instruct- $\mathcal{M}_{1:3}$	78.00	80.00	26.93	15.56	49.00	2.49

Table 2: Performance on ClinicalBench across model variants. ∗ indicates official instruct model. Bold indicates the best performance per task within each base model.

Model Variant	Term $\rightarrow$ Sym.	Term $\rightarrow$ Def.	Sym. $\rightarrow$ Diag.		EHR $\rightarrow$ Diag.	DBI
Model Variant	Term $\rightarrow$ Sym.	Term $\rightarrow$ Def.	Typical	Atypical	EHR $\rightarrow$ Diag.	DBI
Base	53.18	41.98	26.83	17.56	72.00	+0.01
Instruct- $\mathcal{D}$	64.70	48.40	30.24	24.39	76.00	+0.09
Instruct- $\mathcal{C}$	57.72	47.00	35.12	28.29	83.00	-0.09
Instruct- $\mathcal{M}_{1:1}$	62.80	46.72	34.63	24.39	82.00	-0.02

Table 3: Performance on each directional mapping task on Qwen2.5-7B. A positive DBI indicates concept-oriented bias; a negative indicates evidence-to-diagnosis preference. Sym., Def., and Diag. stand for Symptoms, Definition, and Diagnosis respectively. Bold indicates the best performance across model variants.

Model Variant	$K^{-}R^{-}$	$K^{-}R^{+}$	$K^{+}R^{-}$	$K^{+}R^{+}$
Base	72.50	73.01	58.10	44.00
Instruct- $\mathcal{D}$	83.85	73.01	69.83	48.33
Instruct- $\mathcal{C}$	82.13	74.60	70.94	53.00
Instruct- $\mathcal{M}_{1:3}$	82.13	76.19	67.60	50.33
Instruct- $\mathcal{M}_{1:1}$	82.82	74.60	73.18	51.66
Instruct- $\mathcal{M}_{3:1}$	84.87	73.01	71.50	50.00

Table 4: Performance across knowledge-reasoning quadrants on Qwen2.5-7B. Bold indicates the best performance per quadrant.

Model Variant	$K^{-}R^{-}$	$K^{-}R^{+}$	$K^{+}R^{-}$	$K^{+}R^{+}$
Instruct- $\mathcal{M}_{1:3}$	-0.43	+1.99	-3.06	-1.50
Instruct- $\mathcal{M}_{1:1}$	-0.17	+0.80	+2.80	+1.00
Instruct- $\mathcal{M}_{3:1}$	+1.45	-0.40	+1.39	+0.50

Table 5: Synergy analysis with ratio-weighted linear expectation. Positive values indicate performance exceeding the mixture-proportion linear baseline.

Figure 3: Error distribution in the

K^{+}R^{+}

quadrant. Bars show the proportion of each error type within the total errors for each variant in this quadrant.

Model Variant	$E_{\text{Know.}}$	$E_{\text{Reas.}}$	$E_{\text{Ctx.}}$	$E_{\text{Fmt.}}$
Base	229 (67.4%)	38 (11.2%)	60 (17.6%)	13 (3.8%)
Instruct- $\mathcal{D}$	195 (71.4%)	51 (18.7%)	24 (8.8%)	3 (1.1%)
Instruct- $\mathcal{C}$	182 (69.7%)	38 (14.6%)	34 (13.0%)	7 (2.7%)
Instruct- $\mathcal{M}_{1:1}$	185 (70.6%)	43 (16.4%)	30 (11.5%)	4 (1.5%)

Table 6: Error type distribution across data compositions, where

E_{\text{Know.}}

E_{\text{Reas.}}

E_{\text{Ctx.}}

, and

E_{\text{Fmt.}}

denote knowledge, reasoning, context, and format errors, respectively. Percentages indicate the proportion within total errors for each model.

Medical LLMs

Medical large language models are AI systems adapted to clinical and biomedical data for health reasoning, information extraction, and communication.

See all 402 papers

Medical Reasoning with Large Language Models: A Survey and MR-Bench
Mar 17, 2026Xiaohan Ren, Chenxiao Fan, Wenyin Ma +4Medical Reasoning Clinical Reasoning
School of Cyber Science and Technology, University of Science and Technology of China, Anhui, China · School of Artificial Intelligence and Data Science, University of Science and Technology of China, Anhui, China · Department of Infectious Disease, The First Affiliated Hospital of USTC, Anhui, China +2
To Adapt or not to Adapt, Rethinking the Value of Medical Knowledge-Aware Large Language Models
Apr 8, 2026Ane G. Domingo-Aldama, Iker De La Iglesia, Maitane Urruela +2Medical LLMs Domain Adaptation for LLMs
MedKIT: Evaluating Knowledge Integration and Generalization in Large Language Models
Sep 29, 2026Lukas Thede, Yash Kumar Atri, David Chen +4Continual Learning for LLMs Factual Knowledge in Language Models
University of Tübingen, Tübingen AI Center · Helmholtz Munich · Munich Center for Machine Learning (MCML) +4

Mar 17, 2026J/K move · Enter open · S save

Medical Reasoning with Large Language Models: A Survey and MR-Bench

Xiaohan Ren, Chenxiao Fan, Wenyin Ma, Hongliang He+3

School of Cyber Science and Technology, University of Science and Technology of China, Anhui, China · School of Artificial Intelligence and Data Science, University of Science and Technology of China, Anhui, China · Department of Infectious Disease, The First Affiliated Hospital of USTC, Anhui, China+2

Medical Reasoning Clinical Reasoning Clinical Language Model Evaluation Clinical Decision-Making

Open paper AskPDF arXiv

Large language models (LLMs) have achieved strong performance on medical exam-style tasks, motivating growing interest in their deployment in real-world clinical settings. However, clinical decision-making is inherently safety-critical, context-dependent, and conducted under evolving evidence. In such situations, reliable LLM performance depends not on factual recall alone, but on robust medical reasoning. In this work, we present a comprehensive review of medical reasoning with LLMs. Grounded in cognitive theories of clinical reasoning, we conceptualize medical reasoning as an iterative process of abduction, deduction, and induction, and organize existing methods into seven major technical routes spanning training-based and training-free approaches. We further conduct a unified cross-benchmark evaluation of representative medical reasoning models under a consistent experimental setting, enabling a more systematic and comparable assessment of the empirical impact of existing methods. To better assess clinically grounded reasoning, we introduce MR-Bench, a benchmark derived from real-world hospital data. Evaluations on MR-Bench expose a pronounced gap between exam-level performance and accuracy on authentic clinical decision tasks. Overall, this survey provides a unified view of existing medical reasoning methods, benchmarks, and evaluation practices, and highlights key gaps between current model performance and the requirements of real-world clinical reasoning.

Abstract

Medical LLMs

Explore similar work

Medical Reasoning with Large Language Models: A Survey and MR-Bench

To Adapt or not to Adapt, Rethinking the Value of Medical Knowledge-Aware Large Language Models

MedKIT: Evaluating Knowledge Integration and Generalization in Large Language Models