cs.CLJul 22, 2026

DischargeBench: Pre-Deployment Evaluation of LLM Discharge Educators through Persona-Grounded Patient Simulation

Authors: Won Seok Jang, Youngbin Jang, Chanyoung Ko, Hyunwook Kim, Zonghai Yao, Hong Yu

Organizations: University of Massachusetts Lowell

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Effective discharge education is personalized: a clinician adapts the discharge plan to each patient's literacy, recall, and personality. Before large language models (LLMs) take on this role with real patients, they need to be evaluated in conversations with diverse patients, yet existing evaluations target static, artifact-generation, or question-answering tasks. We introduce DischargeBench, a pre-deployment benchmark in which a candidate LLM educator teaches a persona-grounded simulated patient in an open-ended, multi-turn conversation. We curate MIMIC-IV-Ext-DischargeBench, 320 cases across 16 ICD chapters, each paired with a patient persona that varies personality, education level, health literacy, and recall. Each conversation is scored on Conversation Quality, Topic Checklist Score (TCS), Simulated Comprehension Score (SCS), and a Claim-Level Safety Audit (CLSA) by an LLM judge validated against physician annotations. Across eight closed- and open-source LLMs, the best model scored only 0.70 on TCS and 0.72 on SCS, and distrustful and anxious patients lowered comprehension for every model. DischargeBench is intended as a screening step before human evaluation, not as evidence of readiness to educate patients.

Figures & tables

Figure 1: Overview of DischargeBench. Two stages: (1) An Educator (system under test) conducts a multi-turn session with a Virtual Patient, supervised by a Simulation Orchestrator that regulates patient realism and signals session completion (320 cases). (2) An LLM-as-a-Judge scores each dialogue along Conversation Quality, Topic Checklist Score (TCS), Simulated Comprehension Score (SCS), and a Claim-Level Safety Audit (CLSA).
ComponentValues
From MIMIC-IV (per case)
DemographicsAge, gender, race/ethnicity
ICD chapter16 merged chapters, 20 cases each
Clinical profileDiagnoses, medications, allergies, medical and family history, reason for admission
Assigned persona traits
PersonalityNeutral, high-conscientiousness, minimizer, anxious, distrustful
Table 1: Components of the Virtual Patient. Cohort demographics are given in Table 3 ; trait definitions in Appendix A.2 .
Figure 2: One simulation session (illustrative dialogue). (a) Turn-by-turn dialogue with the Simulation Orchestrator’s verdict on every turn. (b) The Orchestrator’s intervention and termination rules.
Figure 3: Conversation Quality by educator model. (A)–(D) Likert scores (1–5; higher is better). (E) Flesch–Kincaid grade level of the educator’s turns; the shaded band marks the patient-education target (grade 6–8).
Figure 4: Topic Checklist Score (A) and Simulated Comprehension Score (B) by educator model (higher is better). Error bars: 95% case-bootstrap CIs ( B=10,000B=10{,}000 ).
Figure 5: Claim-Level Safety Audit by educator model (lower is better): (A) Unsafe Advice and (B) Wrong Instruction rates. Error bars: 95% case-bootstrap CIs ( B=10,000B=10{,}000 ).
Figure 6: TCS and SCS by ICD chapter, pooled across educator models. Parenthesized percentages give the share of notes with no major procedure listed. Error bars: 95% case-bootstrap CIs ( B=10,000B=10{,}000 ).
Figure 7: SCS across the five patient personalities, one line per educator model.
EvaluationMetricP–PJudge–P
Conversation Quality aSpearman ρ\rho0.2130.276
 AC2 (quad)0.8680.820
TCS cCohen’s κ\kappa0.6860.600
SCS cw κ\kappa (quad)0.8150.622
CLSA grounding bCohen’s κ\kappa0.6690.574
CLSA clinical status bw κ\kappa (quad)0.5890.412
Table 2: Judge–physician agreement by evaluation. P–P : physician–physician ceiling. a AC2 is Gwet’s AC2. b 80 claims enriched by the judge’s labels. c 25 cases, excluding items not counted in scoring.
Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

  MissingOverall
n  320
Gender, n (%)female 170 (53.1)
male 150 (46.9)
Age, mean (SD) 047.5 (11.8)
Race, n (%)WHITE 178 (55.6)
BLACK/AFRICAN AMERICAN 59 (18.4)
Appendix
Table 3: Patient Demographics of the 320-case cohort. Race categories are aggregated to the six major groups reported in MIMIC-IV; the ICD-chapter distribution is uniform by construction (16 merged chapters ×\times 20 cases).
 Clinical status
GroundingAppropriateQuestionableHarmful
EntailedFaithful relayVague relayRelayed harm
ContradictedDriftDriftWrong instruction
NeutralUngrounded- plausibleUnsupported vagueUnsafe advice
Appendix
Table 4: CLSA grid. Each atomic claim is assigned one grounding label and one clinical-status label. Bold cells are the two self-generated harms, reported as Wrong Instruction (contradicted and harmful) and Unsafe Advice (neutral and harmful).
TopicWeightQuestion
Discharge diagnosis1Q1. Did the conversation discuss the patient’s main discharge diagnosis (or chief complaint) mentioned in the patient note?
New medication1Q2. Did the conversation discuss newly added discharge medications mentioned in the patient note?
New medication0.5Q2.1. Did the conversation discuss the exact dosage of the medication mentioned in the patient note?
New medication0.5Q2.2. Did the conversation discuss the exact route of the medication mentioned in the patient note (e.g., oral, injection, intravenous)?
New medication0.5Q2.3. Did the conversation discuss the frequency of medication administration mentioned in the patient note (e.g., twice daily, once at bedtime)?
New medication0.5Q2.4. Did the conversation discuss any side effects or precautions related to the medication mentioned in the patient note?
Appendix
Table 5: Topic Checklist Score (TCS) questions. Parent questions carry weight 1 and sub-questions 0.5.
SCS questionClinical area (matching TCS topic)
1. What was the discharge diagnosis?Discharge diagnosis (TCS Q1)
2. What treatments did the patient receive from the hospital?Treatment during stay (TCS Q3)
3. What were the post-discharge treatments recommended to the patient?Post-discharge treatment (TCS Q4)
4. What doctors or clinics does the patient have to follow up with after this visit?Follow-up appointment (TCS Q6)
5. For what symptoms or changes should the patient return to the ED/hospital?Return to hospital (TCS Q5)
6. List the medications that the patient should take, with their name, dosage, and route.Medications (TCS Q2, which covers new medications only)
Appendix
Table 6: SCS questions and the clinical area each shares with the Topic Checklist. Each answer is graded against a reference answer extracted from the discharge note as correct (1), partially correct (0.5), or incorrect (0); a question is not counted when its reference answer is “not applicable”.
Figure 17: Stratification by ICD chapter: per-model SCS and TCS across the 16 merged chapters, pooled over five runs on the 320-case cohort.
Figure 18: Stratification by patient personality: per-model SCS and TCS across the five personalities, pooled over five runs.
Figure 19: Stratification by patient personality: per-model Naturalness (1–5; higher is better) and Readability (Flesch–Kincaid grade level; higher is harder to read, so brighter cells indicate worse readability), pooled over five runs.
Figure 20: Survey questions for the Virtual Patient evaluation, answered by each physician after every conversation.
Figure 21: Virtual Patient Simulation Quality Survey. Two physicians annotated 48 patient cases sampled uniformly across the 16 ICD chapters.
Figure 22: Stage-specific annotation instructions for the judge-validation study, shown to physicians before each block.
 Spearman ρ\rhoGwet’s AC2 (quadratic)
AxisP–PJudge–PP–PJudge–P
Naturalness0.43 [ −- 0.07, 0.77]0.33 [0.04, 0.58]0.79 [0.60, 0.93]0.65 [0.46, 0.81]
Responsiveness0.50 [0.18, 0.77]0.21 [ −- 0.06, 0.46]0.91 [0.83, 0.97]0.91 [0.84, 0.95]
Clarity−- 0.05 [ −- 0.37, 0.31]0.30 [ −- 0.01, 0.57]0.87 [0.72, 0.96]0.89 [0.81, 0.94]
Clinical Relevance−- 0.03 [ −- 0.50, 0.58]0.26 [ −- 0.04, 0.51]0.90 [0.81, 0.96]0.83 [0.73, 0.91]
Mean0.21 [ −- 0.09, 0.49]0.28 [0.07, 0.46]0.870.82
Appendix
Table 7: Conversation Quality agreement per axis, with 95% bootstrap CIs. Spearman ρ\rho measures rank agreement; Gwet’s AC2 with quadratic weights on the 1–5 scale measures absolute agreement. P–P : the two physicians on their 20 shared cases. Judge–P : the LLM judge against the physicians, pooled over the 50 cases each physician annotated alone.
Figure 23: Failure analysis over five runs of the 320-case cohort. (A) Per-model simulation outcomes. (B) Incomplete-simulation rate by model and patient personality.
Figure 24: Failure Analysis of erroneous cases stratified by the 16 merged ICD chapters. Abbreviation legend in Table 8 .
AbbreviationICD Chapter
BLDDiseases of the Blood and Blood-Forming Organs
CIRDiseases of the Circulatory System
CONCongenital Anomalies
DIGDiseases of the Digestive System
ENDEndocrine, Nutritional and Metabolic Diseases
FHSFactors Influencing Health Status and Contact with Health Services
Appendix
Table 8: Full names of the 16 merged ICD chapter abbreviations used in Figure 24 , under the version-neutral taxonomy that merges corresponding ICD-9 and ICD-10 chapters into a single clinical chapter (Appendix A.1 ).
ModelCompleteShort n (%)Char-break n (%)Low-TCS short n (%)Retained
Llama-3.3-70B-Instruct1,505194 (12.9)55 (3.7)98 (6.5)1,353
Qwen3-4B1,453357 (24.6)14 (1.0)47 (3.2)1,392
Qwen3-32B1,470375 (25.5)22 (1.5)68 (4.6)1,380
MedGemma-4b-it1,384236 (17.1)24 (1.7)84 (6.1)1,276
MedGemma-27b-text-it1,531144 (9.4)48 (3.1)62 (4.0)1,421
GPT-5.4-nano1,519150 (9.9)64 (4.2)54 (3.6)1,401
Appendix
Table 9: Clean-complete case audit (1,600 sessions per model: 320 cases ×\times 5 runs). Complete : sessions the Simulation Orchestrator approved as complete. Short : complete sessions with at most 15 turns. Char-break : complete sessions in which the Virtual Patient disclosed being an AI (e.g., “as an AI”). Low-TCS short : short sessions with TCS below 0.5. Retained : complete sessions after removing char-break and low-TCS short sessions. Percentages are relative to complete sessions. This audit is descriptive; all reported metrics use every session except system failures.
More in this topic

Clinical Language Model Evaluation

Clinical language model evaluation measures whether AI systems are accurate, safe, fair, and useful across medical tasks and care settings.

See all 415 papers

Explore similar work

CardsList

  1. EHRNote-ChatQA: A Benchmark for Evidence-Grounded Multi-Turn Clinical Question Answering over Longitudinal Discharge Summaries

    Jun 14, 2026Jiyoun Kim, Muhan Yeo, Eunhye Jang +14Clinical QAEvidence-Grounded Generation

    KAIST · Seoul National University · Seoul National University Bundang Hospital +9

  2. Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging

    Jul 11, 2026Jinglan Gong, Jiefan Lu, Hewei Guo +5Language Model Generation EvaluationPersona Consistency

    SenseTime Research · University of Science and Technology of China · Tsinghua University

  3. CareTransition-Audit: A Benchmark to Audit Discharge Summaries for Efficient Care Transitions

    Apr 7, 2026Akshat Dasula, Prasanna Desikan, Jaideep Srivastava +2LLM AuditingClinical Language Model Evaluation

    Centific AI Research, Redmond, USA · Department of Computer Science & Engineering, University of Minnesota-Twin Cities, Minneapolis, USA