Organizations: Centific AI Research, Redmond, USA · Department of Computer Science & Engineering, University of Minnesota-Twin Cities, Minneapolis, USA
Incomplete or inconsistent discharge documentation drives care fragmentation and avoidable readmissions. Despite its critical role in patient safety, auditing discharge summaries relies on manual review and does not scale. We propose an automated framework for auditing discharge summaries using large language models (LLMs). Our approach operationalizes the DISCHARGED framework into a checklist of 46 questions. Using 50 summaries from the MIMIC-IV database, with clinician ground-truth labels, we benchmark 11 LLMs. Model-assessed mean documentation completeness ranges from 54.9% to 74.2%, and the best-performing models achieve a Cohen's kappa values around 0.5 against clinician labels, indicating moderate agreement. All models struggle to identify ambiguous documentation (Unclear), highlighting a key gap in current automated auditing. This work provides a clinician-validated benchmark and zero-shot baselines for systematic quality improvement in clinical documentation.
Table 2: Overall agreement between each LLM and the clinician on n=50 MIMIC-IV discharge summaries. Values are point estimates with 95% bootstrap confidence intervals (1,000 resamples). Models ranked by κ . n = number of evaluated (summary, question) pairs out of 50×46=2,300 .
Figure 1: Per-summary completeness scores (proportion of Yes )
Figure 2: Cohen’s κ by DISCHARGED component and model
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
#
ID
Question
N/A
Condition
Prompt 1: Demographics + Important Alerts
1
D1
Are basic patient demographics (age or date of birth, and sex) documented in the discharge summary?
No
—
2
D2
Is a patient identifier (e.g. name, medical record number, or patient identification number) documented, even if de-identified?
No
—
3
D3
Is patient contact information (e.g. address or phone number) documented, even if de-identified or blank?
No
—
4
I1
Is the patient’s allergy status documented (either specific allergies listed, or an explicit statement such as NKDA/NDA/no known allergies)?
No
—
5
I2
If specific allergies are listed, are the allergens and their reaction types (e.g. rash, anaphylaxis) documented?
Yes
Patient has no allergies
Appendix
Table 3: Full 46-question DISCHARGED audit checklist, grouped by prompt. N/A is admissible only for conditional questions.
ID
Question (abbreviated)
Count
S2
Pre-hospitalization functional status
11
R6
Restart plan for stopped medications
4
S1
Social history documented
3
C2
Past surgical history documented
3
R5
Clinical rationale for medication changes
2
Add4
Type of discharge documented
2
Appendix
Table 4: Per-question distribution of clinician Unclear labels across 50 summaries. Questions not appearing in the table received zero Unclear labels.
Model
Yes
No
Unclear
N/A
Gemini 3 Flash
20
17
1
0
Sonnet 4.5
15
20
2
0
Qwen 2.5-7B †
10
13
10
5
† Locally deployed.
Appendix
Table 5: Model label distributions on the 38 (summary, question) pairs where the clinician labeled Unclear . Gemini and Sonnet rarely produce Unclear , splitting their forced choices in opposite directions on the same pairs; Qwen hedges far more readily.
Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.
Discharge summaries are lengthy medical documents that summarize a hospital in-patient visit. Automatically generating them can reduce documentation burden and return clinician time to patient care. Whereas Large Language Model (LLMs) could be used for this task, their Achilles heel is hallucinations, which can have drastic consequences for clinical documentation. We present an evidence-driven alignment framework for discharge summarization at the clinical encounter level, that treats provenance as a first-class constraint, using semantic graphs and deep learning models. Each summary sentence is selected and organized via cross-document semantic alignment and is accompanied by explicit evidence links to its source spans. We show our results on two corpora: a publicly available corpus (MIMIC-III) and clinical notes written by physicians at the University of Illinois Hospital (UIC Health). Additionally, we make source code and trained models available.
Paul Landes, Sitara Rao, Aaron Jeremy Chaise +1
Department of Computer Science University of Illinois Chicago Chicago, USA · College of Medicine University of Illinois Chicago Chicago, USA
Writing discharge summaries to transfer medical information is an important but time-consuming process that can be assisted by Large Language Models (LLMs). This prospective mixed methods pilot study evaluated an Electronic Health Record (EHR)-integrated LLM to generate discharge summaries drafts. In total, 379 discharge summaries were generated in clinical practice by 21 residents and 4 physician assistants during 9 weeks in our academic hospital. LLM-generated text was copied in 58.5% of admissions, and identifiable LLM content could be traced to 29.1% of final discharge letters. Notably, 86.9% of users self-reported a reduction in documentation time, and 60.9% a reduction in administrative workload. Intent to use after the pilot phase was high (91.3%), supporting further implementation of this use-case. Accurately measuring the documentation time of users on discharge summaries remains challenging, but will be necessary for future extrinsic evaluation of LLM-assisted documentation.
Nettuno Nadalini, Tarannom Mehri, Anne H Hoekman +6
Department of Health Information Office, Data & Digitalization, University Medical Center, Groningen, the Netherlands · Department of Trauma Surgery / Orthopedics, University Medical Center, Groningen, the Netherlands · Department of Orthopaedic Surgery, Massachusetts General Hospital, Boston, MA, USA +5
Effective discharge education is personalized: a clinician adapts the discharge plan to each patient's literacy, recall, and personality. Before large language models (LLMs) take on this role with real patients, they need to be evaluated in conversations with diverse patients, yet existing evaluations target static, artifact-generation, or question-answering tasks. We introduce DischargeBench, a pre-deployment benchmark in which a candidate LLM educator teaches a persona-grounded simulated patient in an open-ended, multi-turn conversation. We curate MIMIC-IV-Ext-DischargeBench, 320 cases across 16 ICD chapters, each paired with a patient persona that varies personality, education level, health literacy, and recall. Each conversation is scored on Conversation Quality, Topic Checklist Score (TCS), Simulated Comprehension Score (SCS), and a Claim-Level Safety Audit (CLSA) by an LLM judge validated against physician annotations. Across eight closed- and open-source LLMs, the best model scored only 0.70 on TCS and 0.72 on SCS, and distrustful and anxious patients lowered comprehension for every model. DischargeBench is intended as a screening step before human evaluation, not as evidence of readiness to educate patients.