CareTransition-Audit: A Benchmark to Audit Discharge Summaries for Efficient Care Transitions

Akshat Dasula; Prasanna Desikan; Jaideep Srivastava; Shivali Dalmia; Abhishek Mukherji

cs.AIApr 7, 2026

CareTransition-Audit: A Benchmark to Audit Discharge Summaries for Efficient Care Transitions

Authors: Akshat Dasula, Prasanna Desikan, Jaideep Srivastava, Shivali Dalmia, Abhishek Mukherji

Organizations: Centific AI Research, Redmond, USA · Department of Computer Science & Engineering, University of Minnesota-Twin Cities, Minneapolis, USA

Abstract

Incomplete or inconsistent discharge documentation drives care fragmentation and avoidable readmissions. Despite its critical role in patient safety, auditing discharge summaries relies on manual review and does not scale. We propose an automated framework for auditing discharge summaries using large language models (LLMs). Our approach operationalizes the DISCHARGED framework into a checklist of 46 questions. Using 50 summaries from the MIMIC-IV database, with clinician ground-truth labels, we benchmark 11 LLMs. Model-assessed mean documentation completeness ranges from 54.9% to 74.2%, and the best-performing models achieve a Cohen's kappa values around 0.5 against clinician labels, indicating moderate agreement. All models struggle to identify ambiguous documentation (Unclear), highlighting a key gap in current automated auditing. This work provides a clinician-validated benchmark and zero-shot baselines for systematic quality improvement in clinical documentation.

Figures & tables

Component	#Q	Audit Focus
(D)emographics	3	Identity, document placement
(I)mportant Alerts	3	Allergies, risks, precautions
(S)ocial Setup	2	Lifestyle, social context
(C)omp. History	4	Prior diagnoses, medications
(H)istory & Exams	8	Admission, vitals, exam
(A)ssessment & Course	8	Diagnoses, course, management

Table 1: Component-wise Audit Questions’ Structure

Model	Accuracy	$\kappa$	W-F1	$\rho$	$n$
Sonnet 4.5	.804 [.788, .821]	.496 [.455, .533]	.815 [.800, .831]	.432 [ $-$ .178, .641]	2168
Gemini 3 Flash	.814 [.797, .830]	.483 [.441, .520]	.822 [.806, .838]	.537 [ $-$ .321, .706]	2176
DeepSeek V3	.743 [.724, .761]	.423 [.388, .457]	.775 [.759, .791]	.413 [ $-$ .146, .624]	2176
GPT-5.4	.772 [.753, .790]	.420 [.380, .459]	.790 [.774, .808]	.365 [ $-$ .082, .585]	2114
Nova 2 Lite	.747 [.728, .765]	.401 [.365, .439]	.779 [.762, .795]	.359 [ $-$ .102, .578]	2049
Nemotron 49B	.719 [.700, .739]	.373 [.337, .410]	.749 [.732, .765]	.474 [ $-$ .241, .656]	2016

Table 2: Overall agreement between each LLM and the clinician on

n=50

MIMIC-IV discharge summaries. Values are point estimates with 95% bootstrap confidence intervals (1,000 resamples). Models ranked by

\kappa

n

= number of evaluated (summary, question) pairs out of

50\times 46=2{,}300

Figure 1: Per-summary completeness scores (proportion of Yes )

Figure 2: Cohen’s $\kappa$ by DISCHARGED component and model

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

#	ID	Question	N/A	Condition
Prompt 1: Demographics + Important Alerts
1	D1	Are basic patient demographics (age or date of birth, and sex) documented in the discharge summary?	No	—
2	D2	Is a patient identifier (e.g. name, medical record number, or patient identification number) documented, even if de-identified?	No	—
3	D3	Is patient contact information (e.g. address or phone number) documented, even if de-identified or blank?	No	—
4	I1	Is the patient’s allergy status documented (either specific allergies listed, or an explicit statement such as NKDA/NDA/no known allergies)?	No	—
5	I2	If specific allergies are listed, are the allergens and their reaction types (e.g. rash, anaphylaxis) documented?	Yes	Patient has no allergies

Appendix

Table 3: Full 46-question DISCHARGED audit checklist, grouped by prompt. N/A is admissible only for conditional questions.

ID	Question (abbreviated)	Count
S2	Pre-hospitalization functional status	11
R6	Restart plan for stopped medications	4
S1	Social history documented	3
C2	Past surgical history documented	3
R5	Clinical rationale for medication changes	2
Add4	Type of discharge documented	2

Appendix

Table 4: Per-question distribution of clinician Unclear labels across 50 summaries. Questions not appearing in the table received zero Unclear labels.

† Locally deployed.
Model	Yes	No	Unclear	N/A
Gemini 3 Flash	20	17	1	0
Sonnet 4.5	15	20	2	0
Qwen 2.5-7B †	10	13	10	5

Appendix

Table 5: Model label distributions on the 38 (summary, question) pairs where the clinician labeled Unclear . Gemini and Sonnet rarely produce Unclear , splitting their forced choices in opposite directions on the same pairs; Qwen hedges far more readily.

LLM Auditing

Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.

See all 639 papers

Toward Complete Hospital Discharge Summarization with Abstract Meaning Representation
Sep 11, 2026Paul Landes, Sitara Rao, Aaron Jeremy Chaise +1Medical Report Generation LLM Grounding
Department of Computer Science University of Illinois Chicago Chicago, USA · College of Medicine University of Illinois Chicago Chicago, USA
Phase 1 Implementation of LLM-generated Discharge Summaries showing high Adoption in a Dutch Academic Hospital
Mar 27, 2026Nettuno Nadalini, Tarannom Mehri, Anne H Hoekman +6Medical Report Generation Medical LLMs
Department of Health Information Office, Data & Digitalization, University Medical Center, Groningen, the Netherlands · Department of Trauma Surgery / Orthopedics, University Medical Center, Groningen, the Netherlands · Department of Orthopaedic Surgery, Massachusetts General Hospital, Boston, MA, USA +5
DischargeBench: Pre-Deployment Evaluation of LLM Discharge Educators through Persona-Grounded Patient Simulation
Jul 22, 2026Won Seok Jang, Youngbin Jang, Chanyoung Ko +3Clinical Language Model Evaluation Medical LLMs
University of Massachusetts Lowell

Sep 11, 2026J/K move · Enter open · S save

Toward Complete Hospital Discharge Summarization with Abstract Meaning Representation

Paul Landes, Sitara Rao, Aaron Jeremy Chaise, Barbara Di Eugenio

Department of Computer Science University of Illinois Chicago Chicago, USA · College of Medicine University of Illinois Chicago Chicago, USA

Medical Report Generation LLM Grounding Electronic Health Records

Open paper AskPDF arXiv

Discharge summaries are lengthy medical documents that summarize a hospital in-patient visit. Automatically generating them can reduce documentation burden and return clinician time to patient care. Whereas Large Language Model (LLMs) could be used for this task, their Achilles heel is hallucinations, which can have drastic consequences for clinical documentation. We present an evidence-driven alignment framework for discharge summarization at the clinical encounter level, that treats provenance as a first-class constraint, using semantic graphs and deep learning models. Each summary sentence is selected and organized via cross-document semantic alignment and is accompanied by explicit evidence links to its source spans. We show our results on two corpora: a publicly available corpus (MIMIC-III) and clinical notes written by physicians at the University of Illinois Hospital (UIC Health). Additionally, we make source code and trained models available.

Abstract

LLM Auditing

Explore similar work

Toward Complete Hospital Discharge Summarization with Abstract Meaning Representation

Phase 1 Implementation of LLM-generated Discharge Summaries showing high Adoption in a Dutch Academic Hospital

DischargeBench: Pre-Deployment Evaluation of LLM Discharge Educators through Persona-Grounded Patient Simulation