cs.CLNov 10, 2026

Discharge Summary Generation Is Not Just Summarization: A Reality Check for Evaluation

Authors: Radhika Dua, Frederick Wolff, Kyunghyun Cho, Eric Karl Oermann

Organizations: Courant Institute School of Mathematics, Computing, and Data Science, New York University · Department of Neurosurgery, NYU Langone Health · Department of Computer Science, ETH Zurich

Abstract

Discharge summaries combine a retrospective account of hospitalization with discharge-time decisions. Physician-written references therefore contain both facts supported by prior clinical notes and reference-only content, potentially reflecting clinical synthesis, handoffs, or discharge planning. Reference-only evaluation can conflate failure to recover available information with failure to reproduce content absent from the model input. We introduce FairSumEval, an evidence-aware, fact-level framework that assesses whether each reference and generated fact is supported by the model's evidence and captured by the other summary. The resulting outcome map separates matches, omissions, and additions. Two complementary metrics, Soft IoU-S and Soft IoU-NS, measure reference-generation agreement for facts with and without identified evidence support. On 200 MIMIC-III admissions, 18.8% of physician reference facts have identified support in the prior evidence, compared with 65.4% and 63.5% of facts generated by a general-purpose and a medically specialized model. Overlap is substantially higher for supported content (0.255 and 0.236) than for content without identified support (0.030 and 0.026). The two scores are weakly correlated across admissions (rho = 0.22 and 0.17), and conventional metrics do not consistently capture both dimensions. FairSumEval distinguishes evidence-grounded summarization from agreement on content without identified support, revealing what models omit and add in each category.

Figures & tables

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. MAPLE: Medical Aspect-Based Summarization with Phrase-Level Evidence

    Jan 6, 2026Bohao Chu, Hendrik Damm, Tabea M. G. Pakull +5HealthcareText Summarization

  2. Coverage-Controlled Preference Mining from Noisy Claim Verification for Evidence-Grounded Generation

    Mar 11, 2026Weixin Liu, Congning Ni, Qingyuan Song +4Claim VerificationHealthcare

  3. Enhancing Factuality through Consensus and Consistency in Summarization Using Minimum Bayes Risk Decoding

    May 28, 2026Riza Setiawan Soetedjo, Yusuke Sakai, Hidetaka Kamigaito +3MBR DecodingText Summarization