Discharge Summary Generation Is Not Just Summarization: A Reality Check for Evaluation
Authors: Radhika Dua, Frederick Wolff, Kyunghyun Cho, Eric Karl Oermann
Organizations: Courant Institute School of Mathematics, Computing, and Data Science, New York University · Department of Neurosurgery, NYU Langone Health · Department of Computer Science, ETH Zurich
Discharge summaries combine a retrospective account of hospitalization with discharge-time decisions. Physician-written references therefore contain both facts supported by prior clinical notes and reference-only content, potentially reflecting clinical synthesis, handoffs, or discharge planning. Reference-only evaluation can conflate failure to recover available information with failure to reproduce content absent from the model input. We introduce FairSumEval, an evidence-aware, fact-level framework that assesses whether each reference and generated fact is supported by the model's evidence and captured by the other summary. The resulting outcome map separates matches, omissions, and additions. Two complementary metrics, Soft IoU-S and Soft IoU-NS, measure reference-generation agreement for facts with and without identified evidence support. On 200 MIMIC-III admissions, 18.8% of physician reference facts have identified support in the prior evidence, compared with 65.4% and 63.5% of facts generated by a general-purpose and a medically specialized model. Overlap is substantially higher for supported content (0.255 and 0.236) than for content without identified support (0.030 and 0.026). The two scores are weakly correlated across admissions (rho = 0.22 and 0.17), and conventional metrics do not consistently capture both dimensions. FairSumEval distinguishes evidence-grounded summarization from agreement on content without identified support, revealing what models omit and add in each category.
Figures & tables
Figure 1: Why reference-only evaluation is misleading for discharge-summary generation. (a) Physician-written summaries R combine prior evidence E with discharge-time and inference-heavy content, whereas generated summaries G typically use E alone, making them asymmetric evaluation targets. (b) References and model outputs contain comparable numbers of evidence-supported facts, but references contain far more facts not explicitly supported by prior evidence. A summarizer perfectly reproducing all evidence-supported content would still recover only 18.8% of reference facts. (c) Existing metrics see only one side of the task: reference-similarity metrics compare G with R while ignoring E , whereas evidence-consistency metrics compare G with E while ignoring R . Standard evaluation can therefore conflate missed evidence-grounded summarization with missed discharge-time inference and future-care synthesis.
Figure 2: Overview of FairSumEval. (a) Given prior evidence E , physician-written reference R , and generated summary G , the Atomic Fact Extraction Agent extracts facts from both summaries. The Evidence Support and Cross-Summary Matching Agents assess support in E and coverage by the opposite summary, respectively. (b) The Outcome Mapper combines these decisions with fact origin to distinguish omissions, matches, and additions within the support in E and no support in E strata. Match cells retain directional coverage counts without assuming one-to-one correspondences.
Figure 3: Evidence-aware outcome maps for Llama-3.3-70B and Med42-v2-70B. Rows separate facts with identified support in E (blue) from those without (orange). Green, gold, and purple regions represent reference-only facts, matches, and generation-only facts, respectively. Circle areas scale with fact counts in each summary and evidence stratum; gold lens areas scale with the mean directional match count. Gold labels report directional match counts in the order R→G / G→R , with colors corresponding to the reference and generation. These counts may differ because facts are extracted independently from R and G . All counts are pooled across 200 admissions.
SoftIoU-S↑
SoftIoU-NS↑
Generation model
Mean
Median
Mean
Median
Llama-3.3-70B
0.255 [0.240, 0.272]
0.255 [0.190, 0.307]
0.030 [0.026, 0.033]
0.025 [0.011, 0.042]
Med42-v2-70B
0.236 [0.221, 0.252]
0.231 [0.158, 0.297]
0.026 [0.023, 0.029]
0.020 [0.011, 0.034]
Table 1: Admission-level Soft IoU. Soft IoU-S and Soft IoU-NS measure overlap among facts with and without identified evidence support, respectively. Columns report means [95% bootstrap CI] and medians [Q1,Q3] . Both models show substantially closer agreement with the physician reference on supported content than on content without identified support.
Figure 4: Evidence-aware evaluation reveals two non-redundant dimensions of summary overlap. Each point represents one admission for Llama-3.3-70B (left) or Med42-v2-70B (right). Soft IoU-S and Soft IoU-NS measure reference–generation overlap among facts with and without identified evidence support, respectively. Both models show substantially higher supported overlap, but the two scores are only weakly associated (Spearman ρ=0.22 and 0.17 , respectively). Thus, stronger agreement on supported content does not reliably predict greater agreement on content without identified support, motivating separate reporting. Trend lines are only visual aids.
Figure 5: Conventional metrics do not consistently capture evidence-aware agreement. Each cell reports the admission-level Spearman correlation of a metric with Soft IoU-S or Soft IoU-NS ( N=200 for Llama-3.3-70B; N=199 – 200 for Med42-v2-70B). ROUGE-L and BLEU-4 correlate more strongly with Soft IoU-NS for both models, indicating closer alignment with agreement on content without identified evidence support. BERTScore shows this pattern for Llama but no clear preference for either dimension for Med42; SummaC-ZS is weakly associated with both. These metrics therefore provide no consistent substitute for separating agreement by evidence support.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Evaluation objective
Llama
Med42
Tie
Fewer supported omissions
48
98
54
Fewer no support omissions
41
53
106
Fewer supported additions
159
21
20
Fewer no support additions
135
61
4
Overall clinical quality
77
86
37
Appendix
Table 2: Objective specific model preferences. Counts show the number of admissions for which the blinded language model evaluator preferred each generation model or returned a tie. The preferred model changes across objectives, while the overall quality comparison produces no clear winner.
Stage
Quantity
Result
Extraction
Mean rating
4.8
Median [range]
5 [4–5]
Ratings ≥4
5/5
Evidence support
Overall agreement
19/20 (95.0% [76.4, 99.1])
Support in E
10/10 (100% [72.2, 100])
No support in E
9/10 (90.0% [59.6, 98.2])
Appendix
Table 3: Clinician validation of FairSumEval stage decisions. Five clinicians evaluate each stage on distinct admissions, with items balanced across FairSumEval labels. Agreement is computed over non-uncertain responses; binary estimates report 95% Wilson confidence intervals ( Wilson, 1927 ) .
Stage
Quantity
Estimate
n
Extraction
Rating mean ± SD
4.3±0.7
20
Median [range]
4[2,5]
20
Ratings ≥4
19/20
20
Evidence support
Overall agreement
88.0% [85.5, 90.0]
797
Support in E
89.9% [86.6, 92.5]
397
No support in E
86.0% [82.3, 89.1]
400
Appendix
Table 4: Blinded LLM audit of FairSumEval stage decisions. Independent Claude Opus 4.6 judgments versus FairSumEval labels. Rating mean ± SD is over summaries; binary estimates report 95% Wilson confidence intervals ( Wilson, 1927 ) .
Figure 6: The evidence-aware outcome pattern generalizes to Gemma and MedGemma. Rows separate facts with identified support in E from those without identified support. Green, gold, and purple regions denote reference-only facts, directional matches, and generation-only facts, respectively. As in the main-paper experiments, both models show substantially greater overlap within the supported stratum, whereas content without identified support is dominated by a large reference-only pool. Gold labels report directional match counts as R→G/G→R . All counts are pooled across 200 admissions.
SoftIoU-S↑
SoftIoU-NS↑
Generation model
Mean
Median
Mean
Median
Gemma-3-27B
0.239 [0.224, 0.254]
0.235 [0.173, 0.295]
0.032 [0.028, 0.036]
0.026 [0.015, 0.039]
MedGemma-27B
0.249 [0.230, 0.269]
0.239 [0.146, 0.335]
0.033 [0.029, 0.037]
0.026 [0.014, 0.045]
Appendix
Table 5: Admission-level Soft IoU on Gemma and MedGemma confirms the same evidence-aware separation. Repeating the main-paper experiment with an additional general-purpose/medical model pair yields the same separation between the two evidence-aware dimensions. Soft IoU-S is substantially higher than Soft IoU-NS for both Gemma (0.239 vs. 0.032) and MedGemma (0.249 vs. 0.033), showing that the main-paper finding generalizes beyond Llama and Med42. Columns report means [95% bootstrap CI] and medians [Q1,Q3] .
Figure 7: The non-redundancy of Soft IoU-S and Soft IoU-NS generalizes to additional models. Each point represents one admission for Gemma-3-27B (left) or MedGemma-27B (right). Soft IoU-S and Soft IoU-NS measure reference–generation overlap among facts with and without identified evidence support, respectively. The two scores remain only partially associated across admissions (Spearman ρ=0.23 and 0.40 ), showing that stronger agreement on supported content does not determine agreement on content without identified support. This reproduces the central finding from the main-paper experiments that the two evidence-aware dimensions should be reported separately. Trend lines are visual aids only.
Figure 8: Conventional metrics do not consistently capture evidence-aware agreement for Gemma and MedGemma. Each cell reports the admission-level Spearman correlation of a conventional metric with Soft IoU-S or Soft IoU-NS for Gemma-3-27B (left) and MedGemma-27B (right; N=200 for every cell). Columns distinguish reference–generation agreement among facts with identified evidence support from agreement among facts without identified support. ROUGE-L and BLEU-4 correlate more strongly with Soft IoU-NS for both models. BERTScore precision, recall and F1 show larger differences between the two correlations for Gemma than for MedGemma, while SummaC-ZS has weak correlations with both dimensions.
Source
Supported by E (%)
Reference R
18.5±0.2
Llama-3.3-70B ( G )
65.1±0.3
Med42-v2-70B ( G )
64.0±0.5
Appendix
Table 6: Seed stability of evidence-support rates (mean ± SD across evaluator seeds). For each evaluator seed, we compute the fraction of facts classified as supported by evidence E . Values report mean ± SD across three evaluator seeds (123/456/789), with generated summaries and evidence held fixed ( n=200 admissions).
Model
Matches ( R→G )
Omissions ( R -only)
Additions ( G -only)
Llama-3.3-70B
2067±22
4140±62
3492±71
Med42-v2-70B
2634±57
3573±32
4735±98
Appendix
Table 7: Seed stability of supported-stratum outcome-map counts (mean ± SD across evaluator seeds). For each evaluator seed, we pool supported-stratum directional matches, reference-only omissions, and generation-only additions across the 200 admissions. Values report mean ± SD across three evaluator seeds (123/456/789), with generated summaries and evidence held fixed.
Model
Soft IoU-S
Soft IoU-NS
Llama-3.3-70B
0.257±0.001
0.030±0.001
Med42-v2-70B
0.236±0.001
0.025±0.0003
Appendix
Table 8: Seed stability of Soft IoU scores (mean ± SD across evaluator seeds). For each evaluator seed, we compute the mean admission-level Soft IoU-S and Soft IoU-NS across the 200 admissions. The table reports the mean and SD of these values across three evaluator seeds (123/456/789), with generated summaries and evidence held fixed.
Model
IoU-S / IoU-NS
Spearman ρ
Llama-3.3-70B
8.6±0.1
0.19±0.03
Med42-v2-70B
9.3±0.1
0.13±0.04
Appendix
Table 9: Seed stability of the relationship between Soft IoU-S and Soft IoU-NS (mean ± SD across evaluator seeds). For each evaluator seed, we compute the ratio of mean Soft IoU-S to mean Soft IoU-NS and the admission-level Spearman correlation between the two scores. Values report mean ± SD across seeds (123/456/789), with generated summaries and evidence held fixed.
Selection step
Admissions
MIMIC-III admissions
58,976
Exactly one discharge summary
47,006
Held-out test split
4,809
≥1 eligible clinical note
4,786
Generation subset
1,000
Evaluation cohort
200
Appendix
Table 10: Cohort selection and evidence characteristics. Selection flow from MIMIC-III to the 200-admission evaluation cohort, cohort demographics, and the size of the evidence E per admission.