OmitEHR: Scalable Omission Detection in LLM-Generated Clinical Summaries
Authors: Bridget Lin, Ryan Welch, Bobak Seddighzadeh, Fran Riley, Kaitlyn Stoehr, Margarita I. Pipinos, Dimitri Speron, Emma V. Sage, Sneha Jain, Nigam H. Shah, Emily Alsentzer
Organizations: Departments of 1Biomedical Data Science, 2Computer Science, and 3Medicine, Stanford University, USA
Health systems increasingly rely on large language models (LLMs) to draft summaries of patient medical records, yet evaluations now find omissions, the absence of clinically important information, to be the most common failure mode. Detecting omissions in an LLM-generated summary requires knowing which facts in the record are salient, a signal that existing methods hand-specify, ignore, or recompute at inference with costly LLM judges. We present OmitEHR, a reference-free omission detector that learns clinical salience from historical clinician-written summaries predating widespread LLM use. OmitEHR learns to predict which source-note sentences clinicians include in their summaries, then surfaces high-salience sentences not covered by an LLM-generated summary as candidate omissions. We apply OmitEHR to brief hospital course (BHC) summarization using 5,000 admissions from a large academic medical center. On held-out admissions, OmitEHR outperformed all evaluated baselines at predicting salient source sentences, including a prompted LLM judge. The full omission detection pipeline took 17.7 to 19.5 seconds and cost $0.03 to $0.13 per summary depending on the base model, 1.6 to 1.7 times faster and 5.6 to 6.7 times cheaper than scoring every source sentence with an LLM judge. In a blinded review of 25 admissions, two board-certified clinicians confirmed a mean of 5.6 clinically meaningful omissions per summary surfaced by OmitEHR, compared with 3.3 for a full LLM pipeline, counting candidates confirmed by at least one clinician. The efficiency and validity of OmitEHR make it a practical tool for health systems to flag omissions before a summary reaches the clinician.
Figures & tables
Figure 1: The OmitEHR pipeline. Labels indicate content absent from (0), indirectly covered by (1), or directly stated in (2) the BHC. (a) At training time, an LLM aligns each source-note sentence to the clinician-written BHC, and the resulting inclusion labels train a contextual salience model. (b) At deployment, the model ranks the sentences of a new admission by salience, only the highest-ranked sentences are checked for coverage against the LLM-written BHC, and uncovered sentences are surfaced as candidate omissions. Examples are illustrative.
Figure 2: Example candidate omissions with completed fields: source sentence and note type, verdict buttons, LLM-generated rationale, and the coverage judge’s reason. Patient content is synthetic.
BHC excerpt. “Etiology of symptoms include bronchiectasis flare, PNA, or viral process, with the first most likely.”
Label
Definition
Source-note example
Not related (0)
The BHC makes no claim this sentence supports.
“All other systems reviewed and are negative.”
Indirectly related (1)
Supports a BHC claim but adds a value, timing, or method the BHC does not state.
“Chronic bronchiectasis, presented with 3 days of worsening SOB.”
Directly included (2)
States the same fact as the BHC, in the same or different words.
“Etiology of symptoms include bronchiectasis flare, PNA, or viral process.”
Table 1: Inclusion-label rubric used for training and inference. Examples. The review of systems supports no BHC claim (0). The bronchiectasis history relates to the BHC’s etiology claim but adds history and symptom timing the BHC does not state (1). The etiology line states the same candidate causes as the BHC (2).
Family
Model
AUROC
AUPRC
R@100
Heuristic
Position
0.537±.004
0.394±.007
0.313±.014
Length
0.593±.004
0.452±.006
0.360±.014
Repetition
0.506±.004
0.379±.007
0.292±.013
Lexical
Token naive Bayes
0.730±.004
0.588±.008
0.458±.016
Token naive Bayes (length-normalized)
0.740±.004
0.567±.008
0.448±.017
TF–IDF + logistic regression
0.767±.003
0.621±.007
0.476±.016
Table 2: Admission-level salience classification on the held-out test set (750 admissions, over 446K sentences). R@100 is salience recall among the 100 highest-ranked sentences per admission. Each cell reports the point estimate ± the half-width of the 95% hierarchical-bootstrap confidence interval. Bold marks the best result.
Figure 3: Blinded clinician review of candidate omissions surfaced by OmitEHR and the full LLM pipeline (GPT-5-mini) on 25 LLM-generated BHCs. (a) Verdicts from two board-certified clinicians on the ten highest-ranked candidates per admission (250 per pipeline), each reviewed independently. (b) Mean number of omissions per summary confirmed by at least one clinician among the top n ranked candidates. At n=10 , OmitEHR surfaced 2.3 more confirmed omissions per summary than the full LLM pipeline. Additional precision results reporting confirmation by both clinicians are reported in Table 11 .
Figure 4: Mean cost and latency of screening one generated BHC, by pipeline stage, across the 25 evaluation admissions. Each row uses the specified model for all LLM calls. Within each row, OmitEHR and the full LLM pipeline differ only in salience scoring: OmitEHR uses its learned salience model, whereas the full LLM pipeline scores every source sentence with the LLM. OmitEHR’s salience step incurs approximately $0.002 in GPU cost and is too small to be visible. Latencies are wall-clock times with up to 16 concurrent API calls. Details in Appendix G .
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Split
Sentences
0: Not related
1: Indirectly related
2: Directly included
Train
2,054,630
1,312,014 (63.9%)
620,859 (30.2%)
121,757 (5.9%)
Dev
397,998
248,090 (62.3%)
125,557 (31.5%)
24,351 (6.1%)
Test
446,467
282,259 (63.2%)
138,679 (31.1%)
25,529 (5.7%)
Appendix
Table 3: Silver inclusion-label distribution by split. Values are sentence counts with within-split percentages in parentheses.
Figure 5: Interface used by clinicians to review source chunks and assign entailment labels relative to the corresponding clinician-written BHC.
Raters
Exact agr.
Wtd. κ
Clinician 1 vs. 2
81.0%
0.647
Clinician 1 vs. GPT-5-mini
70.7%
0.454
Clinician 2 vs. GPT-5-mini
68.7%
0.414
Appendix
Table 4: Pairwise annotation agreement on the 300-sentence blinded review set. Weighted κ uses linear weights across the ordered three-class label scheme.
Initial disagreement pattern
n
Adjudicator agrees with LLM
R1 and R2 agree against LLM
58
26 (44.8%)
R1 and R2 split, one agrees with LLM
48
31 (64.6%)
R1 and R2 split, neither agrees with LLM
9
4 (44.4%)
All initially disputed sentences
115
61 (53.0%)
Appendix
Table 5: Adjudicator review of the 115 sentences for which at least one clinician disagreed with GPT-5-mini. R1 and R2 denote the two clinicians.
Figure 6: The contextual salience model. Each sentence embedding ei is combined with projected structural features mi and a position embedding, contextualized by an admission-level Transformer encoder, and scored by a per-sentence head.
Family
Model
R@10
R@50
R@100
R@200
R@400
Heuristic
Position
0.031±.002
0.158±.009
0.313±.014
0.567±.020
0.832±.018
Length
0.047±.003
0.200±.009
0.360±.014
0.606±.018
0.847±.017
Repetition
0.030±.002
0.148±.008
0.292±.013
0.547±.019
0.822±.018
Lexical
Token naive Bayes
0.058±.003
0.267±.011
0.458±.016
0.696±.017
0.888±.015
Token naive Bayes (length-normalized)
0.051±.003
0.247±.011
0.448±.017
0.703±.019
0.896±.015
TF–IDF + logistic regression
0.065±.003
0.275±.012
0.476±.016
0.720±.018
0.901±.014
Appendix
Table 6: Admission-level salience recall at K on the held-out test set (750 admissions). R@ K is the fraction of an admission’s salient sentences ranked among its top K , averaged over admissions. Each cell reports the point estimate ± the half-width of the 95% hierarchical-bootstrap confidence interval. Bold marks the best result in each column.
Model
AUROC
AUPRC
R@10
R@50
R@100
R@200
Position
0.589±.010
0.087±.005
0.025±.006
0.172±.016
0.366±.022
0.629±.023
TF–IDF + logistic regression
0.807±.007
0.246±.011
0.145±.011
0.460±.019
0.657±.019
0.844±.016
GPT-5-mini judge (zero-shot)
0.749±.007
0.149±.007
0.080±.009
0.365±.020
0.577±.021
0.775±.018
OmitEHR-fine-tuned
0.882±.005
0.347±.013
0.216±.015
0.590±.019
0.777±.017
0.910±.012
OmitEHR-contextual
0.886±.005
0.356±.014
0.223±.015
0.592±.019
0.785±.017
0.913±.012
Appendix
Table 7: Admission-level salience classification and recall at K with label 2 as the only positive class, on the held-out test set (750 admissions, 5.7% positive). OmitEHR models use BioClinRoBERTa. Position and TF–IDF + logistic regression are the best heuristic and lexical baselines. R@ K is defined as in Table 6 . Each cell reports the point estimate ± the half-width of the 95% hierarchical-bootstrap confidence interval. Bold marks the best result.
AUROC
AUPRC
Training target
Evaluation labels
OmitEHR
LLM judge
p
OmitEHR
LLM judge
p
Labels 1 or 2
Silver (GPT-5-mini)
0.842
0.819
0.313
0.717
0.699
0.621
Gold (clinicians)
0.820
0.857
0.137
0.702
0.764
0.203
Label 2 only
Silver (GPT-5-mini)
0.876
0.692
0.031
0.478
0.330
0.078
Gold (clinicians)
0.846
0.691
0.095
0.502
0.318
0.018
Appendix
Table 8: Admission-level salience ranking of OmitEHR-contextual and the LLM judge on the 300 clinician-annotated sentences of Appendix B , scored against GPT-5-mini labels (silver) and clinician majority labels (gold) with the positive class matching the training target. The gold label-2 evaluation uses the 292 sentences with a resolved majority. In the second block the judge is reprompted for direct inclusion. p -values are from paired admission-level sign-flip tests.
Coverage outcome
n (%)
Directly included evidence
1,248 (49.9%)
Indirectly related
628 (25.1%)
Not related (omitted content)
624 (25.0%)
Total
2,500 (100.0%)
Appendix
Table 9: Coverage of the top 100 deduplicated salient source sentences per admission in the review set ( n=25 admissions; 2,500 sentences).
Clinician A
Clinician B
OmitEHR
LLM pipeline
True omission
True omission
71 (28.4)
17 (6.8)
Not omission
Not omission
109 (43.6)
167 (66.8)
True omission
Not omission
46 (18.4)
42 (16.8)
Not omission
True omission
24 (9.6)
24 (9.6)
Confirmed by either clinician
141 (56.4)
83 (33.2)
Raw agreement
180 (72.0)
184 (73.6)
Appendix
Table 10: Paired clinician decisions on 250 candidates per pipeline across the same 25 admissions. Entries report counts (percentages), except for Cohen’s κ . The one missing clinician judgment for the LLM pipeline was conservatively treated as a not-omission decision.
Confirmation
n
OmitEHR
LLM pipeline
Either clinician
1
48.0%
52.0%
3
61.3%
44.0%
5
64.0%
38.4%
10
56.4%
33.2%
Both clinicians
1
20.0%
16.0%
3
29.3%
8.0%
Appendix
Table 11: Clinician-confirmed Precision@ n across 25 admissions, with K=100 source sentences checked for coverage. Either-clinician confirmation requires at least one clinician to confirm an omission; both clinician confirmation requires both.
OmitEHR
Full LLM pipeline
Rejected by either clinician
179
233
Rejection reasons, n (% of rejected)
Clinically unimportant
146 (81.6)
185 (79.4)
Already covered by BHC
0 66 (36.9)
0 98 (42.1)
Unspecified
00 8 ( 4.5)
0 35 (15.0)
Appendix
Table 12: Rejection reasons for candidates rejected by at least one clinician across 25 admissions. Percentages use each pipeline’s rejected candidates as the denominator. A candidate counts toward every reason cited by either clinician, so reasons are not mutually exclusive. Unspecified : rejected without a reason given.
Split
Input tokens
Output tokens
Total cost (USD)
Train
734.5M
114.4M
$412.37
Dev
233.4M
30.4M
$119.20
Test
303.8M
35.3M
$146.57
Total
1,271.8M
180.1M
$678.14
Appendix
Table 13: Token consumption and estimated cost of the one-time offline labeling of all 5,000 admissions with GPT-5-mini, using 0.25permillioninputtokensand2.00 per million output tokens.
Model
Calls
Tokens (in / out)
Cost
Latency
Salience scoring task
OmitEHR
0
0 / 0
$0.002 †
2.0 s
GPT-5-mini
41.6
492.2K / 5.2K
$0.134
12.7 s
Gemini-2.5-pro
41.9
526.6K / 7.8K
$0.737
15.9 s
Coverage labeling task
GPT-5-mini
25.5
47.4K / 7.8K
$0.027
11.4 s
Appendix
Table 14: Mean per-admission API calls, tokens, cost, and latency of each pipeline step by model over 25 admissions. Both coverage runs judge the same top-100 candidates, and their call counts differ only through retries. † GPU time at $3 per H100-hour, no API charge.
Stage
Median
Min to Max
Shared stages (no API calls)
Sentence splitting
2.9
0.6 to 11.3
Top-100 candidate selection
0.8
0.2 to 3.3
Salience scoring
OmitEHR
1.9
0.4 to 5.8
GPT-5-mini, every sentence
9.3
2.8 to 50.1
Appendix
Table 15: Spread of per-admission latency in seconds for each pipeline stage over 25 admissions. Mean sentence splitting and top-100 selection times are 3.3 s and 1.0 s, and the other means are in Table 14 . A pipeline’s end-to-end latency is the sum of the two shared stages, one salience row, and one coverage row.
Large language models (LLMs) have shown promise on summarization tasks, but they often produce hallucinations, which are unsupported or incorrect statements that limit their reliability in specialized healthcare applications. We introduce Hallucination Detection Guided Self-Refinement (HDSR), an inference-time method that leverages hallucination detectors to guide iterative summary revisions toward factual corrections. Building on this, we propose HDSR for Preference Learning (HDSR-PL), which converts detector-guided refinement trajectories into preference pairs for model finetuning. Extensive experiments show that our methods substantially reduce hallucinations for Llama and Gemma models in summarizing real-world clinical notes from MIMIC-IV-Note v2.2. For example, HDSR reduces 24% and HDSR-PL reduces 48% hallucinations in Llama-3.1-8B-Instruct. Importantly, both methods preserve summary fluency, coherence, and relevance according to human expert and LLM-Jury evaluations. Together, these results demonstrate that detection-informed refinement and preference learning offer an automated solution for improving factual faithfulness in clinical summarization.
Shamanth Kuthpadi Seethakantha, Dung Ngoc Thai, Vara Prasad Gudi +6
1Manning College of Information and Computer Sciences · 2Ensemble HP · 3Columbia University +1
Large language models (LLMs) are increasingly used for medical summarization but can omit medically important information and introduce unsupported claims. Existing error-detection methods produce heuristic or uncalibrated scores, providing no formal control over missed errors and no principled way to trade off safety against clinician review burden. We introduce Conformal Assessment for Risk Evaluation (CARE), a post-hoc risk-control layer that adds calibrated omission and hallucination annotations without retraining the summarizer. CARE uses scalar conformal risk control (CRC) to bound the probability that a document contains an unflagged hallucinated sentence, and Learn-Then-Test fixed-sequence testing (LTT-FST) to control the expected fraction of true omissions left unsurfaced for review. Omission surfacing depends jointly on sentence importance and summary coverage, a two-dimensional threshold-selection problem; we show that calibrating one gate and then imposing an uncalibrated second gate can invalidate risk control. Across five medical summarization tasks, mean held-out violation over 100 random calibration/test resplits remains at or below the target for both controllers. At α=0.15, LTT-FST achieves the lowest review workload on three of five datasets among omission baselines held to the same confidence guarantee. Review workload varies by task and is highest for long, high-compression documents. On 46 clinician-annotated documents, the calibrated hallucination controller flags 88% of clinician-identified hallucinated sentences. In a preliminary study of 75 clinician reviews, assisted review-time omission coverage increased from 50.4% to 79.0% without increasing review time. Overall, CARE provides a principled framework for targeted, risk-controlled review of LLM-generated medical summaries.
Suhana Bedi, Bridget Lin, Anson Y. Zhou +5
Department of Biomedical Data Science, Stanford University · Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University · Department of Computer Science, Stanford University +2
Ambient AI scribes draft clinical notes, and published audits find their dominant error is omission: information the encounter established that the note fails to record. The standard check is an LLM judge: a second model reads the note against the transcript and flags problems. We ask whether judges detect omissions. Public corpora cannot supply the answer key: their clinician reference notes and transcripts are materially discrepant. Our benchmark has 500 single-error note pairs from audited fact sheets, 298 with a named fact certainly absent and 202 added-or-altered controls. Across eight judge designs, paired discrimination (the flawed note below its clean twin, 0.5 a coin flip) reads 0.79-0.94 on added or altered content and 0.50-0.63 on omissions. On single notes, no design flags omissions reliably more often than perfect notes. Wording changes, voting and GEPA prompt optimisation move the operating point without creating usable detection. Restructuring the task recovers it: list the facts the transcript establishes, then check the note for each. Two methods reach it independently and trade off: a per-fact pipeline, and a GEPA-evolved prompt doing the same in one call. The pipeline's flags name the missing fact and its severity at 2.7% false alarms. The single call detects more (36.9% against 24.6%, p=0.002) at 6.2% false alarms and a tenth of the cost per note. A physician author validated 70 items and, where the two routes disagree, sided with the pipeline on 10 of 10 (p=0.002). A second clinician, not an author, graded the severity rubric blind and agrees to within a grade. On real vendor notes from a companion census no benchmark threshold transfers, but the re-calibrated single call detects more than the best of the eight at half its false-alarm rate. Omissions whose fact is restated elsewhere defeat both routes. We release the benchmark, prompts and judgements.