cs.CLNov 14, 2026

OmitEHR: Scalable Omission Detection in LLM-Generated Clinical Summaries

Authors: Bridget Lin, Ryan Welch, Bobak Seddighzadeh, Fran Riley, Kaitlyn Stoehr, Margarita I. Pipinos, Dimitri Speron, Emma V. Sage, 

Organizations: Departments of 1Biomedical Data Science, 2Computer Science, and 3Medicine, Stanford University, USA

Abstract

Health systems increasingly rely on large language models (LLMs) to draft summaries of patient medical records, yet evaluations now find omissions, the absence of clinically important information, to be the most common failure mode. Detecting omissions in an LLM-generated summary requires knowing which facts in the record are salient, a signal that existing methods hand-specify, ignore, or recompute at inference with costly LLM judges. We present OmitEHR, a reference-free omission detector that learns clinical salience from historical clinician-written summaries predating widespread LLM use. OmitEHR learns to predict which source-note sentences clinicians include in their summaries, then surfaces high-salience sentences not covered by an LLM-generated summary as candidate omissions. We apply OmitEHR to brief hospital course (BHC) summarization using 5,000 admissions from a large academic medical center. On held-out admissions, OmitEHR outperformed all evaluated baselines at predicting salient source sentences, including a prompted LLM judge. The full omission detection pipeline took 17.7 to 19.5 seconds and cost $0.03 to $0.13 per summary depending on the base model, 1.6 to 1.7 times faster and 5.6 to 6.7 times cheaper than scoring every source sentence with an LLM judge. In a blinded review of 25 admissions, two board-certified clinicians confirmed a mean of 5.6 clinically meaningful omissions per summary surfaced by OmitEHR, compared with 3.3 for a full LLM pipeline, counting candidates confirmed by at least one clinician. The efficiency and validity of OmitEHR make it a practical tool for health systems to flag omissions before a summary reaches the clinician.

Figures & tables

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

May 27, 2026

Hallucination Detection-Guided Preference Optimization for Clinical Summarization

Large language models (LLMs) have shown promise on summarization tasks, but they often produce hallucinations, which are unsupported or incorrect statements that limit their reliability in specialized healthcare applications. We introduce Hallucination Detection Guided Self-Refinement (HDSR), an inference-time method that leverages hallucination detectors to guide iterative summary revisions toward factual corrections. Building on this, we propose HDSR for Preference Learning (HDSR-PL), which converts detector-guided refinement trajectories into preference pairs for model finetuning. Extensive experiments show that our methods substantially reduce hallucinations for Llama and Gemma models in summarizing real-world clinical notes from MIMIC-IV-Note v2.2. For example, HDSR reduces 24% and HDSR-PL reduces 48% hallucinations in Llama-3.1-8B-Instruct. Importantly, both methods preserve summary fluency, coherence, and relevance according to human expert and LLM-Jury evaluations. Together, these results demonstrate that detection-informed refinement and preference learning offer an automated solution for improving factual faithfulness in clinical summarization.
Jun 8, 2026

CARE: A Conformal Safety Layer for Medical Summarization

Large language models (LLMs) are increasingly used for medical summarization but can omit medically important information and introduce unsupported claims. Existing error-detection methods produce heuristic or uncalibrated scores, providing no formal control over missed errors and no principled way to trade off safety against clinician review burden. We introduce Conformal Assessment for Risk Evaluation (CARE), a post-hoc risk-control layer that adds calibrated omission and hallucination annotations without retraining the summarizer. CARE uses scalar conformal risk control (CRC) to bound the probability that a document contains an unflagged hallucinated sentence, and Learn-Then-Test fixed-sequence testing (LTT-FST) to control the expected fraction of true omissions left unsurfaced for review. Omission surfacing depends jointly on sentence importance and summary coverage, a two-dimensional threshold-selection problem; we show that calibrating one gate and then imposing an uncalibrated second gate can invalidate risk control. Across five medical summarization tasks, mean held-out violation over 100 random calibration/test resplits remains at or below the target for both controllers. At α=0.15α=0.15, LTT-FST achieves the lowest review workload on three of five datasets among omission baselines held to the same confidence guarantee. Review workload varies by task and is highest for long, high-compression documents. On 46 clinician-annotated documents, the calibrated hallucination controller flags 88% of clinician-identified hallucinated sentences. In a preliminary study of 75 clinician reviews, assisted review-time omission coverage increased from 50.4% to 79.0% without increasing review time. Overall, CARE provides a principled framework for targeted, risk-controlled review of LLM-generated medical summaries.
Aug 31, 2026

LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It

Ambient AI scribes draft clinical notes, and published audits find their dominant error is omission: information the encounter established that the note fails to record. The standard check is an LLM judge: a second model reads the note against the transcript and flags problems. We ask whether judges detect omissions. Public corpora cannot supply the answer key: their clinician reference notes and transcripts are materially discrepant. Our benchmark has 500 single-error note pairs from audited fact sheets, 298 with a named fact certainly absent and 202 added-or-altered controls. Across eight judge designs, paired discrimination (the flawed note below its clean twin, 0.5 a coin flip) reads 0.79-0.94 on added or altered content and 0.50-0.63 on omissions. On single notes, no design flags omissions reliably more often than perfect notes. Wording changes, voting and GEPA prompt optimisation move the operating point without creating usable detection. Restructuring the task recovers it: list the facts the transcript establishes, then check the note for each. Two methods reach it independently and trade off: a per-fact pipeline, and a GEPA-evolved prompt doing the same in one call. The pipeline's flags name the missing fact and its severity at 2.7% false alarms. The single call detects more (36.9% against 24.6%, p=0.002) at 6.2% false alarms and a tenth of the cost per note. A physician author validated 70 items and, where the two routes disagree, sided with the pipeline on 10 of 10 (p=0.002). A second clinician, not an author, graded the severity rubric blind and agrees to within a grade. On real vendor notes from a companion census no benchmark threshold transfers, but the re-calibrated single call detects more than the best of the eight at half its false-alarm rate. Omissions whose fact is restated elsewhere defeat both routes. We release the benchmark, prompts and judgements.