MIMIC-Note-Extract: A Large-Scale Structured Dataset Extracted from MIMIC-IV-Note Discharge Summaries
Authors: Kristian Schwethelm, Jonas Kuntzer, Conrad Ketzer, Eva Wende, Moritz Schlager, Daniel Rueckert, Georgios Kaissis, Friederike Jungmann, Paul Hager
Organizations: Chair for AI in Healthcare and Medicine, TU Munich (TUM) and TUM University Hospital, Germany · Department for Trauma Surgery, TUM University Hospital and TU Munich, Germany · Department for Diagnostic and Interventional Radiology, TUM University Hospital and TU Munich, Germany · Chair for AI in Healthcare and Medicine, TU Munich and TUM University Hospital, Germany · Munich Center for Machine Learning (MCML), Germany · Department of Computing, Imperial College London, UK · Hasso Plattner Institute for Digital Engineering, University of Potsdam, Germany
MIMIC is one of the most widely used public hospital datasets for machine learning on electronic health records, yet clinically important information, such as past medical history, allergies, and diagnostic findings, remains confined to free-text discharge summaries. Studies requiring this information must therefore build their own extraction pipelines, duplicating substantial effort across projects. We release MIMIC-Note-Extract (MNE), a large-scale dataset that uses a hybrid extraction pipeline combining deterministic parsing with schema-guided large language model generation to transform 16 sections from 300,720 discharge summaries into structured tables. MNE adds information that has no structured MIMIC counterpart and enriches related tables with broader coverage and greater clinical detail, including 10 times more patient-history entries and medication histories for more than twice as many admissions. Validation by medical doctors confirms extraction accuracy above 92% across sections. Using MNE, we construct a clinical decision-making benchmark for LLM agents, demonstrating how reusable structured representations of clinical narratives reduce data curation effort.
Figures & tables
Figure 1: A raw discharge summary flowing through our extraction pipeline, showing a representative subset of four sections. The example is synthetic and does not correspond to any real admission. The discharge summary (left) is split into sections (middle) and decomposed into relational rows (right). Badges denote the extraction method of each section (rule-based, hybrid, or LLM).
Clinical Section
Key Fields
Method
Admissions
Coverage
Gender
Patient sex assigned at birth
Rule
300,720
100.0%
Services
Clinical service at discharge
Rule
300,720
100.0%
Chief Complaint
Symptom category and clinical description
Hybrid
296,132
98.5%
History of Present Illness
Chronological narrative of events prior to admission
Verbatim
296,620
98.6%
Past Medical History
Prior conditions and interventions by category, chronicity, and subject
LLM
290,212
96.5%
Medications on Admission
Active home medications with dose, route, frequency, and strength
LLM
275,035
91.5%
Table 1: Clinical sections, key concepts, extraction methodologies, and cohort-wide coverage in MNE across 300,720 admissions. Coverage is the percentage of admissions with a non-empty section.
Section
Acc. (%)
AC1
Allergies
99.8 ± 0.32
99.8
Chief Complaint
97.6 ± 2.37
98.0
Discharge Condition
97.4 ± 3.15
98.0
Discharge Diagnosis
97.1 ± 2.65
95.8
Major Procedures
98.3 ± 2.04
98.0
Past Medical History
95.2 ± 1.48
90.1
Table 2: Extraction accuracy (item-level, 95% confidence margins of error) and inter-annotator agreement (Gwet’s AC1) per section.
Section
Contribution
Already exists in MIMIC, but can flag mismatches
Gender
Flag mismatches
Services
Flag mismatches
Discharge Disposition
Flag mismatches; adds ∼44 k missing values
Related to MIMIC information, but broader or richer
Discharge Diagnosis
Clinician-written diagnoses with primary/secondary structure
Table 3: Relationship of extracted sections to MIMIC’s existing tabular data.
Model
Reasoning
Full info
Agentic
Gemini 3.5 Flash
minimal
88.7 (+0.1)
87.9
Gemini 3.1 Flash Lite
minimal
87.4 (+0.7)
81.9
Gemini 3.1 Flash Lite
high
88.6 (+0.0)
84.2
Gemma 4 31B
×
86.7 (+0.3)
83.7
Qwen3.6 35B-A3B
×
90.4 (+0.3)
83.6
Qwen3.6 35B-A3B
✓
89.3 (+0.2)
82.6
Table 4: Diagnosis accuracy (%) on the full-information and agentic CDM benchmarks. Parentheses report how accuracy changes when using our data instead of the original regex-derived data. Appendix Table 13 reports 95% confidence intervals.
Figure 2: Overview of the agentic CDM benchmark built from MIMIC-Note-Extract . The LLM agent starts from the patient presentation and gathers evidence through five tools before making a diagnosis. Green tools are sourced from MNE, blue tools from existing MIMIC data, and hatched from both. In contrast to prior benchmarks, the agent must name the exact item it wants, e.g., a single examination system or a patient-record section, and receives only this requested part.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Table
Cardinality
Key fields
Hub
admissions
1
subject_id , hadm_id , note_id
Single-row attributes
gender
0–1
enum
services
0–1
enum (20 services)
discharge_disposition
0–1
enum
Appendix
Table 5: All 16 tables in the MIMIC-Note-Extract schema. “Cardinality” refers to rows per admission.
Clinical Section
Operational Mode
Production Model
LLM Baseline (€)
Production Cost (€)
Optimized Savings (€)
LLM-Only Extraction
Past Medical History
LLM
Gemini 3.1 Flash-Lite
138.07
138.07
—
Medications on Admission
LLM
Gemini 3.1 Flash-Lite
487.00
487.00
—
Physical Examination
LLM
Gemini 3.1 Flash-Lite
345.00
345.00
—
Pertinent Results
LLM
Gemini 3.1 Flash-Lite
179.00
179.00
—
Discharge Medications
LLM
Gemini 3.1 Flash-Lite
581.00
581.00
—
Appendix
Table 6: Production extraction methodology, LLM model, API expenditures, and rule-first optimization savings by clinical section.
Category / section
Source excerpt
Extracted fields
Error explanation
Ambiguous Medications on admission
Simethicone 40-80 mg PO QID:PRN abd pain
strength : range 40–80 mg dose : 80 mg
The source gives a 40–80 mg range without separately specifying strength and administered dose. The extraction records both a strength range and a single 80 mg dose.
Fabrication Past medical history
R THA / L THA / Stents x2
concept_name : Coronary Artery Stents
The extraction adds a coronary location to the stents, although their anatomical location is not specified in the source.
furosemide 40 mg Tablet ___: One (1) Tablet PO ONCE (Once) for 1 doses .
name : furosemide frequency : 1 per DAY duration : null
A one-time dose is incorrectly represented as daily administration without a duration limit.
Incorrect boundaries Pertinent results
19: collection. 20: Findings were concerning for early or partial small bowel 21: obstruction. The NG tube was kept in place, and medications were
start_line : 6 end_line : 19 Clinician span: 6–21
The extracted span ends at line 19 instead of line 21, omitting the following finding of suspected small-bowel obstruction.
Appendix
Table 7: Examples of clinician-assigned error and ambiguity labels. Bold text highlights the source phrases and extracted values relevant to each judgment; field names are set in monospace. Source excerpts retain the original wording, with line breaks represented by spaces or slashes except in the boundary example; extracted fields are abbreviated for display. Boundary line numbers are zero-based and inclusive.
Section
Judged Items
Omiss.
Attr.
Alter.
Fabr.
Bound.
Ambig.
Allergies
1,488
0 (0.0%)
0 (0.0%)
1 (0.1%)
2 (0.1%)
0 (0.0%)
6 (0.4%)
Chief Complaint
222
1 (0.5%)
4 (1.8%)
0 (0.0%)
0 (0.0%)
0 (0.0%)
3 (1.4%)
Discharge Condition
222
4 (1.8%)
0 (0.0%)
0 (0.0%)
1 (0.5%)
0 (0.0%)
1 (0.5%)
Discharge Diagnosis
222
1 (0.5%)
5 (2.3%)
0 (0.0%)
0 (0.0%)
0 (0.0%)
0 (0.0%)
Major Procedures
222
1 (0.5%)
1 (0.5%)
2 (0.9%)
0 (0.0%)
0 (0.0%)
0 (0.0%)
Past Medical History
2,761
66 (2.4%)
46 (1.7%)
26 (0.9%)
3 (0.1%)
0 (0.0%)
36 (1.3%)
Appendix
Table 8: Counts and percentages of extraction errors and ambiguous judgments by clinical section. Percentages are calculated relative to all judged items within each section.
Ambiguity-excluded Case-level (%)
Section
Acc.
Prec.
Rec.
F1
Allergies
98.5 ± 2.06
99.6
100.0
99.8
Chief Complaint
97.6 ± 2.37
97.9
97.8
97.8
Discharge Condition
97.9 ± 2.35
97.9
97.9
97.9
Discharge Diagnosis
97.1 ± 2.63
97.7
97.4
97.5
Major Procedures
98.2 ± 2.05
98.5
98.4
98.4
Appendix
Table 9: Model extraction performance by clinical section at the case level (with 95% confidence margins of error).
Ambiguity-excluded Item-level (%)
Section
Acc.
Prec.
Rec.
F1
Allergies
99.8 ± 0.32
99.8
100.0
99.9
Chief Complaint
97.6 ± 2.37
97.9
99.7
98.8
Discharge Condition
97.4 ± 3.15
99.4
97.9
98.7
Discharge Diagnosis
97.1 ± 2.65
97.7
99.4
98.5
Major Procedures
98.3 ± 2.04
98.5
99.7
99.1
Appendix
Table 10: Model extraction performance by clinical section at the item level (with 95% confidence margins of error).
Section
Case IAA (tolerant)
Item IAA (tolerant)
AC1 (tolerant)
Allergies
98.0
99.8
99.8
Chief Complaint
96.0
98.0
98.0
Discharge Condition
98.0
98.0
98.0
Discharge Diagnosis
96.0
96.0
95.8
Major Procedures
96.0
98.0
98.0
Past Medical History
56.0
90.4
90.1
Appendix
Table 11: Inter-Annotator Agreement (Raw IAA and Gwet’s AC1) by clinical section.
Section
Related MIMIC table(s)
Contribution
Already exists in MIMIC, but can flag mismatches
Gender
h.patients.gender
Flag mismatches
Services
h.services.curr_service
Flag mismatches
Discharge Disposition
h.admissions.discharge_location
Flag mismatches; adds ∼44 k missing values
Related to MIMIC information, but broader or richer
Discharge Diagnosis
h.diagnoses_icd
Concise clinician-written diagnoses with primary/secondary structure, preserving note wording instead of dense, generic ICD codes
Appendix
Table 12: Extracted sections grouped by how they relate to MIMIC’s existing tables, with the closest related table(s) and the contribution of our extraction. Table prefixes: h=hosp, e=ED, i=ICU.
Model
Reasoning
Full information
Agentic
Gemini 3.5 Flash
minimal
88.7 [87.4, 89.9]
87.9 [87.2, 88.5]
Gemini 3.1 Flash Lite
minimal
87.4 [86.1, 88.7]
81.9 [81.2, 82.7]
Gemini 3.1 Flash Lite
high
88.6 [87.3, 89.8]
84.2 [83.5, 84.9]
Gemma 4 31B
×
86.7 [85.3, 88.0]
83.7 [82.9, 84.4]
Qwen3.6 35B-A3B
×
90.4 [89.1, 91.5]
83.6 [82.9, 84.3]
Qwen3.6 35B-A3B
✓
89.3 [88.0, 90.5]
82.6 [81.8, 83.3]
Appendix
Table 13: Diagnosis accuracy (%) on the full-information and agentic CDM benchmarks, with 95% Wilson confidence intervals in brackets.
Class
Cases
Class
Cases
Heart failure exacerbation
731
Acute cholecystitis
646
Gastrointestinal bleeding
731
Appendicitis
566
Cellulitis
731
Pancreatitis
434
Small bowel obstruction
731
Cholangitis
346
Pneumonia
731
Diverticulitis
219
Urinary tract infection
731
Diabetic ketoacidosis
172
Appendix
Table 14: The 19 benchmark classes and the number of cases drawn.
Tool
Source
Requestable items
Physical examination
MNE
Six examination groups: vital signs, general, cardiopulmonary, abdominal, neurologic, and skin/extremities.
Patient record
MNE
Eight sections: history of present illness, conditions, interventions, substance history, functional status, family history, allergies, and admission medications.
Laboratory
MIMIC
Panels (CBC, basic metabolic, liver function, coagulation, urinalysis, blood gas, urine electrolytes), body-fluid analyses (CSF, ascitic, synovial, pleural), or individual tests from a catalog.
Imaging
MIMIC + MNE
A region (abdomen, chest, head, neck, spine, extremity, venous) paired with a modality (CT, MRI, ultrasound, radiograph, nuclear medicine, fluoroscopy, ERCP).
Clinical study
MNE
One study: echocardiogram, cardiac catheterization, electrocardiogram, stress test, upper endoscopy, or colonoscopy.
Appendix
Table 15: The five tools exposed to the agent. Three are backed by MIMIC-Note-Extract ( MNE ); laboratory and imaging reuse existing MIMIC tables, with imaging additionally drawing on extracted pertinent-result imaging.
Calls per case
Requests (%)
Model
Reasoning
Mean
Median
Malformed
Excessive
Gemini 3.5 Flash
minimal
6.95
7
2.01
1.85
Gemini 3.1 Flash Lite
minimal
7.21
7
1.89
10.16
Gemini 3.1 Flash Lite
high
3.73
4
0.92
0.39
Gemma 4 31B
×
5.54
5
0.56
0.59
Qwen3.6 35B-A3B
×
3.61
4
2.75
0.72
Appendix
Table 16: Mean and median attempted tool calls per completed case, and percentage of calls with malformed or excessive requests. Excessive requests include repetitions and calls denied by request limits.
Figure 3: Fraction of cases in which Gemini 3.5 Flash queries each tool at least once, by pathology. The patient record and physical examination are queried almost universally, whereas imaging and the clinical-study (electrocardiogram, endoscopy) tool are requested selectively.
Model
Reasoning
Appendicitis ( n=345 )
Acute cholecystitis ( n=512 )
Diverticulitis ( n=216 )
Pancreatitis ( n=342 )
Gemini 3.5 Flash
minimal
99.1
98.8
99.1
94.4
Gemini 3.1 Flash Lite
minimal
94.5
97.7
94.9
78.7
Gemini 3.1 Flash Lite
high
90.4
84.8
83.3
47.1
Gemma 4 31B
×
89.6
87.7
81.9
57.9
Qwen3.6 35B-A3B
×
88.4
80.3
76.4
52.3
Qwen3.6 35B-A3B
✓
63.8
69.5
57.9
19.9
Appendix
Table 17: Percentage of cases in which at least one laboratory result was obtained from every disease-specific category. Only cases completed by all model configurations and with results available from every category were included; counts are shown in the headings.
Large language models are increasingly used for structured extraction from clinical free-text notes, but the sensitivity of their output to upstream configuration choices is less understood than their accuracy on fixed benchmarks. This work measures that sensitivity without human-annotated ground truth, by holding the extraction task fixed and varying one choice at a time. The fixed schema comprises 17 clinical documentation flags on a three-way yes/no/not_documented value set and a 47-tag vocabulary for the primary admission reason. Three prompt variants expressing this schema were each run at two model sizes on MIMIC-IV v3.1 discharge summaries. Cross-prompt agreement was measured by Cohen's kappa on ICD-stratified subsets. A paired same-note comparison isolated the effect of model choice, and a post-hoc collapse of the three-way flags to binary tested the schema's contribution to disagreement. On the three-way flags, the two models reach the same pooled cross-prompt agreement (median kappa 0.69 and 0.68); the larger model raises agreement on some fields and lowers it on others, a redistribution rather than the absence of an effect. Collapsing the schema to binary dissolves most of the cross-prompt disagreement, locating it on the absence-versus-silence distinction rather than on whether the finding is present. On the multi-class admission categorization, changing the model reassigns the dominant tag on close to half of all notes while changing the prompt phrasing reassigns it on roughly one in eight, and the larger model places far less mass on residual catch-all categories (44% to 26%). These patterns indicate a schema-imposed source of disagreement concentrated on the absence-versus-silence axis and a dominance of model over prompt phrasing on multi-class categorization, identified by a reusable methodology for auditing extraction reproducibility on a population-scale deployment.
Discharge summaries are crucial clinical documents containing the context of a patient's overall hospital stay, and are routinely reviewed by medical experts for patient readmission, ongoing care, and diagnostic decision-making. When reviewing them, medical experts often must iteratively synthesize information across multiple discharge summaries while verifying the evidence supporting each answer. Although large language models (LLMs) are increasingly explored for clinical question answering, existing benchmarks do not sufficiently reflect this setting: they often evaluate exam-style medical knowledge or focus on single-turn question answering with limited evidence-grounding evaluation. We introduce EHRNote-ChatQA, the first benchmark for evidence-grounded multi-turn clinical question answering over patients' multiple discharge summaries. Built from de-identified MIMIC-IV discharge summaries, EHRNote-ChatQA contains 967 patient-level multi-turn samples spanning one to five notes and 16,072 medical-expert-verified QA pairs (8,036 content questions, each paired with an evidence-grounding question) across eight clinical categories. The benchmark is constructed through an expert-informed pipeline combining discharge-summary structuring schema, expert-curated multi-turn QA templates, and LLM-based generation, followed by review and revision of every single QA sample by 12 medical experts (11 physicians and 1 nurse). Benchmarking 24 proprietary and open-weight LLMs reveals several challenges, including that LLMs struggle more with evidence grounding than content answering, multi-turn errors compound across turns, and single-turn clinical QA performance does not reliably transfer to this setting. These findings establish EHRNote-ChatQA as a rigorous and practical benchmark for evaluating clinical QA systems.
Jiyoun Kim, Muhan Yeo, Eunhye Jang +14
KAIST · Seoul National University · Seoul National University Bundang Hospital +9
Medical concept extraction from electronic health records underpins many downstream applications, yet remains challenging because medically meaningful concepts are frequently implied rather than explicitly stated in medical narratives. Existing benchmarks with human-annotated evidence spans underscore the importance of grounding extracted concepts in medical text. However, they predominantly focus on explicitly stated concepts instead of implicit concepts. We present MedicalBench, a benchmark for medical concept extraction with evidence grounding that evaluates implicit medical reasoning. MedicalBench formulates medical concept extraction as a verification task over medical note-concept pairs, coupled with sentence-level evidence identification. Built from MIMIC-IV discharge summaries and human-verified ICD-10 codes, the dataset is curated through a multi-stage large language model (LLM) triage pipeline followed by medical annotation and expert review. It deliberately includes implicit positives, semantically confusable negatives, and cases where LLM judgments disagree with medical expert assessments. We define two complementary evaluation tasks: (1) medical concept extraction and (2) sentence-level evidence retrieval, enabling assessment of both correctness and interpretability. Benchmarking state-of-the-art LLMs reveals that performance remains modest, highlighting the difficulty of extracting implicitly expressed concepts. We further show that performance is largely invariant to note length, indicating that MedicalBench isolates reasoning difficulty rather than superficial confounders. MedicalBench provides the first systematic benchmark for implicit, evidence-grounded medical concept extraction, offering a foundation for developing medical language models that can both identify medically relevant concepts and justify their predictions in a transparent and medically faithful manner.
Zhichao Yang, Gregory D. Lyng, Sanjit Singh Batra +1