M4Bench: Evaluating Procedural Specification for Clinical EHR Derivation Agents
Authors: Hannes Ill, Rafi Al Attrach, Rajna Fani, Ahram Han, Ryohei Yamamoto, Marlene Louisa Moerig, Dukyong Yoon, Leo Anthony Celi
Organizations: Laboratory for Computational Physiology, Massachusetts Institute of Technology, USA · Department of Surgery, Seoul National University Hospital, Korea · Charité, University Medicine Berlin, Germany · Department of Biomedical Systems Informatics, Yonsei University College of Medicine, Korea · Division of Pulmonary, Critical Care and Sleep Medicine, Beth Israel Deaconess Medical Center, USA
Clinical research over electronic health records depends on derived variables whose definitions sit outside the schema, in item codes, time windows, missingness rules, and output-grain choices that change cohorts and downstream conclusions. We introduce M4Bench, 28 clinical derivation tasks across 15 task families on MIMIC-IV and eICU, to test whether AI coding agents can reproduce such derivations and how clinician-reviewed agent skills change their results. Its controls decompose the value of skills into specification completeness, delivery format, and schema familiarity. In a Docker-isolated Codex campaign (GPT 5.4 Mini and GPT 5.5), skills raise reward by 0.136 over the no-skill baseline, with 26 of 28 tasks improving. Specification completeness accounts for most of this gain: on an eight-task control subset, an operational spec alone raises reward by 0.162, and skills add only 0.024 on top, indistinguishable from zero. On a ten-task subset, decoy skills score like no skill and raw reference SQL nearly matches targeted skills, arguing against generic context or the skill format as the explanation. Schema obfuscation and restructuring preserve 86% and 70% of the skill gain (GPT 5.4 Mini, 26 MIMIC-IV tasks), a descriptive robustness check rather than a bound on memorization. A descriptive Claude Code replication on all 28 tasks (Opus 4.7 and Sonnet 4.6, 560 runs) is directionally consistent (+0.120, 21 of 28 tasks improving). These gains are partial agreement with the reference, not completed tasks: only 25 of 280 with-skill and 13 of 280 no-skill Codex runs pass every strict check, and skills barely change which rows agents return (key F1 +0.003). Skills are released as a versioned, clinician-reviewable markdown delivery format, with a family-level reference-validity audit. M4Bench releases tasks, skills, evaluation code, and sanitized per-run evidence as an open-source benchmark.
Figures & tables
Figure 1: M4Bench evaluation pipeline. Each task pairs a task card with a DuckDB clone whose reference targets are removed. The primary no-skill – with-skill comparison and matched controls test specification completeness, reference-code structure, and schema familiarity under sandboxed execution and contamination auditing.
Term
What the agent receives
Purpose
no-skill
Task instruction and task database
Schema-plus-task baseline; no curated derivation procedure
with-skill
no-skill inputs plus a task-specific procedural skill
Primary test of targeted procedural specification
Decoy skill
no-skill inputs plus a task-mismatched skill artifact
Controls for generic extra context and skill-file presence
Raw reference SQL
no-skill inputs plus reference SQL text
Tests reference-code structure; raw-task cells require SQL adaptation rather than answer copying
Operational spec
no-skill inputs plus complete non-SQL clinical rules, excluding item identifiers and dataset recipes
Separates rule completeness from dataset navigation and skill packaging
Schema perturbations
no-skill or with-skill inputs on obfuscated or restructured schemas
Probes schema-name and table-organization familiarity
Table 1: Glossary of benchmark conditions, controls, and metrics.
Family
Tasks
Standard
Raw
Database
APS III
2
1
1
MIMIC-IV
Charlson
1
0
1
MIMIC-IV
Baseline creatinine
2
1
1
MIMIC-IV
GCS
2
0
2
MIMIC-IV/eICU
KDIGO AKI
2
1
1
MIMIC-IV
MELD
2
1
1
MIMIC-IV
Table 2: M4Bench task inventory. Standard tasks retain selected intermediate features; raw tasks require reconstruction from source tables.
Measure
No skill
With skill
Delta
Improved tasks
Reward / GPT 5.4 Mini
0.639
0.802
0.162 [0.099, 0.232]
26/28
Reward / GPT 5.5
0.767
0.876
0.109 [0.060, 0.166]
24/28
Reward / task-balanced
0.703
0.839
0.136 [0.085, 0.191]
26/28
Key F1 / task-balanced
0.962
0.965
0.003 [-0.010, 0.018]
5/28
Ordered nMAE / task-balanced
0.116
0.060
-0.056 [-0.085, -0.029]
23/26
Table 3: Native-schema Codex skill effect and diagnostics. Reward is mean value-column match rate; key F1 measures row grain; ordered nMAE is lower-is-better. Intervals are 95% task-bootstrap intervals.
Comparison
Tasks
No skill
Operational spec
Targeted skill
Delta
Operational spec–no skill
8
0.606
0.767
–
0.162 [0.072, 0.260]
Targeted skill–operational spec
8
–
0.767
0.792
0.024 [-0.034, 0.099]
Table 4: Codex operational-spec follow-up on the eight-task control subset. Values are task-balanced after averaging the two Codex models within each task.
Task set
Tasks
Runs
Delta [95% CI]
Positive
Strict passes (no/with)
Key F1 delta
All tasks
28
560
0.120 [0.067, 0.177]
21/28
25/280 vs 37/280
0.008
CLI-version matched
20
400
0.113 [0.050, 0.185]
15/20
19/200 vs 27/200
0.008
CLI and routing matched
19
380
0.119 [0.054, 0.192]
14/19
19/190 vs 27/190
0.008
Table 5: Full-coverage Claude replication: task-balanced native-schema with-skill minus no-skill reward on all 28 tasks, on the 20 tasks whose no-skill and with-skill cells share one recorded Claude Code version (CLI) per model, and on the 19 that also share one routing profile. Intervals are 95% task bootstraps. Strict passes and key F1 are separate from reward.
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Mode
Database
Key grain
Scored columns
Reference rows
eicu-gcs
raw
eICU
stay
4
200859
eicu-oasis
raw
eICU
stay
11
200859
mimic-apsiii-24h
standard
MIMIC-IV
stay
17
94458
mimic-apsiii-24h-raw
raw
MIMIC-IV
stay
17
94458
mimic-charlson-raw
raw
MIMIC-IV
admission
19
546028
mimic-creatinine-baseline
standard
MIMIC-IV
admission
3
546028
Appendix
Table 6: Per-task evaluation details. Reference rows are reference-output rows before task-specific evaluation joins.
Task
Reviewers
Source basis
eicu-gcs
C1/C4
eICU concept conventions
eicu-oasis
C2/C4
eICU concept conventions
mimic-apsiii-24h
C1/C4
MIT-LCP concept conventions
mimic-apsiii-24h-raw
C1/C4
MIT-LCP concept conventions
mimic-charlson-raw
C1/C4
MIT-LCP concept conventions
mimic-creatinine-baseline
C1/C4
Public clinical definitions
Appendix
Table 7: Task-level clinical review provenance. Each task is reviewed by two clinicians, identified as C1–C4, and every review covered the task definition, schema, and output. Reviewer identifiers are described in Table 8 .
Reviewer ID
Description
C1
MD, MS; board-certified vascular and trauma surgeon; associate professor at a tertiary academic center; clinical registry and administrative-data researcher; MIMIC-IV/eICU experience.
C2
MD, PhD; board-certified intensivist and emergency physician; clinical epidemiologist; ICU EHR researcher; MIMIC-IV/eICU experience.
C3
MD; doctoral candidate and clinical AI researcher; expertise in explainable AI, ML-based clinical decision support, and multimodal prediction models; MIMIC-IV/eICU experience.
C4
MD, PhD; associate professor at a medical school; physician and clinical AI researcher; expertise in ICU/EHR data research, multimodal clinical foundation models, and clinical AI evaluation; MIMIC-IV research experience.
Antibiotic–microbiology timing rule from public concept code
Broad culture inclusion can overcall suspected infection; downstream Sepsis-3 inherits this target.
SOFA / Sepsis-3
Missing organ components default to zero
Agreement may reflect the public-code missingness convention rather than adjudicated organ failure.
APS III
MIMIC-Code component implementation
Known respiratory and oxygenation pathway discrepancies are retained for reproducibility.
Vasopressor equivalents
Five-agent norepinephrine-equivalent mapping
Angiotensin II administrations are not scored in the current reference.
eICU OASIS
eICU APACHE abstraction tables where source charting is not equivalent
eICU raw tasks are not schema-symmetric with MIMIC raw reconstruction.
Appendix
Table 9: Reference-validity audit for selected task families. These caveats describe the declared benchmark target, not new exclusions from scoring.
Codex primary
Claude sentinel
Canonical runs
760
190
Provenance-only dirs
40
0
Retained dirs
800
190
result.json
800
190
output.csv
800
190
instruction.md
800
190
Appendix
Table 10: Retained paper-facing artifact directories after rerun replacement. Retained size is the uncompressed sum of listed files, not credentialed task databases. Codex has 760 canonical analysis runs plus 40 provenance-only directories.
Codex primary
Claude sentinel
Canonical runs
760
190
Provenance-only dirs
40
0
Retained dirs
800
190
Hardware
MacBook Pro (M3 Pro, 18 GB)
MacBook Pro (M3 Pro, 18 GB)
Agent wall-clock h
56.4
17.5
Uncached input
586,293,569
16,657
Appendix
Table 11: Compute and storage for the retained evidence package. Agent wall-clock sums the per-run agent execution time recorded in result.json . Selected dirs are retained paper-facing run directories on disk. Codex and Claude token rows reflect each provider’s native usage reporting (Codex: total prompt tokens; Claude: split into uncached vs. cached input under provider prompt caching), so uncached counts are not directly comparable across arms.
Asset
Role
Access, license, or terms
M4Bench code, tasks, evaluator, skills
Benchmark artifact
MIT license (repository LICENSE); camera-ready source release tagged m4bench-neurips2026-camera-ready.
MIMIC-IV v3.1
Source EHR data for 26 tasks
PhysioNet credentialed access and MIMIC-IV data-use terms ( https://physionet.org/content/mimiciv/3.1/ ); derived databases are not openly redistributed.
eICU Collaborative Research Database v2.0
Source EHR data for 2 tasks
PhysioNet credentialed access and eICU data-use terms ( https://physionet.org/content/eicu-crd/2.0/ ); derived databases are not openly redistributed.
MIT-LCP MIMIC-Code / eICU concept conventions
Reference SQL lineage and procedural conventions
Credited and cited; adapted SQL/procedural conventions retained with provenance.
Open packages withhold row-level agent outputs and tool outputs; originals stay private under source-data use terms.
Appendix
Table 12: Source assets, release terms, and access restrictions.
Task
Mode
Database
Delta
Decoy/raw SQL
Op. spec
mimic-urine-output-rate-raw
raw
MIMIC-IV
0.471
yes
yes
mimic-urine-output-rate
standard
MIMIC-IV
0.460
yes
mimic-meld-24h
standard
MIMIC-IV
0.354
yes
mimic-oasis-24h
standard
MIMIC-IV
0.296
yes
mimic-suspicion-infection-raw
raw
MIMIC-IV
0.291
yes
mimic-meld-24h-raw
raw
MIMIC-IV
0.283
yes
Appendix
Table 13: Control-subset membership and final Codex targeted-skill delta (task-balanced over both models) for all 28 tasks. Each task’s family follows from its name, and families are listed in Table 2 .
Archive
Contents
Runs
Bytes
m4bench-evidence- submission.tar.zst
Submitted campaigns (Codex primary, Codex follow-up, Claude sentinel, OSS)
Table 14: Open evidence packages, hosted at https://huggingface.co/datasets/hannesill/m4bench-evidence and attached to the tagged release.
Audit quantity
Count
Runs
760
Expected primary/supplementary runs
760
Task/condition/schema/model cells
152
Publishable flag failures
0
Isolation flag failures
0
Filesystem canary failures
0
Appendix
Table 15: Primary Codex campaign integrity checks after replacing matching May 2 runs with the May 4 clinician-review reruns. Publishability means the run is eligible for analysis. It does not mean the submitted derivation is clinically correct.
Figure 2: Per-task native-schema skill delta (with-skill minus no-skill mean reward) for the primary Codex campaign and the full-coverage Claude replication on all 28 tasks, and for the exploratory gpt-oss-20b arm on the eight tasks it covers.
Task
Mean delta
mimic-urine-output-rate-raw
0.471
mimic-urine-output-rate
0.460
mimic-meld-24h
0.354
mimic-oasis-24h
0.296
mimic-suspicion-infection-raw
0.291
mimic-meld-24h-raw
0.283
Appendix
Table 16: Largest native-schema skill gains after averaging task-level deltas across the Codex models.
Task
Mean delta
mimic-kdigo-48h-raw
-0.040
mimic-kdigo-48h
-0.002
mimic-charlson-raw
0.001
mimic-vasopressor-equivalents
0.006
mimic-sofa-24h
0.010
mimic-sirs-24h-raw
0.016
Appendix
Table 17: Largest negative or flat native-schema skill effects after averaging task-level deltas across the Codex models.
Model
Mode
Tasks
Mean delta
Positive tasks
GPT 5.4 Mini
standard
11
0.203
10/11
GPT 5.4 Mini
raw
17
0.136
16/17
GPT 5.5
standard
11
0.105
9/11
GPT 5.5
raw
17
0.112
15/17
Task-balanced mean
standard
11
0.154
10/11
Task-balanced mean
raw
17
0.124
16/17
Appendix
Table 18: Native-schema Codex skill effect split by task mode. Values are task-paired deltas. Final rows average the two Codex models within task before aggregating.
Analysis
Units
Mean delta
95% interval
Task-cluster bootstrap
28 tasks
0.136
[0.085, 0.191]
Family-cluster bootstrap
15 families
0.136
[0.065, 0.207]
Family-balanced bootstrap
15 families
0.120
[0.057, 0.190]
Appendix
Table 19: Cluster sensitivity for the primary native-schema Codex skill effect. Task rows are the paper headline. Family rows resample or balance the 15 task families to check sensitivity to standard/raw sibling tasks.
Condition or worst cell
N
Key F1
Key precision
Extra keys
no-skill
56
0.962
0.967
2384458
with-skill
56
0.965
0.976
2087534
mimic-ventilation-raw / no-skill / GPT 5.4 Mini
5
0.700
0.559
555030
mimic-ventilation-raw / no-skill / GPT 5.5
5
0.717
0.573
525311
mimic-ventilation-raw / with-skill / GPT 5.4 Mini
5
0.720
0.583
505933
mimic-ventilation-raw / with-skill / GPT 5.5
5
0.740
0.603
459225
Appendix
Table 20: Native-schema key diagnostics for Codex. The first two rows summarize native no-skill and with-skill cells. Subsequent rows show high-extra-key cells. N counts cells in the first two rows and runs in the others. Key F1 and key precision are means.
Task
Reward delta
Key F1 delta
nMAE delta
Ordered cols
eicu-gcs
0.054
0.000
-0.029
4
eicu-oasis
0.272
0.000
-0.189
11
mimic-apsiii-24h
0.023
0.000
-0.013
17
mimic-apsiii-24h-raw
0.055
0.000
-0.036
17
mimic-charlson-raw
0.001
0.000
-0.001
19
mimic-creatinine-baseline
0.142
-0.073
0.015
3
Appendix
Table 21: Per-task secondary diagnostics for native-schema Codex. nMAE is computed only for ordered value columns after matching on declared keys. Negative nMAE deltas are improvements.
Control
Tasks
No skill
Targeted
Control
Control–targeted
Control–no skill
Raw reference SQL
10
0.553
0.833
0.837
0.003
0.283
Decoy skill
10
0.553
0.833
0.554
-0.280
0.000
Appendix
Table 22: Matched-context controls on selected tasks in the final combined Codex analysis. “Control–targeted” compares the control condition to the targeted with-skill condition after task-level averaging.
Schema
Tasks
No skill
With skill
Skill delta
Positive tasks
Robustness ratio
native
26
0.637
0.794
0.157 [0.092, 0.229]
24/26
1.000
obfuscated
26
0.561
0.696
0.135 [0.068, 0.208]
19/26
0.862
restructured
26
0.557
0.668
0.110 [0.057, 0.170]
23/26
0.704
Appendix
Table 23: Codex schema-skill generalization follow-up for GPT 5.4 Mini over 26 MIMIC tasks. Transformed rows match May 6 with-skill runs to the earlier transformed-schema no-skill probe.
Audit quantity
Count
Raw result directories
363
Canonical run keys
340
Superseded duplicate directories
23
Operational-spec canonical runs
80
Transformed-schema with-skill canonical runs
260
Publishable flag failures
0
Appendix
Table 24: May 6 Codex validity follow-up integrity. Superseded retry directories are retained in the manifest for audit but excluded from analysis tables.
Task
No skill
Operational spec
Targeted skill
Spec–no
Targeted–spec
mimic-urine-output-rate-raw
0.372
0.802
0.843
0.430
0.041
mimic-ventilation
0.536
0.528
0.782
-0.008
0.254
mimic-creatinine-baseline-raw
0.620
0.794
0.800
0.174
0.006
mimic-suspicion-infection
0.720
0.932
0.820
0.212
-0.112
mimic-vasopressor-equivalents-raw
0.733
0.842
0.856
0.109
0.014
mimic-oasis-24h
0.699
0.987
0.995
0.288
0.008
Appendix
Table 25: Operational-spec follow-up by sentinel task after averaging the two Codex models within each task.
Model
No skill
With skill
Task-paired delta
Positive tasks
Claude Opus
0.738
0.843
0.106
6/8
Claude Sonnet
0.592
0.817
0.226
8/8
Task-balanced mean
–
–
0.166 [0.085, 0.259]
8/8
Appendix
Table 26: Historical (submitted) Claude sentinel native-schema skill effect across eight tasks.
Task
Opus delta
Sonnet delta
Mean delta
urine-output rate raw
0.391
0.484
0.438
baseline creatinine raw
0.160
0.298
0.229
ventilation
0.159
0.263
0.211
OASIS-24h
0.103
0.214
0.158
suspected infection
0.043
0.242
0.143
vasopressor equivalents raw
-0.021
0.219
0.099
Appendix
Table 27: Historical (submitted) Claude sentinel per-task native-schema skill deltas.
Model
No skill
With skill
Task-paired delta
Positive tasks
gpt-oss-20b local
0.085
0.094
0.008 [-0.072, 0.088]
3/8
Appendix
Table 28: Exploratory local OSS native-schema skill effect. This arm is excluded from primary and supplementary paper-facing evidence because local Ollama runs do not meet the release publishability criteria.
Model
No skill
With skill
Task-paired delta
Positive tasks
Claude Opus
0.800
0.883
0.083 [0.038, 0.135]
25/28
Claude Sonnet
0.694
0.851
0.156 [0.087, 0.231]
20/28
Task-balanced mean
0.747
0.867
0.120 [0.067, 0.177]
21/28
Appendix
Table 29: Full-coverage Claude replication: model-specific and task-balanced native-schema with-skill minus no-skill reward differences over 28 tasks (95% task-bootstrap intervals).
Condition
Runs
CLI 2.1.119
CLI 2.1.220
Overrides recorded
Haiku usage
Haiku turns
Agent hours
no-skill
280
167
113
120
160
4
28.6
with-skill
280
100
180
110
170
1
22.6
Appendix
Table 30: Recorded harness indicators of the full-coverage Claude replication by condition: observed Claude Code version, recorded model-slot overrides, runs whose usage reports Haiku, runs with Haiku assistant turns, and summed agent wall-clock hours. Hardware was not recorded per run.
Task
Opus delta
Sonnet delta
Mean delta
Harness
mimic-urine-output-rate-raw
0.476
0.485
0.480
matched
mimic-urine-output-rate
0.415
0.488
0.452
matched
mimic-meld-24h
0.067
0.615
0.341
CLI
mimic-creatinine-baseline
0.232
0.379
0.305
matched
mimic-meld-24h-raw
0.165
0.445
0.305
CLI
mimic-creatinine-baseline-raw
0.160
0.406
0.283
matched
Appendix
Table 31: Full-coverage Claude replication per-task skill deltas. “Harness” marks tasks whose no-skill and with-skill cells match in recorded CLI version and routing profile, or the indicator that differs.
Clinical reasoning agents based on large language models (LLMs) aim to automate tasks such as intensive care unit (ICU) monitoring and patient state tracking from electronic health records (EHRs). Existing systems typically rely on manually curated clinical tools or skills for concepts such as sepsis detection and organ failure assessment. However, maintaining these tool libraries requires substantial expert effort, while zero-shot querying or code generation often produces inefficient and unreliable reasoning chains, especially under institution-specific clinical policies. We introduce CodeClinic, a benchmark built on MIMIC-IV for evaluating whether LLM agents can synthesize and compose reusable clinical skills instead of relying on fixed toolboxes. The benchmark contains two complementary tasks: longitudinal ICU surveillance and compositional information seeking. The longitudinal setting simulates monitoring patient trajectories with structured decisions every four hours across 25 findings and eight clinical families, while the compositional setting spans 63k instances across 259 tasks in nine domains and is stratified by compositional dependency depth to evaluate increasingly complex multi-step reasoning. We further propose an offline autoformalization pipeline that converts natural-language clinical guidelines into reusable and verified Python skill libraries through iterative LLM refinement. Compared with zero-shot code generation, the resulting libraries improve consistency while reducing per-query token usage by up to 40%.
Timothy Ossowski, Xinchi Liu, Danyal Maqbool +6
University of Wisconsin–Madison · 2Microsoft Research
Clinical data-science agents must transform heterogeneous longitudinal records into auditable analyses, yet existing benchmarks largely isolate medical question answering, structured-table reasoning, or generic scientific repositories. We introduce CLINLENS, a benchmark of 200 executable tasks over five linked MIMIC resources spanning structured electronic health records, notes, electrocardiograms, chest radiographs, and echocardiograms. A 4 x 5 taxonomy crosses four patient-time scopes with five analysis capabilities. Program-first reverse synthesis pairs each bounded semi-raw package with an evaluator-private reference workflow and checks required artifacts, cohort and temporal semantics, and the final answer. On a fixed 126-task suite, the strongest of 24 standardized model-scaffold configurations achieves 56.3% scope-macro STRICTPASS despite 100% EXECSUCCESS. For reference, a separately configured coding agent solves 83 of 126 tasks, while five biomedical systems adapted to GPT-4o-mini reach at most 2.9% scope-macro STRICTPASS. These results expose a substantial gap between runnable submissions and correct clinical analyses.
We introduce PhysicianBench, a benchmark for evaluating LLM agents on physician tasks grounded in real clinical setting within electronic health record (EHR) environments. Existing medical agent benchmarks primarily focus on static knowledge recall, single-step atomic actions, or action intent without verifiable execution against the environment. As a result, they fail to capture the long-horizon, composite workflows that characterize real clinical systems. PhysicianBench comprises 100 long-horizon tasks adapted from real consultation cases between primary care and subspecialty physicians, with each task independently reviewed by a separate panel of physicians. Tasks are instantiated in an EHR environment with real patient records and accessed through the same standard APIs used by commercial EHR vendors. Tasks span 21 specialties (e.g., cardiology, endocrinology, oncology, psychiatry) and diverse workflow types (e.g., diagnosis interpretation, medication prescribing, treatment planning), requiring an average of 27 tool calls per task. Solving each task requires retrieving data across encounters, reasoning over heterogeneous clinical information, executing consequential clinical actions, and producing clinical documentation. Each task is decomposed into structured checkpoints (670 in total across the benchmark) capturing distinct stages of completion graded by task-specific scripts with execution-grounded verification. Across 13 proprietary and open-source LLM agents, the best-performing model achieves only 46% success rate (pass@1), while open-source models reach at most 19%, revealing a substantial gap between current agent capabilities and the demands of real-world clinical workflows. PhysicianBench provides a realistic and execution-grounded benchmark for measuring progress toward autonomous clinical agents.
Ruoqi Liu, Imran Q. Mohiuddin, Austin J. Schoeffler +10