cs.AIOct 27, 2026
SaveM4Bench: Evaluating Procedural Specification for Clinical EHR Derivation Agents
Organizations: Laboratory for Computational Physiology, Massachusetts Institute of Technology, USA · Department of Surgery, Seoul National University Hospital, Korea · Charité, University Medicine Berlin, Germany · Department of Biomedical Systems Informatics, Yonsei University College of Medicine, Korea ·
Abstract
Clinical research over electronic health records depends on derived variables whose definitions sit outside the schema, in item codes, time windows, missingness rules, and output-grain choices that change cohorts and downstream conclusions. We introduce M4Bench, 28 clinical derivation tasks across 15 task families on MIMIC-IV and eICU, to test whether AI coding agents can reproduce such derivations and how clinician-reviewed agent skills change their results. Its controls decompose the value of skills into specification completeness, delivery format, and schema familiarity. In a Docker-isolated Codex campaign (GPT 5.4 Mini and GPT 5.5), skills raise reward by 0.136 over the no-skill baseline, with 26 of 28 tasks improving. Specification completeness accounts for most of this gain: on an eight-task control subset, an operational spec alone raises reward by 0.162, and skills add only 0.024 on top, indistinguishable from zero. On a ten-task subset, decoy skills score like no skill and raw reference SQL nearly matches targeted skills, arguing against generic context or the skill format as the explanation. Schema obfuscation and restructuring preserve 86% and 70% of the skill gain (GPT 5.4 Mini, 26 MIMIC-IV tasks), a descriptive robustness check rather than a bound on memorization. A descriptive Claude Code replication on all 28 tasks (Opus 4.7 and Sonnet 4.6, 560 runs) is directionally consistent (+0.120, 21 of 28 tasks improving). These gains are partial agreement with the reference, not completed tasks: only 25 of 280 with-skill and 13 of 280 no-skill Codex runs pass every strict check, and skills barely change which rows agents return (key F1 +0.003). Skills are released as a versioned, clinician-reviewable markdown delivery format, with a family-level reference-validity audit. M4Bench releases tasks, skills, evaluation code, and sanitized per-run evidence as an open-source benchmark.
Figures & tables
Figure 1: M4Bench evaluation pipeline. Each task pairs a task card with a DuckDB clone whose reference targets are removed. The primary no-skill – with-skill comparison and matched controls test specification completeness, reference-code structure, and schema familiarity under sandboxed execution and contamination auditing.
| Term | What the agent receives | Purpose |
| no-skill | Task instruction and task database | Schema-plus-task baseline; no curated derivation procedure |
| with-skill | no-skill inputs plus a task-specific procedural skill | Primary test of targeted procedural specification |
| Decoy skill | no-skill inputs plus a task-mismatched skill artifact | Controls for generic extra context and skill-file presence |
| Raw reference SQL | no-skill inputs plus reference SQL text | Tests reference-code structure; raw-task cells require SQL adaptation rather than answer copying |
| Operational spec | no-skill inputs plus complete non-SQL clinical rules, excluding item identifiers and dataset recipes | Separates rule completeness from dataset navigation and skill packaging |
| Schema perturbations | no-skill or with-skill inputs on obfuscated or restructured schemas | Probes schema-name and table-organization familiarity |
Table 1: Glossary of benchmark conditions, controls, and metrics.
| Family | Tasks | Standard | Raw | Database |
|---|---|---|---|---|
| APS III | 2 | 1 | 1 | MIMIC-IV |
| Charlson | 1 | 0 | 1 | MIMIC-IV |
| Baseline creatinine | 2 | 1 | 1 | MIMIC-IV |
| GCS | 2 | 0 | 2 | MIMIC-IV/eICU |
| KDIGO AKI | 2 | 1 | 1 | MIMIC-IV |
| MELD | 2 | 1 | 1 | MIMIC-IV |
Table 2: M4Bench task inventory. Standard tasks retain selected intermediate features; raw tasks require reconstruction from source tables.
| Measure | No skill | With skill | Delta | Improved tasks |
|---|---|---|---|---|
| Reward / GPT 5.4 Mini | 0.639 | 0.802 | 0.162 [0.099, 0.232] | 26/28 |
| Reward / GPT 5.5 | 0.767 | 0.876 | 0.109 [0.060, 0.166] | 24/28 |
| Reward / task-balanced | 0.703 | 0.839 | 0.136 [0.085, 0.191] | 26/28 |
| Key F1 / task-balanced | 0.962 | 0.965 | 0.003 [-0.010, 0.018] | 5/28 |
| Ordered nMAE / task-balanced | 0.116 | 0.060 | -0.056 [-0.085, -0.029] | 23/26 |
Table 3: Native-schema Codex skill effect and diagnostics. Reward is mean value-column match rate; key F1 measures row grain; ordered nMAE is lower-is-better. Intervals are 95% task-bootstrap intervals.
| Comparison | Tasks | No skill | Operational spec | Targeted skill | Delta |
|---|---|---|---|---|---|
| Operational spec–no skill | 8 | 0.606 | 0.767 | – | 0.162 [0.072, 0.260] |
| Targeted skill–operational spec | 8 | – | 0.767 | 0.792 | 0.024 [-0.034, 0.099] |
Table 4: Codex operational-spec follow-up on the eight-task control subset. Values are task-balanced after averaging the two Codex models within each task.
| Task set | Tasks | Runs | Delta [95% CI] | Positive | Strict passes (no/with) | Key F1 delta |
|---|---|---|---|---|---|---|
| All tasks | 28 | 560 | 0.120 [0.067, 0.177] | 21/28 | 25/280 vs 37/280 | 0.008 |
| CLI-version matched | 20 | 400 | 0.113 [0.050, 0.185] | 15/20 | 19/200 vs 27/200 | 0.008 |
| CLI and routing matched | 19 | 380 | 0.119 [0.054, 0.192] | 14/19 | 19/190 vs 27/190 | 0.008 |
Table 5: Full-coverage Claude replication: task-balanced native-schema with-skill minus no-skill reward on all 28 tasks, on the 20 tasks whose no-skill and with-skill cells share one recorded Claude Code version (CLI) per model, and on the 19 that also share one routing profile. Intervals are 95% task bootstraps. Strict passes and key F1 are separate from reward.
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Mode | Database | Key grain | Scored columns | Reference rows |
|---|---|---|---|---|---|
| eicu-gcs | raw | eICU | stay | 4 | 200859 |
| eicu-oasis | raw | eICU | stay | 11 | 200859 |
| mimic-apsiii-24h | standard | MIMIC-IV | stay | 17 | 94458 |
| mimic-apsiii-24h-raw | raw | MIMIC-IV | stay | 17 | 94458 |
| mimic-charlson-raw | raw | MIMIC-IV | admission | 19 | 546028 |
| mimic-creatinine-baseline | standard | MIMIC-IV | admission | 3 | 546028 |
Table 6: Per-task evaluation details. Reference rows are reference-output rows before task-specific evaluation joins.
| Task | Reviewers | Source basis |
|---|---|---|
| eicu-gcs | C1/C4 | eICU concept conventions |
| eicu-oasis | C2/C4 | eICU concept conventions |
| mimic-apsiii-24h | C1/C4 | MIT-LCP concept conventions |
| mimic-apsiii-24h-raw | C1/C4 | MIT-LCP concept conventions |
| mimic-charlson-raw | C1/C4 | MIT-LCP concept conventions |
| mimic-creatinine-baseline | C1/C4 | Public clinical definitions |
Table 7: Task-level clinical review provenance. Each task is reviewed by two clinicians, identified as C1–C4, and every review covered the task definition, schema, and output. Reviewer identifiers are described in Table 8 .
| Reviewer ID | Description |
|---|---|
| C1 | MD, MS; board-certified vascular and trauma surgeon; associate professor at a tertiary academic center; clinical registry and administrative-data researcher; MIMIC-IV/eICU experience. |
| C2 | MD, PhD; board-certified intensivist and emergency physician; clinical epidemiologist; ICU EHR researcher; MIMIC-IV/eICU experience. |
| C3 | MD; doctoral candidate and clinical AI researcher; expertise in explainable AI, ML-based clinical decision support, and multimodal prediction models; MIMIC-IV/eICU experience. |
| C4 | MD, PhD; associate professor at a medical school; physician and clinical AI researcher; expertise in ICU/EHR data research, multimodal clinical foundation models, and clinical AI evaluation; MIMIC-IV research experience. |
Table 8: Clinical reviewer identifiers. Descriptions summarize credentials relevant to clinical derivation review.
| Family | Reference convention retained | Validity caveat |
|---|---|---|
| Suspected infection / Sepsis-3 | Antibiotic–microbiology timing rule from public concept code | Broad culture inclusion can overcall suspected infection; downstream Sepsis-3 inherits this target. |
| SOFA / Sepsis-3 | Missing organ components default to zero | Agreement may reflect the public-code missingness convention rather than adjudicated organ failure. |
| APS III | MIMIC-Code component implementation | Known respiratory and oxygenation pathway discrepancies are retained for reproducibility. |
| Vasopressor equivalents | Five-agent norepinephrine-equivalent mapping | Angiotensin II administrations are not scored in the current reference. |
| eICU OASIS | eICU APACHE abstraction tables where source charting is not equivalent | eICU raw tasks are not schema-symmetric with MIMIC raw reconstruction. |
Table 9: Reference-validity audit for selected task families. These caveats describe the declared benchmark target, not new exclusions from scoring.
| Codex primary | Claude sentinel | |
| Canonical runs | 760 | 190 |
| Provenance-only dirs | 40 | 0 |
| Retained dirs | 800 | 190 |
| result.json | 800 | 190 |
| output.csv | 800 | 190 |
| instruction.md | 800 | 190 |
Table 10: Retained paper-facing artifact directories after rerun replacement. Retained size is the uncompressed sum of listed files, not credentialed task databases. Codex has 760 canonical analysis runs plus 40 provenance-only directories.
| Codex primary | Claude sentinel | |
| Canonical runs | 760 | 190 |
| Provenance-only dirs | 40 | 0 |
| Retained dirs | 800 | 190 |
| Hardware | MacBook Pro (M3 Pro, 18 GB) | MacBook Pro (M3 Pro, 18 GB) |
| Agent wall-clock h | 56.4 | 17.5 |
| Uncached input | 586,293,569 | 16,657 |
Table 11: Compute and storage for the retained evidence package. Agent wall-clock sums the per-run agent execution time recorded in result.json . Selected dirs are retained paper-facing run directories on disk. Codex and Claude token rows reflect each provider’s native usage reporting (Codex: total prompt tokens; Claude: split into uncached vs. cached input under provider prompt caching), so uncached counts are not directly comparable across arms.
| Asset | Role | Access, license, or terms |
|---|---|---|
| M4Bench code, tasks, evaluator, skills | Benchmark artifact | MIT license (repository LICENSE); camera-ready source release tagged m4bench-neurips2026-camera-ready. |
| MIMIC-IV v3.1 | Source EHR data for 26 tasks | PhysioNet credentialed access and MIMIC-IV data-use terms ( https://physionet.org/content/mimiciv/3.1/ ); derived databases are not openly redistributed. |
| eICU Collaborative Research Database v2.0 | Source EHR data for 2 tasks | PhysioNet credentialed access and eICU data-use terms ( https://physionet.org/content/eicu-crd/2.0/ ); derived databases are not openly redistributed. |
| MIT-LCP MIMIC-Code / eICU concept conventions | Reference SQL lineage and procedural conventions | Credited and cited; adapted SQL/procedural conventions retained with provenance. |
| Per-run evidence packages | Evaluator results, sanitized traces, instructions, egress logs, hash lineage | Open packages withhold row-level agent outputs and tool outputs; originals stay private under source-data use terms. |
Table 12: Source assets, release terms, and access restrictions.
| Task | Mode | Database | Delta | Decoy/raw SQL | Op. spec |
|---|---|---|---|---|---|
| mimic-urine-output-rate-raw | raw | MIMIC-IV | 0.471 | yes | yes |
| mimic-urine-output-rate | standard | MIMIC-IV | 0.460 | yes | |
| mimic-meld-24h | standard | MIMIC-IV | 0.354 | yes | |
| mimic-oasis-24h | standard | MIMIC-IV | 0.296 | yes | |
| mimic-suspicion-infection-raw | raw | MIMIC-IV | 0.291 | yes | |
| mimic-meld-24h-raw | raw | MIMIC-IV | 0.283 | yes |
Table 13: Control-subset membership and final Codex targeted-skill delta (task-balanced over both models) for all 28 tasks. Each task’s family follows from its name, and families are listed in Table 2 .
| Archive | Contents | Runs | Bytes |
|---|---|---|---|
| m4bench-evidence- submission.tar.zst | Submitted campaigns (Codex primary, Codex follow-up, Claude sentinel, OSS) | 1,400 | 7,548,717 |
| SHA-256: f8b72940bd98cd9b2a852fa79c416310e2ed8ec7795b181941358d3b03ca708c | |||
| m4bench-evidence- claude-replication.tar.zst | Full-coverage Claude replication | 560 | 6,398,244 |
| SHA-256: 4aeb5eda83090f89e5a3bf5d0e8c4568c18309efd66b496c4655e11abd13471b | |||
Table 14: Open evidence packages, hosted at https://huggingface.co/datasets/hannesill/m4bench-evidence and attached to the tagged release.
| Audit quantity | Count |
|---|---|
| Runs | 760 |
| Expected primary/supplementary runs | 760 |
| Task/condition/schema/model cells | 152 |
| Publishable flag failures | 0 |
| Isolation flag failures | 0 |
| Filesystem canary failures | 0 |
Table 15: Primary Codex campaign integrity checks after replacing matching May 2 runs with the May 4 clinician-review reruns. Publishability means the run is eligible for analysis. It does not mean the submitted derivation is clinically correct.
Figure 2: Per-task native-schema skill delta (with-skill minus no-skill mean reward) for the primary Codex campaign and the full-coverage Claude replication on all 28 tasks, and for the exploratory gpt-oss-20b arm on the eight tasks it covers.
| Task | Mean delta |
|---|---|
| mimic-urine-output-rate-raw | 0.471 |
| mimic-urine-output-rate | 0.460 |
| mimic-meld-24h | 0.354 |
| mimic-oasis-24h | 0.296 |
| mimic-suspicion-infection-raw | 0.291 |
| mimic-meld-24h-raw | 0.283 |
Table 16: Largest native-schema skill gains after averaging task-level deltas across the Codex models.
| Task | Mean delta |
|---|---|
| mimic-kdigo-48h-raw | -0.040 |
| mimic-kdigo-48h | -0.002 |
| mimic-charlson-raw | 0.001 |
| mimic-vasopressor-equivalents | 0.006 |
| mimic-sofa-24h | 0.010 |
| mimic-sirs-24h-raw | 0.016 |
Table 17: Largest negative or flat native-schema skill effects after averaging task-level deltas across the Codex models.
| Model | Mode | Tasks | Mean delta | Positive tasks |
|---|---|---|---|---|
| GPT 5.4 Mini | standard | 11 | 0.203 | 10/11 |
| GPT 5.4 Mini | raw | 17 | 0.136 | 16/17 |
| GPT 5.5 | standard | 11 | 0.105 | 9/11 |
| GPT 5.5 | raw | 17 | 0.112 | 15/17 |
| Task-balanced mean | standard | 11 | 0.154 | 10/11 |
| Task-balanced mean | raw | 17 | 0.124 | 16/17 |
Table 18: Native-schema Codex skill effect split by task mode. Values are task-paired deltas. Final rows average the two Codex models within task before aggregating.
| Analysis | Units | Mean delta | 95% interval |
|---|---|---|---|
| Task-cluster bootstrap | 28 tasks | 0.136 | [0.085, 0.191] |
| Family-cluster bootstrap | 15 families | 0.136 | [0.065, 0.207] |
| Family-balanced bootstrap | 15 families | 0.120 | [0.057, 0.190] |
Table 19: Cluster sensitivity for the primary native-schema Codex skill effect. Task rows are the paper headline. Family rows resample or balance the 15 task families to check sensitivity to standard/raw sibling tasks.
| Condition or worst cell | N | Key F1 | Key precision | Extra keys |
|---|---|---|---|---|
| no-skill | 56 | 0.962 | 0.967 | 2384458 |
| with-skill | 56 | 0.965 | 0.976 | 2087534 |
| mimic-ventilation-raw / no-skill / GPT 5.4 Mini | 5 | 0.700 | 0.559 | 555030 |
| mimic-ventilation-raw / no-skill / GPT 5.5 | 5 | 0.717 | 0.573 | 525311 |
| mimic-ventilation-raw / with-skill / GPT 5.4 Mini | 5 | 0.720 | 0.583 | 505933 |
| mimic-ventilation-raw / with-skill / GPT 5.5 | 5 | 0.740 | 0.603 | 459225 |
Table 20: Native-schema key diagnostics for Codex. The first two rows summarize native no-skill and with-skill cells. Subsequent rows show high-extra-key cells. N counts cells in the first two rows and runs in the others. Key F1 and key precision are means.
| Task | Reward delta | Key F1 delta | nMAE delta | Ordered cols |
|---|---|---|---|---|
| eicu-gcs | 0.054 | 0.000 | -0.029 | 4 |
| eicu-oasis | 0.272 | 0.000 | -0.189 | 11 |
| mimic-apsiii-24h | 0.023 | 0.000 | -0.013 | 17 |
| mimic-apsiii-24h-raw | 0.055 | 0.000 | -0.036 | 17 |
| mimic-charlson-raw | 0.001 | 0.000 | -0.001 | 19 |
| mimic-creatinine-baseline | 0.142 | -0.073 | 0.015 | 3 |
Table 21: Per-task secondary diagnostics for native-schema Codex. nMAE is computed only for ordered value columns after matching on declared keys. Negative nMAE deltas are improvements.
| Control | Tasks | No skill | Targeted | Control | Control–targeted | Control–no skill |
|---|---|---|---|---|---|---|
| Raw reference SQL | 10 | 0.553 | 0.833 | 0.837 | 0.003 | 0.283 |
| Decoy skill | 10 | 0.553 | 0.833 | 0.554 | -0.280 | 0.000 |
Table 22: Matched-context controls on selected tasks in the final combined Codex analysis. “Control–targeted” compares the control condition to the targeted with-skill condition after task-level averaging.
| Schema | Tasks | No skill | With skill | Skill delta | Positive tasks | Robustness ratio |
|---|---|---|---|---|---|---|
| native | 26 | 0.637 | 0.794 | 0.157 [0.092, 0.229] | 24/26 | 1.000 |
| obfuscated | 26 | 0.561 | 0.696 | 0.135 [0.068, 0.208] | 19/26 | 0.862 |
| restructured | 26 | 0.557 | 0.668 | 0.110 [0.057, 0.170] | 23/26 | 0.704 |
Table 23: Codex schema-skill generalization follow-up for GPT 5.4 Mini over 26 MIMIC tasks. Transformed rows match May 6 with-skill runs to the earlier transformed-schema no-skill probe.
| Audit quantity | Count |
|---|---|
| Raw result directories | 363 |
| Canonical run keys | 340 |
| Superseded duplicate directories | 23 |
| Operational-spec canonical runs | 80 |
| Transformed-schema with-skill canonical runs | 260 |
| Publishable flag failures | 0 |
Table 24: May 6 Codex validity follow-up integrity. Superseded retry directories are retained in the manifest for audit but excluded from analysis tables.
| Task | No skill | Operational spec | Targeted skill | Spec–no | Targeted–spec |
|---|---|---|---|---|---|
| mimic-urine-output-rate-raw | 0.372 | 0.802 | 0.843 | 0.430 | 0.041 |
| mimic-ventilation | 0.536 | 0.528 | 0.782 | -0.008 | 0.254 |
| mimic-creatinine-baseline-raw | 0.620 | 0.794 | 0.800 | 0.174 | 0.006 |
| mimic-suspicion-infection | 0.720 | 0.932 | 0.820 | 0.212 | -0.112 |
| mimic-vasopressor-equivalents-raw | 0.733 | 0.842 | 0.856 | 0.109 | 0.014 |
| mimic-oasis-24h | 0.699 | 0.987 | 0.995 | 0.288 | 0.008 |
Table 25: Operational-spec follow-up by sentinel task after averaging the two Codex models within each task.
| Model | No skill | With skill | Task-paired delta | Positive tasks |
|---|---|---|---|---|
| Claude Opus | 0.738 | 0.843 | 0.106 | 6/8 |
| Claude Sonnet | 0.592 | 0.817 | 0.226 | 8/8 |
| Task-balanced mean | – | – | 0.166 [0.085, 0.259] | 8/8 |
Table 26: Historical (submitted) Claude sentinel native-schema skill effect across eight tasks.
| Task | Opus delta | Sonnet delta | Mean delta |
|---|---|---|---|
| urine-output rate raw | 0.391 | 0.484 | 0.438 |
| baseline creatinine raw | 0.160 | 0.298 | 0.229 |
| ventilation | 0.159 | 0.263 | 0.211 |
| OASIS-24h | 0.103 | 0.214 | 0.158 |
| suspected infection | 0.043 | 0.242 | 0.143 |
| vasopressor equivalents raw | -0.021 | 0.219 | 0.099 |
Table 27: Historical (submitted) Claude sentinel per-task native-schema skill deltas.
| Model | No skill | With skill | Task-paired delta | Positive tasks |
|---|---|---|---|---|
| gpt-oss-20b local | 0.085 | 0.094 | 0.008 [-0.072, 0.088] | 3/8 |
Table 28: Exploratory local OSS native-schema skill effect. This arm is excluded from primary and supplementary paper-facing evidence because local Ollama runs do not meet the release publishability criteria.
| Model | No skill | With skill | Task-paired delta | Positive tasks |
|---|---|---|---|---|
| Claude Opus | 0.800 | 0.883 | 0.083 [0.038, 0.135] | 25/28 |
| Claude Sonnet | 0.694 | 0.851 | 0.156 [0.087, 0.231] | 20/28 |
| Task-balanced mean | 0.747 | 0.867 | 0.120 [0.067, 0.177] | 21/28 |
Table 29: Full-coverage Claude replication: model-specific and task-balanced native-schema with-skill minus no-skill reward differences over 28 tasks (95% task-bootstrap intervals).
| Condition | Runs | CLI 2.1.119 | CLI 2.1.220 | Overrides recorded | Haiku usage | Haiku turns | Agent hours |
|---|---|---|---|---|---|---|---|
| no-skill | 280 | 167 | 113 | 120 | 160 | 4 | 28.6 |
| with-skill | 280 | 100 | 180 | 110 | 170 | 1 | 22.6 |
Table 30: Recorded harness indicators of the full-coverage Claude replication by condition: observed Claude Code version, recorded model-slot overrides, runs whose usage reports Haiku, runs with Haiku assistant turns, and summed agent wall-clock hours. Hardware was not recorded per run.
| Task | Opus delta | Sonnet delta | Mean delta | Harness |
|---|---|---|---|---|
| mimic-urine-output-rate-raw | 0.476 | 0.485 | 0.480 | matched |
| mimic-urine-output-rate | 0.415 | 0.488 | 0.452 | matched |
| mimic-meld-24h | 0.067 | 0.615 | 0.341 | CLI |
| mimic-creatinine-baseline | 0.232 | 0.379 | 0.305 | matched |
| mimic-meld-24h-raw | 0.165 | 0.445 | 0.305 | CLI |
| mimic-creatinine-baseline-raw | 0.160 | 0.406 | 0.283 | matched |
Table 31: Full-coverage Claude replication per-task skill deltas. “Harness” marks tasks whose no-skill and with-skill cells match in recorded CLI version and routing profile, or the indicator that differs.