Objectify, Don't Delete: Complaint-Derived Evaluation of Ambient Clinical Notes
Organizations: Suki AI, California, USA
Abstract
Evaluating the quality of AI-generated clinical notes is hard: standard metrics like ROUGE measure lexical similarity to reference notes, while expert checklists encode what committees believe matters. Neither captures what clinicians actually complain about. Recent work has shown that evaluation criteria can be mined from user feedback and filtered to criteria that LLM judges can reliably enforce. We argue that this is where much of the value is lost. The criteria that fail enforceability are disproportionately the ones clinicians raise most often, because high-frequency complaints are stated as several questions glued together. No yes/no judgment can grade such composites as posed. We introduce an objectification step that rewrites such criteria into sets of scoped, determinate decisions. We demonstrate the approach through an evaluation pipeline that mines criteria from complaints, objectifies them, and deploys LLM judges against both planted defects and production notes. Across eleven defects planted into ACI-bench reference notes, objectified judges localize violations with median recall of 0.95, while reference metrics change by at most 0.037. Given the complaint as one broad question, the same judges still find every planted defect, but they also flag 70% of the unperturbed reference notes; one section-pair question per call raises Youden's J from 0.30 to 0.72 (by 0.19 to 0.26 with prompt effort matched). We release ACI-Defect, a perturbation benchmark; clinician-reviewed instance-level labels showing quality gaps in the ACI-bench gold standard (32 of 40 notes contain hallucinations; 40 of 40 omit discussed facts); and grounding-judge verdicts. On 1,446 production notes, mean instance-rate burden falls at every clinician-rating step (rho=-0.15), a small association. Defect counts have to be kept per span rather than collapsed into one pass/fail bit per note.
Figures & tables
| Family (disjoint) | % | Disposition |
|---|---|---|
| Composite structure | 14.3 | Unenforceable as posed; objectified into leakage, format, redundancy |
| of which misplacement | 10.0 | Four directional leakage judges |
| Atomic / tool-needing | 14.4 | Judged directly, except billing codes, which need a tool |
| Not a note defect | 6.8 | Excluded from the rubric |
| Standing preference | 0.1 | Parameterized, not scored as leakage |
| Unclassified | 64.5 | Keyword lower bound |
| Dimension | Units | Raw P / R | Corr. P / R | F1 raw corr. | Positives | Notes |
|---|---|---|---|---|---|---|
| Hallucination | sentences | 0.21 / 0.86 | 0.57 / 0.97 | 0.34 0.72 | 44 102 | 32 |
| Omission | case facts | 0.56 / 0.79 | 0.66 / 0.83 | 0.66 0.74 | 334 368 | 40 |
| ID | Plant | Target class | Recall | ROUGE-1 | BERTScore | MedCon-p | |
|---|---|---|---|---|---|---|---|
| P1 | A&P sentence History | 28/40 | leakage | 0.96 (27/28) | 0.000 | 0.017 | 0.006 |
| P2 | HPI narrative A&P | 28/40 | leakage | 0.54 (15/28) | 0.000 | 0.017 | 0.000 |
| P3 | Symptom sentence PE | 40/40 | leakage | 1.00 (40/40) | 0.023 | 0.010 | 0.000 |
| P4 | RESULTS sentence PE | 30/40 | leakage | 0.90 (27/30) | 0.018 | 0.009 | 0.000 |
| P5 | Fabricated normal PE finding | 40/40 | fabrication | 1.00 (40/40) | 0.016 | 0.011 | 0.000 |
| P6 | Delete modifier span | 36/40 | omission | 0.83 (30/36) | 0.007 | 0.005 | 0.037 |
| Planted-span hits / | As an instrument | ||||||
|---|---|---|---|---|---|---|---|
| Criterion | P1 | P2 | P3 | P4 | Recall | Clean | |
| Objectified pairs | 27/28 | 15/28 | 40/40 | 27/30 | 0.87 | 0.02–0.26 | 0.72 |
| Complaint, as posed | 28/28 | 28/28 | 40/40 | 30/30 | 1.00 | 0.70 | 0.30 |
| Zhou et al. criterion | 26/28 | 27/28 | — | — | 0.95 | — | — |
| Model | Obj. (mean FPR) | Composite (FPR) | Obj. (union FPR) | P2 obj / cmp |
|---|---|---|---|---|
| Gemini 2.5 Pro | 0.81 (0.18) | 0.62 (0.38) | 0.51 (0.47) | 0.60 / 0.62 |
| GPT-4.1 | 0.66 (0.22) | 0.40 (0.53) | 0.34 (0.55) | 0.19 / 0.22 |
| Qwen3-Next-80B | 0.67 (0.23) | 0.45 (0.51) | 0.28 (0.62) | 0.33 / 0.49 |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Defect clinicians raise | Example (ACI-bench / synthesized) | PDQI-9 / PDSQI-9 | ROUGE / MedCon vs gold | Objectified check |
|---|---|---|---|---|
| Assessment sentence in History. | “I recommend … Robitussin” in History | No (not sectional leakage) | No if tokens unchanged | Directional leakage |
| Fabricated normal exam. | “CN II–XII intact” inserted into PE | Partial (accuracy) | Weak if few tokens | Sentence hallucination |
| Discussed dose missing from plan. | Spoken dose change omitted | Partial (completeness) | Weak if other overlap high | Case-fact omission |
| “History of” on an active problem. | Active diagnoses rewritten “History of X” | No | Near-paraphrase | “History of” misuse |
| Documented sex chart. | “59-year-old male” female; chart stays M | No | One-token swap | Demographics |
| Group | Evaluators | On ACI-bench? |
|---|---|---|
| Grounding | Omission (case-fact level), hallucination (sentence level) | Yes |
| Directional leakage | A&P History, HPI A&P, subjective PE, results PE | Yes |
| Language use | “History of” misuse, provider voice | Yes (defaults) |
| Structure | Diagnosis structure, within-section redundancy | Yes |
| Identity | Demographics documentation | Yes (age / sex; no stored gender-identity prefs) |
| Deterministic | Regex/rule style, verbosity bands, section presence, CC conciseness, section format | Yes |
| Verdict | Corrected gold label |
|---|---|
| Judge correct | Follows the judge |
| Independent label correct | Follows the sheet |
| Both wrong | Negative |
| Grain / uncertain | Dropped (11 hallucination sentences; 8 omission facts) |
| ID | Perturbation | Target evaluator | Why ROUGE / MedCon-proxy should be weak |
|---|---|---|---|
| P1 | Move an A&P sentence into History verbatim | A&P History leakage | Token bag unchanged |
| P2 | Move an HPI narrative sentence into A&P | HPI A&P leakage | Token bag unchanged |
| P3 | Insert a symptom-report sentence into PE | Subjective PE leakage | 17 tokens into a 300–700 word note |
| P4 | Copy a RESULTS sentence into PE | Results PE leakage | Tokens duplicated; concept set unchanged |
| P5 | Insert a fabricated normal exam finding (e.g., cranial nerves II–XII intact) | Hallucination | Adds plausible clinical tokens; concept F1 may increase |
| P6 | Delete every copy of a case-fact modifier span (dose, laterality + anatomy) | Omission | Remaining lexical overlap high |
| Comment theme | Mapped judge rate (findings / opportunities) |
|---|---|
| Omission | Inappropriately missing facts / case facts |
| Hallucination | Unsupported sentences / sentences |
| “History of” | Misuse instances / “history of” phrases |
| Misplacement | A&P HPI leakage rate |
| Disorganization | Bundled diagnosis headings / diagnosis headings |
| Named family | Named rate | Other rate | ||
|---|---|---|---|---|
| Omission | 62 | 0.27 | 0.25 | |
| History-of | 18 | 0.63 | 0.78 | |
| Hallucination | 23 | 0.045 | 0.085 | |
| Misplacement (A&P HPI) | 105 | 0.109 | 0.102 | |
| Disorg. (bundled heading) | 41 | 0.024 | 0.015 |
Clinical Language Model Evaluation
Clinical language model evaluation measures whether AI systems are accurate, safe, fair, and useful across medical tasks and care settings.
See all 457 papers