Evaluating the quality of AI-generated clinical notes is hard: standard metrics like ROUGE measure lexical similarity to reference notes, while expert checklists encode what committees believe matters. Neither captures what clinicians actually complain about. Recent work has shown that evaluation criteria can be mined from user feedback and filtered to criteria that LLM judges can reliably enforce. We argue that this is where much of the value is lost. The criteria that fail enforceability are disproportionately the ones clinicians raise most often, because high-frequency complaints are stated as several questions glued together. No yes/no judgment can grade such composites as posed. We introduce an objectification step that rewrites such criteria into sets of scoped, determinate decisions. We demonstrate the approach through an evaluation pipeline that mines criteria from complaints, objectifies them, and deploys LLM judges against both planted defects and production notes. Across eleven defects planted into ACI-bench reference notes, objectified judges localize violations with median recall of 0.95, while reference metrics change by at most 0.037. Given the complaint as one broad question, the same judges still find every planted defect, but they also flag 70% of the unperturbed reference notes; one section-pair question per call raises Youden's J from 0.30 to 0.72 (by 0.19 to 0.26 with prompt effort matched). We release ACI-Defect, a perturbation benchmark; clinician-reviewed instance-level labels showing quality gaps in the ACI-bench gold standard (32 of 40 notes contain hallucinations; 40 of 40 omit discussed facts); and grounding-judge verdicts. On 1,446 production notes, mean instance-rate burden falls at every clinician-rating step (rho=-0.15), a small association. Defect counts have to be kept per span rather than collapsed into one pass/fail bit per note.
Figures & tables
Figure 1: Objectification keeps the criterion and changes the question. Three items Zhou et al. (2025) drop (quoted from their appendix) become directional leakage, per-fact omission, and a pair of determinate checks. Their filter is correct under a one-section, no-N/A scope; we change that scope rather than delete the item.
Figure 2: Evaluation loop. Comments induce the rubric (C1). Judges score sentences or case facts, not the note as a whole. Production ratings are compared to those instance rates; comments and ratings are hidden from judges. Deployed prompts, orchestration, and production notes are not released.
Family (disjoint)
%
Disposition
Composite structure
14.3
Unenforceable as posed; objectified into leakage, format, redundancy
of which misplacement
10.0
Four directional leakage judges
Atomic / tool-needing
14.4
Judged directly, except billing codes, which need a tool
Not a note defect
6.8
Excluded from the rubric
Standing preference
0.1
Parameterized, not scored as leakage
Unclassified
64.5
Keyword lower bound
Table 1: Complaint families from a year of production feedback, user-weighted over the 480 users who left an unfavorable comment (rating <4 and more than two words). Families are disjoint and sum to 100 % up to rounding; the indented row is the largest tag within composites, not a sixth family. The keyword audit is a lower bound, so the unclassified residual is not a claim that those comments are defect-free.
Figure 3: ACI-bench reference notes have quality gaps. Left: raw judge fire rates on unmodified gold, before adjudication. The grounding judges fire on almost every note, while leakage, language, structure, and identity stay low. (Table 2 reports the smaller confirmed counts.) Right: six judge verdicts on test1 gold, each pairing the note line with the transcript evidence; all grounding findings are clinician-confirmed.
Dimension
Units
Raw P / R
Corr. P / R
F1 raw → corr.
Positives
Notes
Hallucination
sentences
0.21 / 0.86
0.57 / 0.97
0.34 → 0.72
44 → 102
32
Omission
case facts
0.56 / 0.79
0.66 / 0.83
0.66 → 0.74
334 → 368
40
Table 2: Objectified judges vs gold on n=40 test1 originals, before and after discrepancy adjudication. Instance grain. Hallucination = binary has-hallucination (any of fabrication / transformation / inference / misattribution; benign-only excluded). Omission = inappropriately missing or partially so. Notes = of 40 with ≥1 positive. Adjudicator was not blinded: corrected precision is an upper bound.
ID
Plant
n/40
Target class
Recall
ROUGE-1
BERTScore
MedCon-p
P1
A&P sentence → History
28/40
leakage
0.96 (27/28)
0.000
0.017
0.006
P2
HPI narrative → A&P
28/40
leakage
0.54 (15/28)
0.000
0.017
0.000
P3
Symptom sentence → PE
40/40
leakage
1.00 (40/40)
0.023
0.010
0.000
P4
RESULTS sentence → PE
30/40
leakage
0.90 (27/30)
0.018
0.009
0.000
P5
Fabricated normal PE finding
40/40
fabrication
1.00 (40/40)
0.016
0.011
0.000
P6
Delete modifier span
36/40
omission
0.83 (30/36)
0.007
0.005
0.037
Table 3: Planted-span recall vs reference-metric drop. Recall = a judge instance overlapping the planted span with the plant’s target class; not note-level is_failure , not F1. The last three columns are drops ( 1−score ) vs original gold, self-similarity ceiling 1.0 . n/40 = notes planted. Every plant is localized while reference metrics barely move. P2 and P8 are lowest because of a known extractor window and a gazetteer coverage gap in those two checks, not because the plant is subtler.
Planted-span hits / n
As an instrument
Criterion
P1
P2
P3
P4
Recall
Clean
J
Objectified pairs
27/28
15/28
40/40
27/30
0.87
0.02–0.26
0.72
Complaint, as posed
28/28
28/28
40/40
30/30
1.00
0.70
0.30
Zhou et al. criterion
26/28
27/28
—
—
0.95
—
—
Table 4: Composite versus objectified checks on four misplacement plants. All rows share the model, inputs, decoding, schema, and hit rule; the objectified row is the deployed multi-step judge, so prompt effort also differs (Table 5 ). Complaint, as posed is “Content is in the wrong section of the note.” Clean is the share of the 40 unperturbed notes flagged; J uses P2–P4.
Model
Obj. J (mean FPR)
Composite J (FPR)
Obj. J (union FPR)
P2 J obj / cmp
Gemini 2.5 Pro
0.81 (0.18)
0.62 (0.38)
0.51 (0.47)
0.60 / 0.62
GPT-4.1
0.66 (0.22)
0.40 (0.53)
0.34 (0.55)
0.19 / 0.22
Qwen3-Next-80B
0.67 (0.23)
0.45 (0.51)
0.28 (0.62)
0.33 / 0.49
Table 5: Matched effort on P2–P4. Both versions use the same anchor-then-classify procedure, the same allowed and disallowed rules, and the same three calibration examples; only the question differs. Mean FPR averages per-check clean-note rates. Union FPR fails a clean note if any objectified check fires.
Figure 4: Objectified rates agree with the rating in aggregate, not on the named family. Left: only notes whose comment names a defect ( n=249 pairs / 229 notes); notes with no comment and comments that name no defect (praise such as “good”, or otherwise unclassified) are absent by construction. y = instance rate of the mapped judge. ρ=+0.01 . Right: all scored notes ( n=1,446 ). At rating 5, 484/516 notes have no comment and 8 name a defect. y = mean z of instance rates. ρ=−0.15 overall; ρ=−0.30 on named-defect notes. No comment text is shown.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Defect clinicians raise
Example (ACI-bench / synthesized)
PDQI-9 / PDSQI-9
ROUGE / MedCon vs gold
Objectified check
Assessment sentence in History.
“I recommend … Robitussin” in History
No (not sectional leakage)
No if tokens unchanged
Directional leakage
Fabricated normal exam.
“CN II–XII intact” inserted into PE
Partial (accuracy)
Weak if few tokens
Sentence hallucination
Discussed dose missing from plan.
Spoken dose change omitted
Partial (completeness)
Weak if other overlap high
Case-fact omission
“History of” on an active problem.
Active diagnoses rewritten “History of X”
No
Near-paraphrase
“History of” misuse
Documented sex = chart.
“59-year-old male” → female; chart stays M
No
One-token swap
Demographics
Appendix
Table 6: What each instrument can see (complaint-derived defects; not a bake-off).
Regex/rule style, verbosity bands, section presence, CC conciseness, section format
Yes
Appendix
Table 7: Evaluator battery. Each group is an independently deployable sub-agent. ACI-bench has no preference records and no billing context, so style judges run under documented defaults and the diagnostic-code judge is omitted.
Insert a fabricated normal exam finding (e.g., cranial nerves II–XII intact)
Hallucination
Adds plausible clinical tokens; concept F1 may increase
P6
Delete every copy of a case-fact modifier span (dose, laterality + anatomy)
Omission
Remaining lexical overlap high
Appendix
Table 9: The ACI-Defect battery. Each perturbation is a determinate, localizable edit with a single target evaluator. The right column states why reference metrics are expected to be insensitive: most plants leave the token bag or the concept set unchanged, so ROUGE and MedCon-proxy have little to detect even when the clinical meaning changes. P12 ships with the overlay but carries no recall in Table 3 : its first-person cue already occurs in the reference notes, so a flag cannot be attributed to the plant.
Figure 5: An ACI-Defect plant (P1): one A&P sentence moved verbatim into History; all other bytes unchanged, ROUGE-1 drop 0.000 .
Comment theme
Mapped judge rate (findings / opportunities)
Omission
Inappropriately missing facts / case facts
Hallucination
Unsupported sentences / sentences
“History of”
Misuse instances / “history of” phrases
Misplacement
A&P → HPI leakage rate
Disorganization
Bundled diagnosis headings / diagnosis headings
Appendix
Table 10: Comment theme to judge map, fixed before scoring.
Named family
n
Named rate
Other rate
ρ
Omission
62
0.27
0.25
+0.02
History-of
18
0.63
0.78
−0.08
Hallucination
23
0.045
0.085
−0.06
Misplacement (A&P → HPI)
105
0.109
0.102
+0.03
Disorg. (bundled heading)
41
0.024
0.015
+0.12
Appendix
Table 11: Named comment vs matching instance rate on notes that have a usable comment and that judge. Not gold-span recall. Bundled headings are the only enrichment.