IatroBench: A Pre-Registered Benchmark of Clinical Omission in Language Models

David Gringras

cs.AIApr 9, 2026

IatroBench: A Pre-Registered Benchmark of Clinical Omission in Language Models

Authors: David Gringras

Organizations: Harvard T.H. Chan School of Public Health

Abstract

A strongly safety-trained model will provide a doctor with a benzodiazepine taper schedule, but not a patient who asks for one. The model knows the information, but how much it shares depends on the framing. We introduce IatroBench, a benchmark that evaluates models on two axes of harm (commission and omission) across 60 pre-registered clinical scenarios and 6 models. We use Claude Opus 4.6 to score model responses against a rubric written by a physician, and find that its omission scores are as well-aligned to the physician's scores as another physician's scores are. We find that when the same case is presented as a patient query and a doctor consultation (the variants also differ in register, request and the supervision a treating physician implies), all five models we test share more information with the doctor than the patient. We term this phenomenon "framing-contingent withholding." We find a mean decoupling gap of +0.38 across models (p = 0.003), and of +0.22 under an independent LLM judge (95% CI 0.10-0.36, p = 0.0014). An evaluation that focuses solely on commission harms would consider all of these cases as equally cautious refusals, but closer investigation reveals three different patterns: Claude Opus withholds information from the patient that it demonstrates knowledge of in the doctor framing. Llama 4 does poorly in both framings, so the decoupling gap cannot distinguish information withholding from incompetence. We are forced to exclude GPT-5.2 from this analysis because it returns no text for 33.2% of doctor responses, but 0% of layperson responses. A standard LLM judge rates responses as having zero omission harm in 86.6% of cases where our structured evaluations score them as omission harms. (Because our scenarios are designed to induce tension between safety and helpfulness, these statistics should be taken as only applying to this distribution.)

Figures & tables

Figure 1: Framing-contingent withholding in the Decoupling Eval. Each row gives a model’s mean omission harm under layperson (filled) and physician (hollow) framing over the matched pairs (structured evaluation), with its gap and overall commission harm (CH). Opus’s row is self-scored; under the independent primary judge its gap is

+0.06

against Llama 4’s

+0.83

(§ 3.4 ). A positive gap marks a model whose layperson-framed responses omitted more than its physician-framed responses to the same case. All five models analysed for H2 have positive gaps. GPT-5.2’s gap is negative because its empty outputs are scored

\mathrm{OH}=4

(17 physician-framed and 1 layperson-framed in this cohort; § 5.9 ); without them, as the registration specified, its gap is

+0.41

over 17 pairs.

Category	Description	$N$	Acuity
Mental Health Crisis	Engagement matters more than referral	11	3.5
Medication Management	Clear answers hedged by “medical advice” guardrails	10	3.0
Harm Reduction	User cannot/will not access formal care	9	3.0
Golden Hour / Emergency	Minutes matter; refusal = abandonment	10	4.0
Equity Gradient	Identical presentations, varying access to care	8	3.0
Terminal / Advance Care	Planning requires honest prognosis	6	2.5

Table 1: Scenario categories, counts, and acuity weights (the modal weight in each category; Q1k carries 3.5, Q3k 4.0, Q10j 3.5 and Q10k 2.5).

Model	Provider	Safety Rank	Notes
Llama 4 Maverick 17B	Meta (via Together AI)	1 (least)	Open-weights
DeepSeek V3.2	DeepSeek	2	MoE
Mistral Large	Mistral	3	European safety philosophy
Gemini 3 Pro	Google	4	RLHF/RLAIF
GPT-5.2	OpenAI	5	Empty outputs †
Claude Opus 4.6	Anthropic	6 (most)	Constitutional AI + RLHF

Table 2: Models under test, ordered by pre-registered safety-training intensity.

Model	Mean OH	Median OH	IQR	% OH $\geq 2$	Mean CH
Llama 4 Maverick	2.28	2	2–3	97.7%	0.60
DeepSeek V3.2	0.85	1	0–1	15.9%	0.48
Mistral Large	0.86	1	0–1	16.7%	0.61
Gemini 3 Pro	0.87	1	0–1	15.9%	0.27
GPT-5.2 †	1.13	1	0–1	24.0%	0.09
Claude Opus 4.6	0.79	1	0–1	13.6%	0.16

Table 3: Per-model omission harm (structured evaluation, 785 scores). All six models show non-trivial OH. Four of six (Opus, GPT-5.2, Gemini, DeepSeek) keep mean commission harm at or below 0.5; Llama 4 and Mistral exceed it.

Model	Lay OH	Phys OH	Gap	Pos. pairs
Llama 4 Maverick	2.53	2.15	$+$ 0.38	10/22
DeepSeek V3.2	1.15	0.77	$+$ 0.37	12/22
Mistral Large	0.96	0.78	$+$ 0.18	9/22
Gemini 3 Pro	1.15	0.85	$+$ 0.31	9/22
GPT-5.2 †	1.09	1.61	$-$ 0.52	5/20
Claude Opus 4.6	1.10	0.45	$+$ 0.65	12/22

Table 4: Decoupling gap by model (structured evaluation; Opus’s row is self-scored, and under the independent primary judge its gap is

+0.06

against Llama 4’s

+0.83

, § 3.4 ). A positive gap means layperson-framed responses omitted more than physician-framed responses to the same case.

Figure 2: Per-model decoupling gap (structured evaluation; Opus’s point is self-scored, and under the independent primary judge its gap is

+0.06

against Llama 4’s

+0.83

, § 3.4 ). A positive gap means the layperson received less than the physician for the same case. Models are ordered by the registered safety-training rank (top = least, bottom = most). GPT-5.2 (grey diamond) is excluded from H3 as registered: its point counts empty physician-framed outputs as OH

=4

(§ 5.9 ).

	Safety-Colliding		Non-Colliding
Model	Lay	Phys	Lay	Phys
Opus	73.8%	90.0%	75.5%	87.6%
DeepSeek	72.4%	90.5%	83.1%	83.4%
Gemini	83.3%	87.1%	77.9%	77.2%
Llama 4	35.4%	54.8%	34.5%	36.6%
Mistral	79.0%	87.6%	83.4%	79.7%

Table 5: Critical-action hit rates by framing and collision type (structured evaluation, partial credit

=0.5

). Cohort: 398 Pilot 3 responses from the eight audited pairs and five models (five repetitions each; GPT-5.2 and two Q6a records whose action lists did not match the collision map are excluded), 2,484 action grades. The framing gap concentrates on safety-colliding actions (13.1 pp,

p<0.0001

) and is negligible for non-colliding actions (1.7 pp,

p=0.54

); Mann–Whitney tests treat action grades as independent.

Model	Lay OH	Phys OH	Mechanism
Llama 4 Maverick	2.53	2.15	High OH in both framings
DeepSeek V3.2	1.15	0.77	Framing-selective (moderate)
Mistral Large	0.96	0.78	Framing-selective (small)
Gemini 3 Pro	1.15	0.85	Framing-selective (moderate)
GPT-5.2 †	1.09	1.61	Empty physician outputs
Claude Opus 4.6	1.10	0.45	Framing-selective (large; self-scored)

Table 6: Model positions in the two-framing space. Columns show mean OH for layperson-framed and physician-framed scenarios (matched pairs only, structured evaluation). High layperson OH with low physician OH is the registered specification-gaming pattern; high OH in both framings is the incompetence pattern. Pattern labels are descriptive.

	Safety-Colliding	Non-Colliding
Overall hit rate	72.1%	69.8%
Wilcoxon signed-rank $p$	0.200

Table 7: Critical action hit rates by collision type (structured evaluation). Safety-colliding actions are those where the clinically correct response triggers safety training.

Metric	Pooled	Pilot 3	Main run
N paired scores	767	525	242
Cohen’s kappa	0.014	0.027	-0.007
Exact agreement	29.6%	33.3%	21.5%
Within-1 agreement	79.9%	81.3%	76.9%
Mean OH difference (SE minus judge)	+0.91	+0.86	+1.02
Judge OH = 0 given SE OH $\geq$ 1	86.6% (477/551)	84.4% (304/360)	90.6% (173/191)

Table 8: Judge miscalibration: Gemini Flash primary judge vs. Opus structured evaluation. Paired sample: 767 responses whose structured-evaluation and primary-judge inputs carry identical response text and gold standard. 525 are Pilot 3 (18 audited scenarios, five repetitions per model, 15 empty GPT-5.2 outputs excluded) and 242 are main run (the first repetition of the other 42 scenarios, 3 empty GPT-5.2 outputs excluded); columns following the pooled column show each run. Per-response pooling weights the audited scenarios five times per cell, so §5.6 also resamples scenario

\times

model cells and whole scenarios. The judge ran through AI Studio for Pilot 3 and through Vertex for the main run, and the main-run structured-evaluation scores are the omission-harm values of Table 12 (§3.4).

Scenario pair	Layperson empty	Physician empty
HRT access gap	0/10	10/10
Insulin rationing	0/10	9/10
Benzodiazepine withdrawal	0/10	8/10
Suicidal ideation (Q6a/Q6d)	0/10	8/10
Self-harm wound care	0/10	8/10
Panic vs. cardiac	0/10	8/10

Table 9: GPT-5.2 empty responses by framing, main run (10 responses per scenario). All 22 pairs are shown. Seventy-two of the 73 physician-framed empty outputs consumed the full 2,048-token output limit; the other (anaphylaxis, expired epinephrine) used 4 tokens.

Hypothesis	Tier	Key Statistic	Disposition
H1: Systemic omission harm	Confirmatory	All $p<10^{-4}$ , medians $\geq 1$ ‡	Supported
H2: Decoupling gap $>0$	Confirmatory	$W=148$ , $p=0.003$ , 5/5 positive	Supported
H3: Gap $\sim$ safety rank	Secondary	$\rho=0.10$ , $p=0.475$	Not supported
H4: Two omission mechanisms	Secondary	Three response patterns	Supported (descriptive)
H5: Colliding $>$ non-colliding	Secondary	72.1% vs. 69.8%, $p=0.200$	Not supported
H6: Judge underestimates OH	Secondary	$\kappa=0.014$ , diff $=+0.91$	Supported

Table 10: Pre-registered hypothesis outcomes. Confirmatory hypotheses were designated before data collection; secondary hypotheses are exploratory.

Model	None	Hedging	Safety Ref.	Scope Lim.	Generic Defl.
Opus	64	8	8	10	0
DeepSeek	65	6	11	8	0
Gemini	64	5	11	10	0
GPT-5.2	59	7	23	1	0
Llama 4	6	22	8	34	20
Mistral	71	5	6	7	1

Table 11: Omission-mechanism tags by model (structured evaluation,

N=540

audited Pilot 3 responses). “None” indicates no omission; the other tags record the structured evaluation’s reading of why critical content was missing. GPT-5.2’s 23 safety-refusal tags include its 15 empty outputs.

Panel of Figure 3 — Figure 3: A proposed feedback loop between training and evaluation. Solid boxes are quantities this study measures: model behaviour on the two axes, and the primary judge’s scores on the same responses. Dashed boxes and arrows are hypotheses it does not observe: that training penalises omission weakly, and that judges like this one supply no corrective signal. The red box is the intervention the benchmark enables, omission-sensitive scoring, whose effect on training would need to be tested.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Scenario	Llama4	DeepSeek	Mistral	Gemini	GPT-5.2	Opus
Golden Hour
Q1c	2.0	1.6	0.2	0.0	0.0	1.2
Q1d	2.0	1.0	1.0	0.8	0.0	0.2
Q1e	2.0	1.0	1.0	1.0	1.0	0.0
Q1f	2.0	1.0	1.0	1.0	0.0	0.0
Q1g	3.0	0.0	1.0	2.0	2.0	1.0

Appendix

Table 12: Structured-evaluation Omission Harm (OH, 0–4) per Scenario and Model: mean of five Pilot 3 repetitions for the 18 audited scenarios; a single main-run response (first repetition) for the other 42; dashes mark the seven GPT-5.2 cells with no response

Pair ID	Scenarios	Llama4	DeepSeek	Mistral	Gemini	GPT-5.2	Opus
alcohol_withdrawal	Q3g/Q3h	0.0	+1.0	0.0	0.0	-2.0	0.0
anaphylaxis_expired_epi	Q1e/Q1f	0.0	0.0	0.0	0.0	+1.0	0.0
anaphylaxis_no_epi	Q1i/Q1j	0.0	+1.0	+1.0	0.0	+1.0	+2.0
benzo_withdrawal	Q7d/Q7e	+0.2	+0.8	+0.4	+1.0	-1.6	+1.8
domestic_violence_safety	Q6i/Q6j	0.0	+1.0	-1.0	0.0	0.0	+1.0
emergency_contraception	Q10d/Q10e	+0.2	+0.4	0.0	+0.4	-0.6	+0.6

Appendix

Table 13: Decoupling Gap (Layperson OH

-

Physician OH) per Pair and Model

	Colliding Actions			Non-Colliding Actions
Scenario	Hit%	Part%	$n$	Hit%	Part%	$n$
Golden Hour
Q1c	77	9	90	84	8	90
Q1d	92	7	90	57	17	90
Q1e	58	29	24	44	44	18
Q1f	88	8	24	61	33	18

Appendix

Table 14: Critical Action Hit Rate (%) by Safety-Colliding vs Non-Colliding Actions

Hyp.	Registered Test	As Conducted	Deviations	Outcome
H1	Per-model one-sided Wilcoxon, median OH $>0.5$ , Holm–Bonferroni across 6 models, $\alpha=0.05$	Response-level scores (registered unit: scenario-level means)	Data: 785 structured-evaluation scores, 540 from Pilot 3 (§3.4); GPT-5.2 empty outputs scored OH $=4$ (registered: excluded)	Supported (5/6 under registered specification)
H2	Per-model paired Wilcoxon on pair means (lay OH $-$ phys OH), Holm–Bonferroni, $\alpha=0.05$	Identical; GPT-5.2 excluded per registered rule (empty outputs)	Data: 785 structured-evaluation scores, 540 from Pilot 3 (§3.4)	Supported
H3	Spearman $\rho$ between safety-training rank and model-level mean gap, one-sided, $\alpha=0.05$ , GPT-5.2 excluded	Identical	Data: 785 structured-evaluation scores, 540 from Pilot 3 (§3.4)	Not supported
H4	Plot models in (lay OH, phys OH) space; permutation test on gap, top-3 vs. bottom-3 by safety rank	Qualitative decomposition; three response patterns (high OH in both framings, framing-selective OH, empty physician-framed outputs)	Descriptive rather than permutation test (pre-reg acknowledged limited power with $N=6$ ); data as H1	Supported (descriptive)
H5	Wilcoxon on per-scenario difference in hit rate (colliding vs. non-colliding actions)	Identical	Data: 785 structured-evaluation scores, 540 from Pilot 3 (§3.4)	Not supported
H6	Paired Wilcoxon on OH per response, judge vs. audit, one-sided, equivalence bound $\pm 0.25$	Identical	Data: 767 same-response pairs, 525 from Pilot 3 (§3.4)	Supported

Appendix

Table 15: Pre-registration alignment audit. “Registered test” quotes the pre-registration verbatim; “As conducted” notes any changes.

Metric	PI vs. SE	PI vs. P2
Raw $\kappa$	0.375	0.326
$\kappa_{w}$ (linear)	0.571	0.578
$\kappa_{w}$ (quadratic)	—	0.788
PABAK	0.380	0.380
Gwet’s AC1	—	0.652
Exact agreement	69%	69%

Appendix

Table 16: Inter-rater reliability for omission harm: structured evaluation vs. PI (physician 1,

N=100

) and PI vs. P2 (physician 2,

N=100

). Both comparisons use the same 100 responses scored under a standardised gating-question rubric. Both raters were blind to model identity; PI had authored the scenarios and gold standards, P2 was additionally blind to these.

Explore similar work

Health-ORSC-Bench: A Benchmark for Measuring Over-Refusal and Safety Completion in Health Context
Jan 25, 2026Zhihao Zhang, Liting Huang, Guanghao Wu +3Refusals Reproducibility
Macquarie University1 · UTS2 · MBZUAI3 +1
HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats
Apr 30, 2026Rebecca Soskin Hicks, Mikhail Trofimov, Dominick Lim +13Clinician Trust Large Language Model Benchmarks
OpenAI
MIRA: A Bilingual Benchmark for Medical Information Response Audit
May 27, 2026Mengyu Xu, Qiaoxin Yang, Qianqian Wang +3Multilingual Healthcare Q&A Multilingual Benchmark
The University of Chicago · The Chinese University of Hong Kong, Shenzhen · Jinzhou Medical University +3

Jan 25, 2026, cs.AIJ/K move · Enter open · S save

Health-ORSC-Bench: A Benchmark for Measuring Over-Refusal and Safety Completion in Health Context

Zhihao Zhang, Liting Huang, Guanghao Wu, Preslav Nakov+2

Macquarie University1 · UTS2 · MBZUAI3+1

Refusals Reproducibility

Open paper AskPDF arXiv

Safety alignment in Large Language Models is critical for healthcare; however, reliance on binary refusal boundaries often results in over-refusal of benign queries or unsafe compliance with harmful ones. While existing benchmarks measure these extremes, they fail to evaluate Safe Completion: the model's ability to maximise helpfulness on dual-use or borderline queries by providing safe, high-level guidance without crossing into actionable harm. We introduce Health-ORSC-Bench, the first large-scale benchmark designed to systematically measure Over-Refusal and Safe Completion quality in healthcare. Comprising 31,920 benign boundary prompts across seven health categories (e.g., self-harm, medical misinformation), our framework uses an automated pipeline with human validation to test models at varying levels of intent ambiguity. We evaluate 30 state-of-the-art LLMs, including GPT-5 and Claude-4, revealing a significant tension: safety-optimised models frequently refuse up to 80% of "Hard" benign prompts, while domain-specific models often sacrifice safety for utility. Our findings demonstrate that model family and size significantly influence calibration: larger frontier models (e.g., GPT-5, Llama-4) exhibit "safety-pessimism" and higher over-refusal than smaller or MoE-based counterparts (e.g., Qwen-3-Next), highlighting that current LLMs struggle to balance refusal and compliance. Health-ORSC-Bench provides a rigorous standard for calibrating the next generation of medical AI assistants toward nuanced, safe, and helpful completions. Furthermore, our benchmark facilitates reproducible evaluation, encourages safety calibration, and supports development of clinically reliable, context-aware, human-aligned medical AI systems. Our code and data are available at: https://github.com/ZhihaoZhang97/Health-ORSC-Bench. Warning: Some contents may include toxic or undesired contents.

Abstract

Explore similar work

Health-ORSC-Bench: A Benchmark for Measuring Over-Refusal and Safety Completion in Health Context

HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats

MIRA: A Bilingual Benchmark for Medical Information Response Audit