WHBench: Evaluating Frontier LLMs with Expert-in-the-Loop Validation on Women's Health Topics

Sneha Maurya; Spandana Govindgari; Girish Kumar; Akhara AI

cs.CLMar 11, 2026

WHBench: Evaluating Frontier LLMs with Expert-in-the-Loop Validation on Women's Health Topics

Authors: Sneha Maurya, Spandana Govindgari, Girish Kumar, Akhara AI

Organizations: Columbia University · Akhara AI

Abstract

Large language models are increasingly used for medical guidance, but women's health remains under-evaluated in benchmark design. We present the Women's Health Benchmark (WHBench), a targeted evaluation suite of 47 expert-crafted scenarios across 10 women's health topics, designed to expose clinically meaningful failure modes including outdated guidelines, unsafe omissions, dosing errors, and equity-related blind spots. We evaluate 22 models using a 23-criterion rubric spanning clinical accuracy, completeness, safety, communication quality, instruction following, equity, uncertainty handling, and guideline adherence, with safety-weighted penalties and server-side score recalculation. Across 3,102 attempted responses (3,100 scored), no model mean performance exceeds 75 percent; the best model reaches 72.1 percent. Even top models show low fully correct rates and substantial variation in harm rates. Inter-rater reliability is moderate at the response label level but high for model ranking, supporting WHBench utility for comparative system evaluation while highlighting the need for expert oversight in clinical deployment. WHBench provides a public, failure-mode-aware benchmark to track safer and more equitable progress in womens health AI.

Figures & tables

Failure Mode	$n$	Example
Missing information	14	Omitting follow-up for chronic anovulation
Factual / outdated	12	Pre-2017 TSH thresholds in pregnancy
Health equity gaps	4	Ignoring racial disparities in fibroids
Incorrect treatment	4	Surgery over IVF for bilateral tubal disease
Contraindication / dosage	6	Wrong folic acid dose with valproate
Other (urgency, dx, recs)	7	Not flagging post-retrieval fever

Table 1: Failure mode taxonomy. Each WHBench question targets one primary failure mode.

Category	Criterion	Type	+	$-$
A: Clinical Accuracy	A1 Core clinical conclusion	Bin	6	3
	A2 Numerical precision	Bin	5	2
	A3 Guideline alignment †	Ord	4	2
	A4 Factual error absence	Bin	4	3
B: Completeness	B5 Clinical considerations †	Ord	5	2
	B6 Differential diagnosis †	Ord	3	1

Table 2: WHBench v3.0 rubric: 23 criteria across 8 categories.

\dagger

= ordinal (3-level). Point values reflect clinical severity.

#	Model	Score	95% CI	C%	P%	I%	Harm%
1	Claude Opus 4.6	72.1	[69.6, 74.4]	35.5	58.2	6.4	12.8
2	Claude Sonnet 4.6	67.1	[64.5, 69.6]	22.7	67.4	9.9	27.0
3	GPT-5.4	66.8	[64.5, 69.2]	21.3	67.4	11.3	47.5
4	Gemini 3 Flash Preview	64.7	[61.7, 67.7]	25.5	62.4	12.1	32.6
5	OpenAI o3	63.6	[61.3, 65.9]	15.0	76.4	8.6	38.6
6	DeepSeek V3.2	61.3	[58.6, 63.9]	12.8	68.8	18.4	44.0

Table 3: WHBench v3.0 leaderboard. Mean normalized score (%) across 3 runs with 95% bootstrap CI (

n{=}10{,}000

). C/P/I = Correct / Partially Correct / Incorrect rates. Harm = percentage of responses where either C9a (unsafe commission) or C9b (unsafe omission) failed.

Figure 1: Model performance on WHBench v3.0. Mean normalized score (%) with 95% bootstrap confidence intervals ( $n{=}10{,}000$ ). The dashed line marks the 80% threshold for “Correct” classification. Claude Opus 4.6 comes close at 72.1%; most frontier models cluster in the low to mid 60s.

Figure 2: Model Safety performance on WHBench v3.0. Mean normalized score (%) with Safety Category Mean Pass rate(%) ( $n{=}10{,}000$ ). The dashed line marks the median overall score on x-axis and median safety on y-axis. Only two models - Claude Opus 4.6 and Claude Sonnet 4.6 passed the safety ; rest of the latest SOTA models cluster in 80 to 90% band.

Figure 3: Model $\times$ topic performance heatmap (mean normalized score %). Darker shading indicates higher scores. Pregnancy , Cancer Screening and Hormonal Health show high cross-model variance; Contraception is uniformly difficult.

Metric	$\kappa$ / $\rho$	Agreement
Overall label (C/P/I)	$\kappa=0.238$	51.6%
Category-level (8 dims)	$\kappa=0.538$	—
Spearman rank correlation	$\rho=0.916$	—
Pearson score correlation	$r=0.611$	—
Highest category agreement
E Instruction Follow	$\kappa=0.641$	85.7%

Table 4: Inter-rater reliability between Claude Sonnet 4.6 (primary judge) and GPT-5.4 (secondary).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Access	API Identifier
GPT-5.4	API	openai/gpt-5.4
GPT-4o	API	openai/gpt-4o
GPT-4.1	API	openai/gpt-4.1
OpenAI o3	API	openai/o3
Claude Sonnet 4.6	API	anthropic/claude-sonnet-4.6
Claude Opus 4.6	API	anthropic/claude-opus-4.6

Appendix

Table 5: Model configuration. All evaluated at temperature

T=0

, max tokens

=4{,}096

, with 3 independent runs per question. All evaluations conducted March 2026.

ID	Specialty	Credentials	Experience
E1	Obstetrics & Gynecology	MBBS, DNB, MS, MD(US) MRCOG(UK)	20 years
E2	Orthopaedic Surgery / General Medicine	MBBS, MS, MD(US)	10 years
E3	Fertility Nursing (Dandi Fertility)	BSN, RN	8 years
E4	Gynecologic Oncology	MD	10 years
E5	Gynecologic surgeons	MBBS, MS	11 years

Appendix

Table 6: Expert panel credentials. All experts answered independently without access to other experts’ responses or model outputs. Multiple experts from same speciality answered the questions. Below is coverage of the specialities and their average years of experience.

Clinical Language Model Evaluation

Clinical language model evaluation measures whether AI systems are accurate, safe, fair, and useful across medical tasks and care settings.

See all 418 papers

The Validity Gap in Health AI Evaluation: A Cross-Sectional Analysis of Benchmark Composition
Mar 18, 2026Alvin Rajkomar, Pavan Sudarshan, Angela Lai +1Benchmark Validity Clinical Language Model Evaluation
Apple
Health-ORSC-Bench: A Benchmark for Measuring Over-Refusal and Safety Completion in Health Context
Jan 25, 2026Zhihao Zhang, Liting Huang, Guanghao Wu +3LLM Safety Benchmarks Language Model Safety Evaluation
Macquarie University1 · UTS2 · MBZUAI3 +1
HealthBench-Psych: A Mental Health Subset of OpenAI's HealthBench
Aug 25, 2026Matthew Flathers, Phuong Anh Nguyen, Jill Noorily +7Clinical Language Model Evaluation LLM Evaluation
Division of Digital Psychiatry, Beth Israel Deaconess Medical Center, Boston, MA · Department of Psychiatry, Beth Israel Deaconess Medical Center, Boston, MA

Mar 18, 2026J/K move · Enter open · S save

The Validity Gap in Health AI Evaluation: A Cross-Sectional Analysis of Benchmark Composition

Alvin Rajkomar, Pavan Sudarshan, Angela Lai, Lily Peng

Apple

Benchmark Validity Clinical Language Model Evaluation Medical LLMs

Open paper AskPDF arXiv

Background: Clinical trials rely on transparent inclusion criteria to ensure generalizability. In contrast, benchmarks validating health-related large language models (LLMs) rarely characterize the "patient" or "query" populations they contain. Without defined composition, aggregate performance metrics may misrepresent model readiness for clinical use. Methods: We analyzed 18,707 consumer health queries across six public benchmarks using LLMs as automated coding instruments to apply a standardized 16-field taxonomy profiling context, topic, and intent. Results: We identified a structural "validity gap." While benchmarks have evolved from static retrieval to interactive dialogue, clinical composition remains misaligned with real-world needs. Although 42% of the corpus referenced objective data, this was polarized toward wellness-focused wearable signals (17.7%); complex diagnostic inputs remained rare, including laboratory values (5.2%), imaging (3.8%), and raw medical records (0.6%). Safety-critical scenarios were effectively absent: suicide/self-harm queries comprised <0.7% of the corpus and chronic disease management only 5.5%. Benchmarks also neglected vulnerable populations (pediatrics/older adults <11%) and global health needs. Conclusions: Evaluation benchmarks remain misaligned with real-world clinical needs, lacking raw clinical artifacts, adequate representation of vulnerable populations, and longitudinal chronic care scenarios. The field must adopt standardized query profiling--analogous to clinical trial reporting--to align evaluation with the full complexity of clinical practice.

Abstract

Clinical Language Model Evaluation

Explore similar work

The Validity Gap in Health AI Evaluation: A Cross-Sectional Analysis of Benchmark Composition

Health-ORSC-Bench: A Benchmark for Measuring Over-Refusal and Safety Completion in Health Context

HealthBench-Psych: A Mental Health Subset of OpenAI's HealthBench