The Validity Gap in Health AI Evaluation: A Cross-Sectional Analysis of Benchmark Composition

Alvin Rajkomar; Pavan Sudarshan; Angela Lai; Lily Peng

cs.AIMar 18, 2026

The Validity Gap in Health AI Evaluation: A Cross-Sectional Analysis of Benchmark Composition

Authors: Alvin Rajkomar, Pavan Sudarshan, Angela Lai, Lily Peng

Organizations: Apple

Abstract

Background: Clinical trials rely on transparent inclusion criteria to ensure generalizability. In contrast, benchmarks validating health-related large language models (LLMs) rarely characterize the "patient" or "query" populations they contain. Without defined composition, aggregate performance metrics may misrepresent model readiness for clinical use. Methods: We analyzed 18,707 consumer health queries across six public benchmarks using LLMs as automated coding instruments to apply a standardized 16-field taxonomy profiling context, topic, and intent. Results: We identified a structural "validity gap." While benchmarks have evolved from static retrieval to interactive dialogue, clinical composition remains misaligned with real-world needs. Although 42% of the corpus referenced objective data, this was polarized toward wellness-focused wearable signals (17.7%); complex diagnostic inputs remained rare, including laboratory values (5.2%), imaging (3.8%), and raw medical records (0.6%). Safety-critical scenarios were effectively absent: suicide/self-harm queries comprised <0.7% of the corpus and chronic disease management only 5.5%. Benchmarks also neglected vulnerable populations (pediatrics/older adults <11%) and global health needs. Conclusions: Evaluation benchmarks remain misaligned with real-world clinical needs, lacking raw clinical artifacts, adequate representation of vulnerable populations, and longitudinal chronic care scenarios. The field must adopt standardized query profiling--analogous to clinical trial reporting--to align evaluation with the full complexity of clinical practice.

Figures & tables

Element	Description
Context Dimensions
Population	Age-specific demographic (pediatric, adult, older adult), derived from user-provided context cues.
Conversation Type	Single question versus multi-turn conversation.
Length Detail	Short text string versus extended narrative.
Objective Data	Presence of quantitative health data (labs, vitals, imaging).
Context Depth	Information richness (age, timeline, clinical anchors).

Table 1: Query Profile Taxonomy: Elements and Descriptions

Figure 1: CONSORT-style flow diagram showing study progression from initial dataset assessment through tagging methods to final analytic cohort of 18,707 consumer health queries.

Figure 2: Demographic and geographic representation gaps. (A) Age distribution: benchmarks default to a “standard adult” (89%), while older adults ( $>$ 65 years)—who have the highest office-based physician visit rate (550 per 100 persons)—represent only 4.6% of benchmark queries, and pediatric populations are similarly underrepresented. Visit rates from NCHS Data Brief No. 408 (2021). The visit rate for adults aged 18–64 (228 per 100 persons) is a derived aggregate of the 18–44 and 45–64 age brackets provided in the original NCHS brief. (B) Global disease representation: despite causing over 1.8 million deaths annually (WHO Global TB Report 2024; WHO World Malaria Report 2024), malaria ( $N=39$ ) and tuberculosis ( $N=25$ ) are nearly absent from benchmarks, while COVID-19 ( $N=365$ ) receives disproportionate coverage relative to its current global mortality burden.

Figure 3: Benchmark intent distribution compared to real-world office-based physician visit reasons (NCHS Data Brief No. 408, 2021). Chronic disease management accounts for 39% of physician visits but only 5.5% of benchmark queries—a 7-fold gap. Prevention and screening shows an even larger disparity (23% vs. 2.7%). Only acute symptom evaluation is proportionally represented in benchmarks (24.2% vs. 24.0%). Benchmark percentages do not sum to 100% because additional intent categories (e.g., education, non-health) are not shown.

Figure 4: Behavioral health crisis scenarios by benchmark generation. Crisis conditions (suicidal ideation, self-harm, substance use) were rare overall ( $<$ 0.7% of corpus) and concentrated almost entirely in Generation 2 forum narratives. Generation 3 conversational benchmarks—the modality where chatbots actually operate—contained fewer than 20 crisis queries combined, leaving crisis de-escalation in dialogue settings effectively untested. Broader mental health conditions (anxiety, depression) showed the same generational skew.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Tag Field	Assigned Value
user_type	consumer
conversation_structure	single_turn
intent_top	general_health_advice
intent_sub	fitness_exercise (fitness) or sleep_hygiene (sleep)
topic_area_path	[ Holistic Health & Wellness , Sleep & Lifestyle ]
specialty	sports_medicine (fitness) or sleep_medicine (sleep)

Appendix

Table S1: Fixed tag assignments for GoogleFitbit datasets reflecting wearable data context

Domain	Dimensions	Avg Agreement	Avg $\kappa$
Context	16	92.6%	0.77
Topic	3	83.7%	0.79
Intent	2	78.5%	0.76

Appendix

Table S2: Agreement by Query Profile Domain

Domain	Dimension	Agreement	$\kappa$
Context
	Conversation Structure	100.0%	1.00
	Language	99.8%	0.99
	Population	98.6%	0.95
	Raw Medical Text	97.8%	0.71
	Region	96.9%	0.77

Appendix

Table S3: Agreement Metrics by Dimension

Figure S1: Distribution comparison between GPT-5.2 and Opus-4.5 across key dimensions. Bars show percentage of queries assigned to each category.

Figure S2: Confusion matrices showing agreement patterns for intent, risk sensitivity, and specialty. Values show row-normalized percentages. Strong diagonal dominance indicates high agreement.

Dimension	$n$	Mean Likert	% Reasonable	95% CI
Risk Sensitivity	128	4.88	96.9%	[92–99%]
Top-Level Intent	124	4.84	96.8%	[92–99%]
Topic Area (Level 1)	136	4.90	97.1%	[93–99%]

Appendix

Table S4: Clinician-judged reasonableness of GPT-5.2 tags (Likert

\geq 4

= reasonable). Each cell pools both reviewers across 80 queries.

Dimension	$n$	Both $\geq 4$	Exact Likert	Within 1 pt
Risk Sensitivity	64	93.8%	82.8%	93.8%
Top-Level Intent	62	95.2%	83.9%	96.8%
Topic Area (Level 1)	68	94.1%	86.8%	94.1%

Appendix

Table S5: Inter-rater agreement between two physician reviewers on the reasonable/not-reasonable binary (

\geq 4

threshold).

Dimension	GPT pref.	Opus pref.	Both OK	Sign $p$
Risk Sensitivity	18	12	2	0.362
Top-Level Intent	17	14	5	0.720
Topic Area (Level 1)	10	12	2	0.832

Appendix

Table S6: Model preference on disagreement cases (blinded). “GPT pref.” and “Opus pref.” count queries where the respective model’s tag was preferred; “Both OK” indicates both tags were acceptable with no preference. Sign-test

p

-values test

H_{0}

: equal preference.

Figure S3: The Evolution of the Validity Gap. Each generation of benchmarks advances structural capabilities but retains specific validity gaps. Generation 1 evaluates fact retrieval but lacks acuity for triage (1.2% triage intent, $<$ 2% high-risk). Generation 2 introduces clinical reasoning with rich objective data but relies on synthesized, static narratives (100% single-turn) where data is pre-packaged rather than elicited. Generation 3 achieves interactivity but suffers from sparse clinical content: $<$ 0.7% behavioral crisis, 5.5% chronic care, and clinical data density drops compared to Generation 2. Critically, demographic bias persists across all generations: pediatric and geriatric populations represent $<$ 11% of queries, validating models on a “standard adult default.”

Figure S4: Distribution of health topics and query intents across three generations of benchmarks. (A) Health topics: Generation 1 (Search Era) queries span common symptoms and general health concerns. Generation 2 (Case Presentation Era) concentrates in internal medicine subspecialties with higher clinical complexity. Generation 3 (Interactive & Data Era) shows bimodal distribution: HealthBench Main covers diverse acute and chronic conditions, while GoogleFitbit datasets focus predominantly on wellness, sleep, and fitness tracking. (B) Query intents: Generation 1 benchmarks are dominated by queries to learn about health topics (92%), while Generation 3 benchmarks capture broader intent diversity including wellness and lifestyle guidance, medical research queries, and prevention and screening.

Dimension/Category	Count (%)
1. Intent Distribution
Education	2,645 (83.4%)
Education Explainer	2,635 (99.6%)
Basic Science	10 (0.4%)
Symptom Check	380 (12.0%)
Management Plan	235 (61.8%)

Appendix

Table 16

Dimension/Category	Count (%)
1. Intent Distribution
Education	2,389 (68.5%)
Education Explainer	2,250 (94.2%)
Basic Science	139 (5.8%)
Medication Information	332 (9.5%)
Side Effects	157 (47.3%)

Appendix

Table 17

Dimension/Category	Count (%)
1. Intent Distribution
Symptom Check	2,656 (52.3%)
Differential Diagnosis	1,235 (46.5%)
Management Plan	774 (29.1%)
Triage Disposition	647 (24.4%)
Tests and Results	702 (13.8%)

Appendix

Table 18

Dimension/Category	Count (%)
1. Intent Distribution
Symptom Check	1,384 (37.5%)
Management Plan	681 (49.2%)
Triage Disposition	392 (28.3%)
Differential Diagnosis	311 (22.5%)
Medication Information	729 (19.8%)

Appendix

Table 19

Dimension/Category	Count (%)
1. Intent Distribution
General Health Advice	1,521 (100.0%)
Sleep Hygiene	1,521 (100.0%)
2. Topic Distribution (Top 5)
Holistic Health & Wellness	1,521 (100.0%)
Sleep & Lifestyle	1,521 (100.0%)

Appendix

Table 20

Dimension/Category	Count (%)
1. Intent Distribution
General Health Advice	1,750 (100.0%)
Fitness/Exercise	1,750 (100.0%)
2. Topic Distribution (Top 5)
Holistic Health & Wellness	1,750 (100.0%)
Sleep & Lifestyle	1,750 (100.0%)

Appendix

Table 21

Benchmark Validity

Benchmark validity concerns whether evaluation tasks, scores, and rankings support the capability claims researchers make about AI systems.

See all 174 papers

Clinician use of language models diverges from how the models are evaluated
Oct 8, 2026Krithik Vishwanath, Haitong Lin, Anton Alyakin +19Clinical Language Model Evaluation Medical LLMs
Department of Neurosurgery, NYU Langone Health, New York, NY, USA · Department of Technology Management, NYU Tandon School of Engineering, New York University, New York, NY, USA · Washington University School of Medicine, St. Louis, MO, USA +18
WHBench: Evaluating Frontier LLMs with Expert-in-the-Loop Validation on Women's Health Topics
Mar 11, 2026Sneha Maurya, Spandana Govindgari, Girish Kumar +1Clinical Language Model Evaluation LLM Safety Benchmarks
Columbia University · Akhara AI
Addressing Benchmarking Gaps in Large Language Models for Health and Medicine with Dynamic Red-Teaming
Jul 30, 2025Jiazhen Pan, Bailiang Jian, Paul Hager +20LLM Safety Benchmarks Language Model Safety Evaluation
Cherise · School of Computation, Information and Technology, Technical University of Munich (TUM), Munich, Germany · Department of Engineering Science, University of Oxford, Oxford, United Kingdom +9

Oct 8, 2026J/K move · Enter open · S save

Clinician use of language models diverges from how the models are evaluated

Krithik Vishwanath, Haitong Lin, Anton Alyakin, Jin Vivian Lee+18

Department of Neurosurgery, NYU Langone Health, New York, NY, USA · Department of Technology Management, NYU Tandon School of Engineering, New York University, New York, NY, USA · Washington University School of Medicine, St. Louis, MO, USA+18

Clinical Language Model Evaluation Medical LLMs Benchmark Validity

Open paper AskPDF arXiv

Large language model (LLM) assistants are being deployed to clinicians across health systems, and judgments about their readiness rest largely on benchmark scores, most of them derived from examination questions or curated cases. A benchmark predicts performance in deployment only to the extent that its items resemble real use, yet whether benchmarks reflect the work these systems receive has rarely been measured. Here we analyze 127,833 queries sent by 6,342 physicians, advanced practice providers and nurses in 35 specialties to an institutional assistant during an eight-month roll-out. We characterize each query with RCQ-Map, a clinician-validated framework grounded in taxonomies of clinical questions and of LLM evaluation, which records its task, intent, answerability, missing information and potential harm. Documentation and administration (36.2%) and knowledge retrieval (28.9%) made up nearly two-thirds of use, and diagnosis 3.7%; more than a third of queries could not be answered well as posed. Applying RCQ-Map to 58 public benchmarks drawn from major evaluation suites and frontier model reports, which we assemble into the Clinical AI Benchmark Atlas, showed that the median benchmark contained no documentation requests and shared 31% of the task mix of real use, less than an even spread across task categories would. Benchmarks in suites designed to resemble clinical practice were individually no closer to real use than those used in frontier model reports. Benchmark scores therefore say little about how clinical AI performs on most of the work it is actually given, and evaluation should be matched to real clinical use.

Abstract

Benchmark Validity

Explore similar work

Clinician use of language models diverges from how the models are evaluated

WHBench: Evaluating Frontier LLMs with Expert-in-the-Loop Validation on Women's Health Topics

Addressing Benchmarking Gaps in Large Language Models for Health and Medicine with Dynamic Red-Teaming