Auditing evaluation artifacts and data leakage in large language model systems for rare disease diagnosis

Jian Yang; Peilin Liu; Xiaohan Li; Songda Li; Yu Cao; Yuqing Feng; Haomin Li

cs.AIOct 24, 2026

Auditing evaluation artifacts and data leakage in large language model systems for rare disease diagnosis

Authors: Jian Yang, Peilin Liu, Xiaohan Li, Songda Li, Yu Cao, Yuqing Feng, Haomin Li

Organizations: The Children’s Hospital, Zhejiang University School of Medicine, Hangzhou, 310052, China. · The College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, 310027, China.

Abstract

Augmented large language model (LLM) systems have reported high benchmark accuracies in rare disease diagnosis, yet their real-world clinical utility remains unproven. A recent meta-analysis flagged augmented LLM diagnostic evaluations as having a high risk of bias due to suspected data leakage; however, the precise impact of these vulnerabilities on model inference has yet to be fully quantified. In this study, we present a comprehensive empirical audit to quantify evaluation artifacts across prevailing AI paradigms, using a repository of over 60,000 cases from diverse public and independent clinical cohorts. By reconstructing a state-of-the-art agentic framework, we demonstrate that candidate-space contamination artificially inflates performance: enforcing strict cross-dataset isolation resulted in a marked performance decline, with the cross-dataset mean Recall@1 decreasing from 54.3% (under conventional one-vs-rest evaluation) to 40.3% (Hedges' g = 0.94). Similar spatial overfitting in supervised classifiers caused diagnostic accuracy to drop from an internal mean of 42.9% to 16.7% across external cohorts (Cohen's d = 2.01). Furthermore, we quantify how LLMs infer diagnoses by exploiting retrospective therapeutic cues in clinical narratives, and how temporal misalignment in foundational knowledge bases leads to performance fluctuations. We conclude that prevailing evaluation protocols for augmented LLM systems inadvertently reflect data leakage rather than genuine medical reasoning. To address these systematic artifacts, we propose a rigorous evaluation framework - coupling strict data-isolation protocols with standardized benchmarking safeguards - to ensure future diagnostic AI for rare diseases is evaluated on actual clinical robustness.

Figures & tables

No.	Dataset	Country/Region	Case-level PMID	Total Cases	Unique Diseases	Mean HPO Terms
Literature-derived cohorts
-	RareArena	Multi-country	$\checkmark$	49,760	4,597	11.5
I	Phenopacket	Multi-country	$\checkmark$	8,101	516	9.3
II	LIRICAL	Multi-country	$\checkmark$	384	262	11.2
III	PhenoDP	Multi-country	$\checkmark$	130	72	14.2
IV	RDmaster	Multi-country	$\checkmark$	201	62	10.0

Table 1: Overview of benchmark datasets.

Figure 1: Structural data leakage and generalization collapse across rare disease diagnostic AI paradigms. (a) Diagnostic performance of the agentic system DeepRare across six evaluation cohorts under one-vs-rest (OVR), full-test, and zero-shot LLM configurations. (b) Diagnostic Recall@1 of the standalone patient retrieval module across 10 cohorts under candidate bank partitioning schemes, transitioning from OVR through

n

-fold variants to full-test. (c) Diagnostic Recall@1 (Mean

\pm

95% CI) across four literature-derived cohorts evaluated under the full-test setting with and without cross-dataset PMID-level deduplication. (d) Diagnostic Recall@1 (Mean

\pm

95% CI) of the minimalist RAG prototype across ten datasets evaluated under OVR, full-test, and standalone LLM without RAG configurations. (e–g) Pairwise cross-dataset diagnostic Recall@1 for (e) linear (SGDClassifier), (f) non-linear (XGBoost), and (g) deep learning (multilayer perceptron) models (datasets I–X correspond to Table 1 ; the internal numbers indicate Recall@1). (h) Summary comparison of internal versus external validation accuracy (Mean

\pm

SD) across supervised linear, non-linear, and deep learning classifiers. Note: *

P<0.05

, **

P<0.01

, ***

P<0.001

Figure 2: Post-hoc cue leakage in clinical narratives. (a) Stepwise performance inflation (Mean

\pm

95% CI) of zero-shot LLM diagnostics as retrospective clinical narrative segments—chief complaint (CC), physical examination (PE), ancillary examinations (AE), and clinical treatment (CT)—are sequentially appended. (b) Diagnostic recall (Mean

\pm

95% CI) with sequential addition of clinical text segments to extracted HPO terms.

Figure 3: Longitudinal assessment of clinical diagnostic performance and knowledge base evolution. (a) Recall@1 performance trajectories (Mean

\pm

95% CI) of the likelihood ratio (LR) and probabilistic graphical (PG) models across six annual snapshots (2021–2026) of ORPHA on diverse public and clinical datasets. (b) Corresponding temporal evaluation using OMIM. (c) Temporal evolution of disease entities in OMIM and ORPHA. (d) Temporal evolution of disease-phenotype annotations in OMIM and ORPHA.

System (year)	AI Paradigm	Diagnostic Features	Scope	CC	SO	PCL	TKB	Overall Risk
DeepRare (2026) 51	Agent	HPO Terms # & WES	P	+	–	–	+	+
RareAgents (2026) 6	Agent	HPO Terms & Symptoms	P	+	–	–	+	+
MINT (2026) 43	FT LLM	HPO Terms & Demographics	P	–	+	–	+	+
RareSeek (2025) 50	RAG & FT LLM	Clinical Text	P	–	+	+	+	+
Deep-DxSearch (2025) 53	Agent & RL	Clinical Text	P	+	+	–	+	+
MEDDxAgent (2025) 34	Agent	HPO Terms	R	+	–	–	+	+

Table 2: Evaluation artifact audit across rare disease diagnostic AI systems.

Evaluation Artifact	Manifestation	Proposed Evaluation Guideline
Temporal Knowledge Bias	Inconsistencies in the KB versions that models rely on lead to substantial performance fluctuations, with diagnostic accuracy shifting based on version differences.	Chronological Synchronization: Enforce a single, chronologically aligned KB version across all models to prevent KB version-driven evaluation artifacts.
Spatial / Institutional Overfitting	Models memorize institutional documentation habits, cohort-specific diagnosis-phenotype associations and disease prevalence rather than learning generalizable phenotype-disease profiles.	Cross-institutional Validation: Replace standard intra-dataset splits with mandatory out-of-distribution validation using independent, institutionally isolated clinical cohorts.
Candidate-space & Literature Contamination	Retrieval systems exploit structural data leakage by matching query cases to homologous records originating from the exact same publication, family pedigree, or institution.	Source-level Deduplication: Exclude structurally and institutionally related cases from the reference bank, enforcing PMID-level cross-deduplication for literature-derived cohorts.
Post-Hoc Cue Leakage	Retrospective clinical narratives embed downstream therapeutic interventions and targeted lab results, allowing LLMs to infer diagnoses retrospectively.	Semantic Decoupling & Sanitization: Benchmark primarily using standardized phenotypic profiles such as HPO terms. When evaluating raw clinical texts, systematically redact post-hoc therapeutic and administrative cues.
Post-hoc Hypothesis Shifting	Iterative prompt engineering and hyperparameter tuning directly on the evaluation set cause benchmark overfitting and compromise generalizability.	Blinded Evaluation & Pre-registration: Lock model weights, prompt templates, and evaluation pipelines prior to testing. Utilize independent validation sets to prevent test-set optimization.
Dataset Heterogeneity	Substantial variations in diagnostic difficulty and disease spectrum across case datasets make single-source evaluations unrepresentative.	Multi-tiered Cross-dataset Auditing: Diagnostic systems must be benchmarked across multi-tiered datasets to reliably verify generalization across diverse data origins and clinical complexities.

Table 3: Proposed evaluation guidelines for general rare disease diagnostic AI.

Clinical Language Model Evaluation

Clinical language model evaluation measures whether AI systems are accurate, safe, fair, and useful across medical tasks and care settings.

See all 419 papers

Unified Multiparadigm Fusion Intelligence for Rare Disease Diagnosis
Oct 16, 2026Jian Yang, Peiling Liu, Songda Li +10Rare Disease Diagnosis Clinical Decision Support
The Children’s Hospital, Zhejiang University School of Medicine, Hangzhou, 310052, China. · The College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, 310027, China.
CLExEval: A Human-in-the-Loop Framework for Qualitative Evaluation of LLM Clinical Reasoning
Jun 30, 2026Ajmal M., Abin Roy, Afthab Salam Kanniyan +4LLM-as-a-Judge Human-in-the-Loop Evaluation
1MBZUAI · 2IIT Madras · 3Calicut Medical College
A specialized reasoning large language model for accelerating rare disease diagnosis: a randomized AI physician assistance trial
Jun 23, 2026Haichao Chen, Songchi Zhou, Zhengyun Zhao +28Rare Disease Diagnosis Clinical Decision Support
Tsinghua Medicine, Tsinghua University, Beijing, China. · Department of Ophthalmology, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences & Peking Union Medical College, Beijing, China. · Department of Statistics and Data Science, Tsinghua University, Beijing, China. +19

Oct 16, 2026J/K move · Enter open · S save

Unified Multiparadigm Fusion Intelligence for Rare Disease Diagnosis

Jian Yang, Peiling Liu, Songda Li, Yu Cao+9

The Children’s Hospital, Zhejiang University School of Medicine, Hangzhou, 310052, China. · The College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, 310027, China.

Rare Disease Diagnosis Clinical Decision Support

Open paper AskPDF arXiv

Rare diseases (RDs) affect an estimated 300 million people globally, leaving patients to navigate a protracted diagnostic odyssey constrained by clinical scarcity and fragmented medical knowledge. Artificial intelligence (AI) has progressively advanced RD diagnostics, from symbolic methods and deep learning to large language model (LLM)-based systems. However, each paradigm carries structural limitations: symbolic methods are bounded by knowledge completeness, neural models sacrifice interpretability, and LLM-based systems are vulnerable to evaluation biases like data leakage. In this study, we introduce FusionRare, a unified multiparadigm fusion framework integrating knowledge-driven symbolic reasoning, deep-learning pattern recognition, case-based patient retrieval, and LLM-based heuristic inference via statistical meta-aggregation and knowledge-enhanced agentic reflection. Evaluated on 2,528 strictly isolated clinical cases, FusionRare achieves state-of-the-art RD prioritization, attaining a macro-averaged Recall @1 of 51% for phenotype-only diagnosis and 79% for phenotype-genotype joint analysis. These results represent decisive advantages over the advanced agentic system DeepRare, frontier LLMs including GPT 5.5, and specialized tools like PhenoBrain and LIRICAL under stringent anti-leakage protocols. A randomized, crossover clinician-AI validation demonstrated that FusionRare elevates the true diagnostic rank in 45% of decisions, improves physician confidence in 62%, and reduces time to diagnosis by 21%. These findings indicate that reconciling these orthogonal AI paradigms significantly enhances diagnostic capabilities, bridging the gap between computational benchmarks and actionable clinical reliability. More broadly, this work provides a replicable framework for deploying trustworthy diagnostic AI across complex, high-stakes, and underserved medical domains.

Abstract

Clinical Language Model Evaluation

Explore similar work

Unified Multiparadigm Fusion Intelligence for Rare Disease Diagnosis

CLExEval: A Human-in-the-Loop Framework for Qualitative Evaluation of LLM Clinical Reasoning

A specialized reasoning large language model for accelerating rare disease diagnosis: a randomized AI physician assistance trial