An artificial intelligence framework for end-to-end rare disease phenotyping from clinical notes using large language models
Authors: Cathy Shyr, Yan Hu, Rory J. Tinker, Thomas A. Cassini, Kevin W. Byram, Rizwan Hamid, Daniel V. Fabbri, Adam Wright, +3 more
Organizations: Department of Biomedical Informatics, Vanderbilt University Medical Center, 2525 West End Avenue, Nashville, 37203, TN, USA. · Department of Biostatistics, Vanderbilt University Medical Center, 2525 West End Avenue, Nashville, 37203, TN, USA. · Department of Pediatrics, Vanderbilt University Medical Center, 2200 Children’s Way, Nashville, 37232, TN, USA. · McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston, 7000 Fannin St #600, Houston, 77030, TX, USA. · Department of Medical Genetics and Genomics, Icahn School of Medicine at Mount Sinai, 1 Gustave L. Levy Pl, New York, 10028, New York, USA. · Department of Medicine, Vanderbilt University Medical Center, 1161 21st Ave S, Nashville, 37232, TN, USA. · Department of Computer Science, Vanderbilt University, 1400 18th Avenue S, Nashville, 37212, TN, USA. · Department of Biomedical Informatics and Data Science, Yale School of Medicine, 101 College Street, New Haven, 06520, CT, USA.
Phenotyping is fundamental to rare disease diagnosis, but manual curation of structured phenotypes from clinical notes is labor-intensive and difficult to scale. Existing artificial intelligence approaches typically optimize individual components of phenotyping but do not operationalize the full clinical workflow of extracting features from clinical text, standardizing them to Human Phenotype Ontology (HPO) terms, and prioritizing diagnostically informative HPO terms. We developed RARE-PHENIX, an end-to-end AI framework for rare disease phenotyping that integrates large language model-based phenotype extraction, ontology-grounded standardization to HPO terms, and supervised ranking of diagnostically informative phenotypes. We trained RARE-PHENIX using data from 2,671 patients across 11 Undiagnosed Diseases Network clinical sites, and externally validated it on 16,357 real-world clinical notes from Vanderbilt University Medical Center. Using clinician-curated HPO terms as the gold standard, RARE-PHENIX consistently outperformed a state-of-the-art deep learning baseline (PhenoBERT) across ontology-based similarity and precision-recall-F1 metrics in end-to-end evaluation (i.e., ontology-based similarity of 0.70 vs. 0.58). Ablation analyses demonstrated performance improvements with the addition of each module in RARE-PHENIX (extraction, standardization, and prioritization), supporting the value of modeling the full clinical phenotyping workflow. By modeling phenotyping as a clinically aligned workflow rather than a single extraction task, RARE-PHENIX provides structured, ranked phenotypes that are more concordant with clinician curation and has the potential to support human-in-the-loop rare disease diagnosis in real-world settings.
Figures & tables
Figure 1: Overview of RARE-PHENIX . RARE-PHENIX is an end-to-end AI system for automating the extraction, standardization, and prioritization of rare disease phenotypes from unstructured clinical text. This system consists of three modules for 1) extracting rare disease features from clinical notes with large language models (LLMs); 2) standardizing these features to structured Human Ontology Phenotype (HPO) terms using retrieval-augmented generation; and 3) prioritize diagnostically informative HPO terms using a supervised ranking model. LLMs include LLaMA-2-chat (7b, 13b, and 70b) , LLaMA-3-instruct (8b, and 70b) LLaMA-3.1-instruct (8b and 70b), LLaMA-3.2-instruct (1b and 3b), LLaMA-3.3-instruct (70b) , and a secure instance of Azure OpenAI’s ChatGPT-4o (v2024-06-01) provisioned for handling protected health information in accordance with institutional data governance policies.
Clinical Site
Number of patients, n (%)
Age, median (IQR)
Number of HPO terms, median (IQR)
Training Cohorts
UCLA
248 (8.8%)
11.0 (4.2–23.8)
22.0 (13.0–38.2)
Baylor
358 (12.7%)
8.0 (3.0–18.0)
16.0 (9.0–25.0)
Duke
302 (10.7%)
7.0 (3.0–14.5)
21.5 (12.0–31.0)
Stanford
325 (11.5%)
11.0 (5.0–29.8)
10.0 (6.0–17.0)
Harvard-affiliate
240 (8.5%)
12.5 (4.0–29.0)
13.0 (7.0–22.0)
Table 1: Overview of Undiagnosed Diseases Network (UDN) data . HPO = Human Phenotype Ontology. IQR = interquartile range.
Clinical note type
N (%)
Number of notes per patient, median (IQR)
Number of characters per note, median (IQR)
Progress
11083 (67.8%)
16.0 (8.0–71.5)
2326.0 (819.5–5008.0)
Consultations
2074 (12.7%)
9.0 (2.5–27.0)
901.0 (506.2–2200.5)
Procedures / Operative
926 (5.7%)
9.5 (4.0–17.8)
942.0 (205.0–2189.2)
Imaging
712 (4.4%)
4.0 (2.0–14.0)
2276.5 (569.5–3376.2)
Rehabilitation & Therapy
298 (1.8%)
4.0 (1.2–12.0)
2993.5 (610.0–4137.8)
Nutrition
169 (1.0%)
2.0 (1.0–14.0)
349.0 (221.0–588.0)
Table 2: Summary of clinical notes in the external evaluation cohort . IQR = interquartile range.
Figure 2: End-to-end performance results of RARE-PHENIX and PhenoBERT on the external validation cohort. For legibility, only the top-performing large language models are shown in the figure (i.e., ChatGPT-4o, LLaMA-2-70b, LLaMA-3-70b, LLaMA 3.1-70b ) in addition to the baseline comparator (PhenoBERT) across top- k cutoffs. The end-to-end performance results of other RARE-PHENIX configurations are provided in Supplementary Table S2 .
Figure 3: Module-based Ablation Analysis Results of RARE-PHENIX Across Extraction and Standardization Modules
Figure 4: Contribution of phenotype prioritization to diagnostic utility. Improvement in performance using the prioritization module (Module 3) relative to a random ordering of the same extracted phenotypes. For each patient, extracted HPO terms were randomly permuted 200 times, and performance was evaluated at top- k cutoffs ( k=10,20,30,40,50 ). Values represent the mean difference ( Δ = ranking by Module 3 − ranking by random ordering), and shaded regions indicate 95% bootstrap intervals obtained by resampling at the patient level.
Figure 5: Results of systematic error analysis . False negatives and false positives of RARE-PHENIX with the best-performing large language model configurations ( ChatGPT-4o, LLaMA-2-70b, LLaMA-3-70b, LLaMA-3.1-70b ) and the baseline comparator (PhenoBERT) at different top-k cutoffs (k=10,20,30,40,50) .
Rare disease differential diagnosis is a critical yet arduous clinical task, requiring physicians to identify precise phenotypes from complex, unstructured patient symptoms and execute intricate reasoning within a vast search space. However, existing AI approaches typically rely on pipeline-based phenotype extraction or retrieval-augmented generation, which suffer from critical information loss due to predefined ontologies, retrieval bottlenecks, and a lack of diagnostic logic. To address these challenges, we introduce RareDxR1, an end-to-end reasoning-centric large language model designed for open-domain rare disease diagnosis directly from unstructured clinical notes. We design a progressive end-to-end training framework by synergizing knowledge internalization with autonomous evolutionary learning, thereby bypassing reliance on structured phenotypes and closed-set decision-making. To overcome the limitations of RAG and phenotype restriction, we enabled the deep internalization of fragmented rare-disease knowledge directly into the model's parameters. Moreover, to bridge the gap between model generation and expert reasoning, we propose Reflection-Enhanced Reasoning Sampling (RERS), a strategy that synthesizes expert-level diagnostic trajectories by learning from failures without human annotation. Additionally, we propose a dual-level curriculum reinforcement learning approach for gradually mastering rare disease diagnosis. Experimental results demonstrate that RareDxR1 achieves state-of-the-art accuracy across different benchmarks, marking a significant breakthrough in open-domain rare disease diagnosis. Our code and dataset will be publicly available.
Deyang Jiang, Haoran Wu, Ziyi Wang +4
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · Peking Union Medical College Hospital, Beijing, China · School of Future Technology, University of Chinese Academy of Sciences, Beijing, China
More than 300 million people worldwide are affected by one of over 7,000 known rare diseases, yet diagnosis remains difficult because patients initially present with incomplete and heterogeneous phenotypes. We present HPOQuest, a training-free framework for sequential phenotype acquisition in rare-disease diagnosis. Starting from a small set of observed patient phenotypes, HPOQuest maintains a probabilistic disease ranking and iteratively selects informative follow-up questions to support clinicians during patient assessment. Confirmed phenotypes update the disease ranking, while all responses update the candidate question set. Across four benchmark cohorts, HPOQuest substantially improves diagnosis from sparse initial phenotypes, with gains of up to 30% points at Recall@1 and 45% points at Recall@5. These results demonstrate that sequential phenotype acquisition can substantially improve rare-disease diagnosis from limited initial clinical evidence.
Kamilia Zaripova, Nassir Navab, Azade Farshad +1
Technical University of Munich, Munich, Germany · Computational Health Center, Helmholtz Center Munich, Oberschleißheim, Germany · Munich Center for Machine Learning (MCML), Munich, Germany +2
Rare-disease diagnosis is a long-tail reasoning problem: phenotypes are incomplete, individual disorders are sparsely documented, and relevant evidence is distributed across ontologies, gene annotations, and biomedical text. Language models consequently favor common conditions, miss rare candidates, or produce plausible but invalid names. We introduce RareDx, which couples controlled evidence use with knowledge-graph-grounded policy optimization. RareDx-Harness normalizes heterogeneous records into one ranked-diagnosis task and compares direct inference, static retrieval, adaptive tools, and structured phenotype-gene-disease reasoning over a shared knowledge layer. The training pipeline combines Top-10 post-training with RareDx-KGPO, our knowledge-graph-grounded policy optimization method. Its reward projects predictions into a canonical disease graph and integrates curated graded relevance, ontology proximity, biomedical similarity, and phenotype consistency. Vocabulary and output-budget constraints prevent dense partial credit from rewarding fabricated or overlong differentials. Across eight benchmarks, the complete RareDx system centered on Qwen3.5-9B reaches 38.34 macro Hit@10, 1.60 points above GPT-5.5 under the archived protocol; a disjoint validation-selection audit retains a 6.80-point routing gain over Direct on held-out cases. The 27B system reaches 23.53/36.56/40.76 at Hit@1/5/10. Controlled ablations show that retrieval is not uniformly helpful and that controlled routing is central to the gain. These results indicate that structured medical knowledge can turn a compact model into a competitive diagnostic ranker across heterogeneous long-tail settings in clinical practice.
Bo Zhang, Yuchen Wang, Dongbai Li +6
Xi’an Jiaotong University · UIUC · Tsinghua University +4