An artificial intelligence framework for end-to-end rare disease phenotyping from clinical notes using large language models

Cathy Shyr; Yan Hu; Rory J. Tinker; Thomas A. Cassini; Kevin W. Byram; Rizwan Hamid; Daniel V. Fabbri; Adam Wright

cs.AIFeb 23, 2026

An artificial intelligence framework for end-to-end rare disease phenotyping from clinical notes using large language models

Authors: Cathy Shyr, Yan Hu, Rory J. Tinker, Thomas A. Cassini, Kevin W. Byram, Rizwan Hamid, Daniel V. Fabbri, Adam Wright, +3 more

Organizations: Department of Biomedical Informatics, Vanderbilt University Medical Center, 2525 West End Avenue, Nashville, 37203, TN, USA. · Department of Biostatistics, Vanderbilt University Medical Center, 2525 West End Avenue, Nashville, 37203, TN, USA. · Department of Pediatrics, Vanderbilt University Medical Center, 2200 Children’s Way, Nashville, 37232, TN, USA. · McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston, 7000 Fannin St #600, Houston, 77030, TX, USA. · Department of Medical Genetics and Genomics, Icahn School of Medicine at Mount Sinai, 1 Gustave L. Levy Pl, New York, 10028, New York, USA. · Department of Medicine, Vanderbilt University Medical Center, 1161 21st Ave S, Nashville, 37232, TN, USA. · Department of Computer Science, Vanderbilt University, 1400 18th Avenue S, Nashville, 37212, TN, USA. · Department of Biomedical Informatics and Data Science, Yale School of Medicine, 101 College Street, New Haven, 06520, CT, USA.

Abstract

Phenotyping is fundamental to rare disease diagnosis, but manual curation of structured phenotypes from clinical notes is labor-intensive and difficult to scale. Existing artificial intelligence approaches typically optimize individual components of phenotyping but do not operationalize the full clinical workflow of extracting features from clinical text, standardizing them to Human Phenotype Ontology (HPO) terms, and prioritizing diagnostically informative HPO terms. We developed RARE-PHENIX, an end-to-end AI framework for rare disease phenotyping that integrates large language model-based phenotype extraction, ontology-grounded standardization to HPO terms, and supervised ranking of diagnostically informative phenotypes. We trained RARE-PHENIX using data from 2,671 patients across 11 Undiagnosed Diseases Network clinical sites, and externally validated it on 16,357 real-world clinical notes from Vanderbilt University Medical Center. Using clinician-curated HPO terms as the gold standard, RARE-PHENIX consistently outperformed a state-of-the-art deep learning baseline (PhenoBERT) across ontology-based similarity and precision-recall-F1 metrics in end-to-end evaluation (i.e., ontology-based similarity of 0.70 vs. 0.58). Ablation analyses demonstrated performance improvements with the addition of each module in RARE-PHENIX (extraction, standardization, and prioritization), supporting the value of modeling the full clinical phenotyping workflow. By modeling phenotyping as a clinically aligned workflow rather than a single extraction task, RARE-PHENIX provides structured, ranked phenotypes that are more concordant with clinician curation and has the potential to support human-in-the-loop rare disease diagnosis in real-world settings.

Figures & tables

Figure 1: Overview of RARE-PHENIX . RARE-PHENIX is an end-to-end AI system for automating the extraction, standardization, and prioritization of rare disease phenotypes from unstructured clinical text. This system consists of three modules for 1) extracting rare disease features from clinical notes with large language models (LLMs); 2) standardizing these features to structured Human Ontology Phenotype (HPO) terms using retrieval-augmented generation; and 3) prioritize diagnostically informative HPO terms using a supervised ranking model. LLMs include LLaMA-2-chat (7b, 13b, and 70b) , LLaMA-3-instruct (8b, and 70b) LLaMA-3.1-instruct (8b and 70b), LLaMA-3.2-instruct (1b and 3b), LLaMA-3.3-instruct (70b) , and a secure instance of Azure OpenAI’s ChatGPT-4o (v2024-06-01) provisioned for handling protected health information in accordance with institutional data governance policies.

Training Cohorts
Clinical Site	Number of patients, n (%)	Age, median (IQR)	Number of HPO terms, median (IQR)
UCLA	248 (8.8%)	11.0 (4.2–23.8)	22.0 (13.0–38.2)
Baylor	358 (12.7%)	8.0 (3.0–18.0)	16.0 (9.0–25.0)
Duke	302 (10.7%)	7.0 (3.0–14.5)	21.5 (12.0–31.0)
Stanford	325 (11.5%)	11.0 (5.0–29.8)	10.0 (6.0–17.0)
Harvard-affiliate	240 (8.5%)	12.5 (4.0–29.0)	13.0 (7.0–22.0)

Table 1: Overview of Undiagnosed Diseases Network (UDN) data . HPO = Human Phenotype Ontology. IQR = interquartile range.

Clinical note type	N (%)	Number of notes per patient, median (IQR)	Number of characters per note, median (IQR)
Progress	11083 (67.8%)	16.0 (8.0–71.5)	2326.0 (819.5–5008.0)
Consultations	2074 (12.7%)	9.0 (2.5–27.0)	901.0 (506.2–2200.5)
Procedures / Operative	926 (5.7%)	9.5 (4.0–17.8)	942.0 (205.0–2189.2)
Imaging	712 (4.4%)	4.0 (2.0–14.0)	2276.5 (569.5–3376.2)
Rehabilitation & Therapy	298 (1.8%)	4.0 (1.2–12.0)	2993.5 (610.0–4137.8)
Nutrition	169 (1.0%)	2.0 (1.0–14.0)	349.0 (221.0–588.0)

Table 2: Summary of clinical notes in the external evaluation cohort . IQR = interquartile range.

Figure 2: End-to-end performance results of RARE-PHENIX and PhenoBERT on the external validation cohort. For legibility, only the top-performing large language models are shown in the figure (i.e., ChatGPT-4o, LLaMA-2-70b, LLaMA-3-70b, LLaMA 3.1-70b ) in addition to the baseline comparator (PhenoBERT) across top-

k

cutoffs. The end-to-end performance results of other RARE-PHENIX configurations are provided in Supplementary Table S2 .

Figure 3: Module-based Ablation Analysis Results of RARE-PHENIX Across Extraction and Standardization Modules

Figure 4: Contribution of phenotype prioritization to diagnostic utility. Improvement in performance using the prioritization module (Module 3) relative to a random ordering of the same extracted phenotypes. For each patient, extracted HPO terms were randomly permuted 200 times, and performance was evaluated at top-

k

cutoffs (

k=10,20,30,40,50

). Values represent the mean difference (

\Delta

= ranking by Module 3

-

ranking by random ordering), and shaded regions indicate 95% bootstrap intervals obtained by resampling at the patient level.

Figure 5: Results of systematic error analysis . False negatives and false positives of RARE-PHENIX with the best-performing large language model configurations ( ChatGPT-4o, LLaMA-2-70b, LLaMA-3-70b, LLaMA-3.1-70b ) and the baseline comparator (PhenoBERT) at different top-k cutoffs

(k=10,20,30,40,50)

Explore similar work

RareDxR1: Autonomous Medical Reasoning for Rare Disease Diagnosis Beyond Human Annotation
Jun 30, 2026Deyang Jiang, Haoran Wu, Ziyi Wang +4Rare Disease Diagnosis RL for Language Model Reasoning
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · Peking Union Medical College Hospital, Beijing, China · School of Future Technology, University of Chinese Academy of Sciences, Beijing, China
HPOQuest: A Rare-Disease Diagnostic Agent Using Active Phenotype Acquisition
Sep 16, 2026Kamilia Zaripova, Nassir Navab, Azade Farshad +1Rare Disease Diagnosis Clinical Decision Support
Technical University of Munich, Munich, Germany · Computational Health Center, Helmholtz Center Munich, Oberschleißheim, Germany · Munich Center for Machine Learning (MCML), Munich, Germany +2
RareDx: Controlled Knowledge Integration and Graph-Grounded Policy Optimization for Rare-Disease Diagnosis
Sep 28, 2026Bo Zhang, Yuchen Wang, Dongbai Li +6Rare Disease Diagnosis LLM Reasoning with Graphs
Xi’an Jiaotong University · UIUC · Tsinghua University +4

Jun 30, 2026J/K move · Enter open · S save

RareDxR1: Autonomous Medical Reasoning for Rare Disease Diagnosis Beyond Human Annotation

Deyang Jiang, Haoran Wu, Ziyi Wang, Yiming Rong+3

The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · Peking Union Medical College Hospital, Beijing, China · School of Future Technology, University of Chinese Academy of Sciences, Beijing, China

Rare Disease Diagnosis RL for Language Model Reasoning Clinical Reasoning Medical LLMs

Open paper AskPDF arXiv

Rare disease differential diagnosis is a critical yet arduous clinical task, requiring physicians to identify precise phenotypes from complex, unstructured patient symptoms and execute intricate reasoning within a vast search space. However, existing AI approaches typically rely on pipeline-based phenotype extraction or retrieval-augmented generation, which suffer from critical information loss due to predefined ontologies, retrieval bottlenecks, and a lack of diagnostic logic. To address these challenges, we introduce RareDxR1, an end-to-end reasoning-centric large language model designed for open-domain rare disease diagnosis directly from unstructured clinical notes. We design a progressive end-to-end training framework by synergizing knowledge internalization with autonomous evolutionary learning, thereby bypassing reliance on structured phenotypes and closed-set decision-making. To overcome the limitations of RAG and phenotype restriction, we enabled the deep internalization of fragmented rare-disease knowledge directly into the model's parameters. Moreover, to bridge the gap between model generation and expert reasoning, we propose Reflection-Enhanced Reasoning Sampling (RERS), a strategy that synthesizes expert-level diagnostic trajectories by learning from failures without human annotation. Additionally, we propose a dual-level curriculum reinforcement learning approach for gradually mastering rare disease diagnosis. Experimental results demonstrate that RareDxR1 achieves state-of-the-art accuracy across different benchmarks, marking a significant breakthrough in open-domain rare disease diagnosis. Our code and dataset will be publicly available.

Abstract

Explore similar work

RareDxR1: Autonomous Medical Reasoning for Rare Disease Diagnosis Beyond Human Annotation

HPOQuest: A Rare-Disease Diagnostic Agent Using Active Phenotype Acquisition

RareDx: Controlled Knowledge Integration and Graph-Grounded Policy Optimization for Rare-Disease Diagnosis