Large language models as synthetic clinical experts to inform longitudinal rare-disease modeling

Clemens Schächter; Astrid Pechmann; Janbernd Kirschner; Jan Hasenauer; Harald Binder

cs.AIAug 17, 2026

Large language models as synthetic clinical experts to inform longitudinal rare-disease modeling

Authors: Clemens Schächter, Astrid Pechmann, Janbernd Kirschner, Jan Hasenauer, Harald Binder

Organizations: Institute of Medical Biometry and Statistics (IMBI), Faculty of Medicine and Medical Center – University of Freiburg, Freiburg, Germany · Freiburg Center for Data Analysis, Modeling and AI – University of Freiburg, Freiburg, Germany · Department of Neuropediatrics and Muscle Disorders, Faculty of Medicine and Medical Center – University of Freiburg, Freiburg, Germany · Bonn Center for Mathematical Life Sciences – University of Bonn, Germany ·

Abstract

Due to the limited amount of information, modeling longitudinal rare-disease data can benefit from integrating clinical knowledge. Yet, elicitation of expert knowledge and formalization for model fitting is challenging, in particular due to limited time of clinical experts. To nevertheless make domain knowledge accessible during model fitting, we use large language models (LLMs) as synthetic clinical experts to supervise a variational-autoencoder-based approach that learns low-dimensional latent summaries of visit-level observations. Specifically, LLMs are queried offline on textual descriptions of patient observations to obtain judgments, e.g., the suspected clinical category. To improve the variational autoencoder fit, we train a differentiable surrogate model on these judgments and augment the loss function to encourage reconstructions that preserve the clinical-label distribution of their corresponding input profile. In an application to longitudinal motor-function assessments from children with spinal muscular atrophy, we map visit-level clinical profiles to low-dimensional representations that are linked by a multivariate mixed-effects model. The synthetic expert loss discourages reconstructions that remain numerically close in data space but alter the clinical interpretation of the reconstructed motor function profile, such as by crossing a disease-type boundary. We thus reduced disagreement between original and reconstructed SMA type labels from about 11 to 7 percent. Furthermore, informing the latent representation by the synthetic expert improved prediction of motor function milestones compared with unsupervised latent representations and a data-level baseline. These results suggest that incorporating LLMs into model fitting can make clinical knowledge available to representation learning and improve clinical faithfulness for longitudinal rare-disease data.

Figures & tables

Figure 1: Overview of the proposed synthetic-expert supervision framework. Part A: Offline surrogate construction. 1) Each observed clinical profile

\mathbf{y}_{i,t}

and its contextual information

\mathbf{s}_{i,t}

are rendered as a natural-language prompt. 2) The LLM rater produces pairwise responses for the specified candidate labels, which are checked for consistency and aggregated into retained synthetic-expert labels. 3) The retained LLM-derived labels are used to train a differentiable surrogate network

S_{\psi}

that maps clinical profiles and contextual information to distributions over the clinical labels. Part B: Synthetic-expert-supervised model fitting. 4) The encoder maps the longitudinal observations to low-dimensional latent representations, whose trajectories are linked through the latent mixed-effects model. 5) The decoder reconstructs the clinical profiles, and both the observed profiles

\mathbf{y}_{i,t}

and their reconstructions

\hat{\mathbf{y}}_{i,t}

are evaluated by the frozen surrogate network using the same contextual information

\mathbf{s}_{i,t}

. The third observation is reconstructed with the wrong label. 6) The Jensen–Shannon divergence between the corresponding surrogate distributions is added to the training objective as a synthetic-expert loss term, thereby discouraging reconstructions whose clinical interpretation differs from that of the observed profile.

Panel of Figure 2 — Figure 2: Interpretability of the synthetic-expert surrogate in profile space and latent space. The upper panel shows age-dependent SMA type probabilities for three patient groups: sitting without standing (observations with HINE-2 sitting item score = 4, standing < 2), standing without walking (standing = 3, walking < 2), and walking (walking = 3, corresponds typically with a full HINE-2 sum score with no recorded deficiencies). Colors indicate SMA type labels and marker shapes indicate the LLM model. The lower panel visualizes the latent space at age = 1 year colored by different synthetic experts. Each point in the latent grid is decoded into a motor-function profile and then classified by the selected LLM-specific surrogate network.

Synthetic expert	Ordinal consistency (%)	Rev. pairwise agreement (%)	Rev. final-label agreement (%)	Surrogate accuracy (%)	Uncertainty ratio
Coin Toss	9.4	50.0	1.7	33.3	–
Gemma-4-E2B	74.1	85.3	52.8	89.6	1.5
Gemma-4-E4B	80.0	84.6	47.8	84.8	1.3
Gemma-4-12B	87.1	95.7	85.1	94.9	11.4
Gemma-4-31B	89.9	97.8	92.7	97.2	9.6
GPT-OSS-20B	74.1	85.8	48.7	87.5	1.6

Table 1: Diagnostics of LLM-derived synthetic-expert judgments and surrogate training. Ordinal consistency is the percentage of profiles with internally consistent pairwise answers. Pairwise and final-label agreement are measured under label-order reversal. Final-label agreement is defined as when the final label agrees under both prompt presentation orders, or the pairwise questions completely align for inconsistent answer patterns. Surrogate accuracy is the cross-validated accuracy for reproducing retained synthetic-expert labels. Values for the coin toss baseline are analytical values. The uncertainty ratio is the mean self-reported uncertainty for pairwise answers that changed under label-order reversal, divided by the mean for answers that remained stable. Here, low was mapped to an uncertainty score of 0, medium to 1 and high to 2.

Synthetic expert	$\lambda_{\mathrm{SE}}$	JS divergence	Label disagreement (%)	HINE-2 sum-score error	HINE-2 error among mismatches
Gemma-4-31B	0	0.054	10.9	1.359	2.451
	10	0.043	9.0	1.357	2.012
	25	0.038	8.2	1.360	2.010
	50	0.034	7.5	1.377	2.004
	100	0.031	6.8	1.382	2.001
GPT-OSS-120B	0	0.034	10.1	1.359	2.354

Table 2: Effect of synthetic-expert supervision weight on reconstruction consistency and reconstruction error. Lower values indicate better consistency or lower reconstruction error. Label disagreement is averaged over the forward and reverse label presentation orders. Bold values indicate the best value within each synthetic-expert model class and metric.

		Sitting			Standing			Walking
Feature set	$\lambda_{\mathrm{SE}}$	Brier	AUC	ECE	Brier	AUC	ECE	Brier	AUC	ECE
HINE-2 sum score	–	0.271	0.817	0.314	0.180	0.834	0.175	0.142	0.845	0.134
HINE-2 sum score + covariates	–	0.209	0.890	0.255	0.155	0.908	0.153	0.124	0.919	0.119
$\mathbf{Z}$	0	0.097	0.944	0.078	0.098	0.951	0.102	0.087	0.954	0.088
$\mathbf{Z}$ informed by Gemma-4-31B	10	0.085	0.954	0.067	0.084	0.962	0.076	0.067	0.964	0.065
	25	0.084	0.956	0.064	0.088	0.960	0.081	0.071	0.962	0.067

Table 3: Milestone prediction on latent representations influenced by synthetic-expert supervision. Rows report one-year landmark prediction performance for a linear time-varying Cox model. Brier denotes the inverse-probability-of-censoring weighted Brier score, AUC the corresponding time-dependent AUC, and ECE the weighted expected calibration error across calibration bins. Lower Brier and ECE values and higher AUC values indicate better performance. Bold values indicate the best performance across all latent-representation feature sets for each milestone-metric combination.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

HINE-2 item	Text fragment by item score
Introduction	The child is {years} years {months} months old when its motor function is evaluated. The child has the following motor function skills:
Upper limb function	0 : Cannot achieve useful hand function to grab objects. 1 : Achieves useful hand function to grab objects.
Raise hands	0 : Cannot raise its hands to the mouth. 1 : Raises its hands to the mouth.
Reach overhead	0 : Cannot reach overhead. 1 : Reaches overhead.
Head control	0 : Has no stable head control and cannot maintain the head upright. 1 : Holds the head upright briefly, but the head wobbles and control is unsteady. 2 : Maintains the head upright consistently and does not show head wobbling.
Sitting	0 : Cannot maintain a sitting position in a meaningful way even with external support. 1 : Sits only if the examiner stabilizes the pelvis or hips and cannot sit independently. 2 : Sits with propping and uses the arms or hands for balance, but cannot sit freely without upper limb support. 3 : Sits independently without obvious propping, but cannot rotate or pivot while sitting. 4 : Sits independently and rotates and pivots while sitting.

Appendix

Table S1 : Text rendering of HINE-2 profile scores used for the synthetic-expert prompts. Each generated profile was converted by selecting the sentence corresponding to the observed item score and concatenating the resulting item-level descriptions with the patient’s age.

Prompt part	Text
Comparison instruction	Decide between two candidate motor-function labels for a child being evaluated for spinal muscular atrophy (SMA): {label A} or {label B} .
Decision rule	Use the child’s age to contextualize the observed motor abilities. If neither label is a reasonable match, choose the allowed label whose severity is closer to the observed motor impairment. When {label A} or {label B} is presymptomatic the following additional instruction was included: Choose presymptomatic when motor abilities are age-appropriate and no SMA-related motor impairment is observed.
Profile text	{profile text}
Output constraint	Think briefly about your answer. Report your confidence of your final decision and conclude by returning the following JSON object: {"label": "label B", "certainty": "high\|medium\|low"} or {"label":"label A", "certainty": "high\|medium\|low"} .

Appendix

Table S2 : Prompt template for one pairwise synthetic-expert comparison. The placeholders {label A} , {label B} , and {profile text} were filled for each candidate-label pair and generated HINE-2 profile.

Milestone	Events	Definition
Sitting	432	Patient sits up straight with the head erect for at least 10 seconds. Patient does not use arms or hands to balance the body or to support the position.
Standing	304	Patient stands in an upright position on both feet, not on the toes, with the back straight. The legs support 100% of the patient’s weight. There is no contact with an object or a person. Patient stands alone for at least 10 seconds.
Walking	264	Patient takes at least 5 steps independently in an upright position with the back straight. One leg moves forward while the other supports most of the body weight. There is no contact with a person or object.

Appendix

Table S3 : Definitions of the motor milestones used for Cox regression. Events denote exact milestone attainments contributing an event interval after excluding left-censored patient-milestone pairs and pairs without a motor-function visit before the event time.

Representation Learning

Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.

See all 799 papers

Jun 23, 2026

A specialized reasoning large language model for accelerating rare disease diagnosis: a randomized AI physician assistance trial

Rare diseases affect millions of individuals worldwide, yet timely diagnosis remains a major public health challenge due to scarcity of specialized clinical expertise. While large language models (LLMs) show promise to support rare disease diagnosis, current models are constrained by insufficient clinical deployability, limited clinically grounded evidence, and scarcity of training data. Here we present RaDaR (Rare Disease navigatoR), an open-source, compact reasoning LLM (32B parameters) for rare disease diagnosis. RaDaR was trained with 49,170 publicly available free-text cases and 104,666 synthetic cases with reasoning-enhanced training. RaDaR showed the strongest performance among evaluated open-source models, including the 671B DeepSeek-R1, across public benchmarks and four external validation centers. In a retrospective cohort, RaDaR prioritized the final diagnosis before documented clinical suspicion in 61.06 percent of cases, corresponding to a potential lead time of 1.87 months and 50.18 percent of the within-center interval. In a randomized physician-assistance trial, RaDaR assistance improved physicians' rare-disease diagnostic accuracy by 21.44 percentage points compared with internet search alone. Synthetic-data ablations suggested that phenotype-anchored narratives provide useful training signal for long-tail rare diseases, with a monotonic scaling trend within the tested data range. Together, RaDaR and its development and validation framework provide a deployable rare-disease reasoning model and a reproducible development framework for diagnostic AI under data scarcity.

Haichao Chen, Songchi Zhou, Zhengyun Zhao +28

Tsinghua Medicine, Tsinghua University, Beijing, China. · Department of Ophthalmology, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences & Peking Union Medical College, Beijing, China. · Department of Statistics and Data Science, Tsinghua University, Beijing, China. +19

Rare Disease Diagnosis Clinical Decision Support Medical LLMs Large Reasoning Models

Ask PDF

Mar 30, 2026

An Underexplored Frontier: Large Language Models for Rare Disease Patient Education and Communication -- A scoping review

Rare diseases affect over 300 million people worldwide and are characterized by complex care pathways, limited clinical expertise, and substantial unmet communication needs throughout the long patient journey. Recent advances in large language models (LLMs) offer new opportunities to support patient education and communication, yet their application in rare diseases remains unclear. We conducted a scoping review of studies published between January 2022 and March 2026 across major databases, identifying 12 studies on LLM-based rare disease patient education and communication. Data were extracted on study characteristics, application scenarios, model usage, and evaluation methods, and synthesized using descriptive and qualitative analyses. The literature is highly recent and dominated by general-purpose models, particularly ChatGPT. Most studies focus on patient question answering using curated question sets, with limited use of real-world data or longitudinal communication scenarios. Evaluations are primarily centered on accuracy, with limited attention to patient-centered dimensions such as readability, empathy, and communication quality. Multilingual communication is rarely addressed. Overall, the field remains at an early stage. Future research should prioritize patient-centered design, domain-adapted methods, and real-world deployment to support safe, adaptive, and effective communication in rare diseases.

Zaifu Zhan, Yu Hou, Kai Yu +4

Division of Computational Health Sciences, Department of Surgery, University of Minnesota, Minneapolis, USA · Department of Electrical and Computer Engineering, University of Minnesota, Minneapolis, USA · Clinical Bioinformatics Lab, Université Paris Cité, Institut Imagine, INSERM UMR 1163, Paris, France +1

Medical LLMs Healthcare Medical QA

Ask PDF

May 18, 2026

Systematic Evaluation of the Quality of Synthetic Clinical Notes Rephrased by LLMs at Million-Note Scale

Large language models (LLMs) can generate or synthesize clinical text for a wide range of applications, from improving clinical documentation to augmenting clinical text analytics. Yet evaluations typically focus on a narrow aspect -- such as similarity or utility comparisons -- even though these aspects are complementary and best viewed in parallel. In this study, we aim to conduct a systematic evaluation of LLM-generated clinical text, which includes intrinsic, extrinsic, and factuality evaluations of synthetic clinical notes rephrased from MIMIC databases at million-note scale. Our analysis demonstrates that synthetic notes preserve core clinical information and predictive utility for coarse-grained tasks despite substantial linguistic changes, but lose fine-grained details for task like ICD coding. We show this loss of detail can be substantially mitigated by rephrasing notes by chunks rather than by the whole note, but at the cost of reduced factual precision under incomplete context. Through fact-checking and error analysis, we further find that synthesis errors are dominated by misinterpretation of clinical context, alongside temporal confusion, measurement errors, and fabricated claims. Finally, we show that the synthetic notes -- despite their task-agnostic nature -- can effectively augment task-specific training for rare ICD codes.

Jinghui Liu, Sarvesh Soni, Anthony Nguyen

Australian e-Health Research Centre, CSIRO, Australia · National Library of Medicine, National Institutes of Health, USA

Synthetic Data Augmentation Automated Medical Coding Synthetic Data Generation Clinical Language Model Evaluation Clinical NLP Synthetic Data Evaluation

Ask PDF

Abstract

Representation Learning

Explore similar work

A specialized reasoning large language model for accelerating rare disease diagnosis: a randomized AI physician assistance trial

An Underexplored Frontier: Large Language Models for Rare Disease Patient Education and Communication -- A scoping review

Systematic Evaluation of the Quality of Synthetic Clinical Notes Rephrased by LLMs at Million-Note Scale