ProtoGestalt: Interpretable Prototypes for Facial Phenotyping and Diagnosis of Rare Genetic Disorders

Ranga Baminiwatte; Alexander Hustinx; Fabio Hellmann; Aaron J Masino

cs.CVNov 5, 2026

ProtoGestalt: Interpretable Prototypes for Facial Phenotyping and Diagnosis of Rare Genetic Disorders

Authors: Ranga Baminiwatte, Alexander Hustinx, Fabio Hellmann, Aaron J Masino

Organizations: School of Computing, Clemson University, USA · Institute for Human Genetics, Clemson University, USA · Institute for Genomic Statistics and Bioinformatics, Rheinische Friedrich-Wilhelms Universität Bonn, Germany · Human-centered Artificial Intelligence Lab, Augsburg University, Germany

Abstract

Rare genetic disorders are difficult to diagnose due to their diversity and overlapping clinical features. Facial morphology can provide diagnostic clues, yet most diagnostic models offer limited interpretability and are not designed to support broad screening for undiagnosed disease. We present ProtoGestalt, an interpretable prototype-based framework for facial phenotyping, rare-disease screening, and syndrome identification. The method extends prototype learning by constraining learned visual features to select facial regions, allowing predictions to be explained through localized and additive evidence. We assess ProtoGestalt's ability to distinguish syndromic from unaffected faces on multiple control datasets and class prevalence values. We further evaluate it's diagnostic performance on syndromes observed during training. The learned prototype-based representations provide significant separation between syndromic and case-matched control images and retain useful performance as the proportion of controls increases. The framework also demonstrates the ability to differentiate among syndromes. Although reasonably accurate specialized models for facial classification of rare disorders already exist, ProtoGestalt offers an interpretable alternative that links predictions to anatomically meaningful facial regions, potentially supporting clinical decision-making and the discovery of phenotype-gene associations.

Figures & tables

Figure 1: Overview of the ProtoGestalt framework. Given an input facial image $\mathbf{x}$ , a CNN backbone $f_{\theta}$ produces $P$ spatial prototype activation maps of size $H\times W$ . Pooling generates prototype similarity scores which are passed to a prediction head, $h_{\phi}$ for output inference. MediaPipe Face Mesh detects 478 facial landmarks, which define ten anatomical regions. Each prototype is assigned to one region, and its presence score is obtained using region-masked max pooling; global max pooling is used when the required landmarks are unavailable. The synthetic image shown in the illustration was generated with GestaltGAN ( Kirchhoff et al. 2025 ) for use in this publication.

Dataset	Images	Patients	Syndromes
GMDB-Frequent	9671	7294	275
Training	7787	5855	275
Validation	1002	778	275
Test	882	661	275
GMDB-Rare	1309	1052	306
GMDB-Total	10980	8346	581

Table 1: Overview of GMDB (v. 1.1.0) dataset. Training and validation sets of GMDB-Frequent are used for training and fine-tuning prototypes. Test set of GMDB-Frequent and GMDB-Rare sets are used for evaluation.

Region $g$	Landmark indices $\mathcal{I}_{g}$
left eye	468, 469, 470, 471, 472, 33, 133, 159, 145
right eye	473, 474, 475, 476, 477, 263, 362, 386, 374
left brow	105, 66, 107, 55, 65
right brow	334, 296, 336, 285, 295
nose	1, 2, 4, 5, 6, 19, 94
left cheek	50, 116, 117, 118, 205

Table 2: The ten facial regions and their defining MediaPipe Face Mesh landmark sets

\mathcal{I}_{g}

. MediaPipe provides 468 facial-mesh landmarks and 10 additional iris landmarks, for a total of 478 points. Regions are indexed in the order shown, and each prototype is assigned to one region according to Eq. ( 4 ).

Figure 2: Visualizations of a global Max Pooled prototype and a region Masked prototype at a specific region. Fixing prototypes to specific regions mitigates the potential mirror effect of CNNs (Example: recognizing the right eye as a left eye). Furthermore, region Masked prototypes allow us to track which prototype corresponds to which facial region.

Pooling	Controls	Acc	PPV	Sen	Spec	F1
GMP	C-WF	0.89	0.93	0.86	0.93	0.89
	UTKF	0.94	0.96	0.91	0.96	0.93
	UTKF-MC	0.85	0.82	0.91	0.80	0.86
RM	C-WF	0.87	0.97	0.76	0.98	0.85
	UTKF	0.86	0.96	0.75	0.97	0.84
	UTKF-MC	0.73	0.71	0.75	0.70	0.73

Table 3: Performance of prototype-based representations for distinguishing syndromic facial images from controls. For each pooling method, logistic-regression classifiers were trained using 2,000 GMDB-Frequent images and 2,000 control images from either CASIA-WebFace or UTKFace. Performance was evaluated on a fixed syndromic cohort comprising the GMDB-Frequent test set and GMDB-Rare dataset (n=2,191), paired with held-out controls from CASIA-WebFace, UTKFace, UTKFace match controls. Legend: Acc - Accuracy, PPV - Positive Predictive Value (aka precision), Sen - Sensitivity (aka recall), Spec - Specificity, GMP - Global Max Pooled, RM - Region Masked, C-WF - CASIA-WebFace, UTKF - UTKFace, UTKF-MC - UTKFace Mactched Controls

Figure 3: AUPRC values as the number of control images increases while the syndromic cohort remains fixed, evaluating LR model’s (Global max pooled) robustness under increasing control-to-syndromic class imbalance and decreasing syndrome prevalence.AUPRC declined progressively yet remained above 0.70 at all evaluated imbalance levels.

Model	Number of prototypes	GMDB-Frequent
Model	Number of prototypes	Top-1	Top-5	Top-10
RN50:GMP	2048	21.8	34.1	40.9
RN50:RM	2048	23.0	31.2	37.9
RN101:GMP	2048	7.0	17.1	25.2
RN101:RM	2048	4.8	14.5	20.3
IRN100:GMP	512	33.3	52.3	59.9

Table 4: Syndrome-identification performance on the GMDB-Frequent test set (

n=882

) using the trained PIP-Net classification head (all values are percents). The number of prototypes is reported for each CNN backbone (

f_{\theta}

). Top-

1

, Top-

5

, and Top-

10

accuracies (

\mathrm{Acc}_{k}

) are reported for global max-pooled and region-masked prototype representations across five CNN backbones. Legend: RN50 - ResNet50, RN101 - ResNet101, IRN100 - IResNet100, CNX13 - ConvNeXt13, CNX26 - ConvNeXt26, GMP - Global Max Pooled, RM - Region Masked

Figure 4: Local explanation of a prediction for a case of Cornelia de Lange syndrome. Prototypes are ranked by their contribution to the predicted class (contribution(c) = prototype presence value( $\mathbf{p}$ ) × its classifier weight( $\mathbf{w}$ )). For each top prototype, the patches show the region of the image where it was most strongly activated, and the bar chart shows the corresponding contribution scores. The synthetic image shown in the illustration was generated with GestaltGAN for use in this publication.

Prototype	Region	Patients (of 43)	Fraction
p683	right_brow	27	0.63
p733	right_brow	27	0.63
p574	nose	27	0.63
p222	left_brow	26	0.60
p201	right_eye	25	0.58
p20	left_eye	24	0.56

Table 5: Prototypes most consistently driving the Cornelia de Lange syndrome prediction, ranked by the fraction of the 43 patients in whose top-10 contribution they appear.

Rare Disease Diagnosis

Rare disease diagnosis uses clinical, genetic, and imaging evidence to identify uncommon conditions and guide patient care.

See all 32 papers

FaceKit: a Toolkit for Interpretable Facial Phenotyping, Synthetic Image Generation and Privacy Analysis in Rare Diseases
Oct 7, 2026Hongzhuo Chen, Zhanliang Wang, Florent Pollet +9Rare Disease Diagnosis Synthetic Data Evaluation
Raymond G. Perelman Center for Cellular and Molecular Therapeutics, Children’s Hospital of Philadelphia, Philadelphia, PA, USA · Applied Mathematics and Computational Science Graduate Program, University of Pennsylvania, Philadelphia, PA, USA · Department of Biomedical Informatics, Columbia University, New York, NY, USA +6
GestaltMML: Enhancing Rare Genetic Disease Diagnosis through Multimodal Machine Learning Combining Facial Images and Clinical Text
Dec 23, 2023Da Wu, Zhanliang Wang, Hongzhuo Chen +12Rare Disease Diagnosis Multimodal Classification
Raymond G. Perelman Center for Cellular and Molecular Therapeutics, Children’s Hospital of Philadelphia, Philadelphia, PA 19104, USA · Department of Mathematics, University of Pennsylvania, Philadelphia, PA 19104, USA · Department of Biomedical Informatics, Columbia University Irving Medical Center, New York, NY 10032, USA +8
Multi-Level Evidence Aggregation for Robust Facial Phenotype Retrieval in Rare Genetic Disorder Prioritization
Aug 11, 2026Alexander Hustinx, Carolin Kaffiné, Behnam Javanmardi +2Rare Disease Diagnosis Image Matching
Institute for Genomic Statistics and Bioinformatics, University Hospital Bonn, Bonn, Germany

Oct 7, 2026J/K move · Enter open · S save

FaceKit: a Toolkit for Interpretable Facial Phenotyping, Synthetic Image Generation and Privacy Analysis in Rare Diseases

Hongzhuo Chen, Zhanliang Wang, Florent Pollet, Mian Umair Ahsan+8

Raymond G. Perelman Center for Cellular and Molecular Therapeutics, Children’s Hospital of Philadelphia, Philadelphia, PA, USA · Applied Mathematics and Computational Science Graduate Program, University of Pennsylvania, Philadelphia, PA, USA · Department of Biomedical Informatics, Columbia University, New York, NY, USA+6

Rare Disease Diagnosis Synthetic Data Evaluation Synthetic Data Generation Synthetic Data Augmentation

Open paper AskPDF arXiv

Many rare genetic diseases are associated with recognizable craniofacial features. However, traditional approaches for describing facial morphology rely largely on qualitative clinical observation and free-text descriptions, which are often subjective, non-standardized, and difficult to reproduce across observers and institutions. Although the Human Phenotype Ontology (HPO) provides controlled terms for describing facial features, these terms are typically categorical rather than quantitative and may vary depending on examiner experience and interpretation. Here, we present FaceKit, a computational framework for quantitative facial phenotyping from frontal facial photographs. FaceKit extracts standardized measurements of facial landmarks and derived 120 morphological features, then reports feature-level z-scores representing deviation from population reference distributions. The reference distributions are built from the FairFace dataset spanning diverse ancestral groups. We evaluated FaceKit on a curated subset of the GestaltMatcher Database covering 50 rare-disease cohorts. In addition to quantitative facial analysis, FaceKit includes synthetic facial image generation to support rare disease model development and data augmentation. We also performed privacy evaluation to assess whether synthetic images reveal identifiable information from real patient photographs and could compromise patient privacy. Across disease case studies, FaceKit-derived quantitative measurements captured known facial features associated with rare genetic disorders and provided objective support for clinical phenotyping. Together, these results establish FaceKit as a useful tool for quantitative phenotyping, and has the potential to improve rare disease diagnosis, support genotype-phenotype studies, and enable more reproducible clinical characterization across diverse patient populations.

Abstract

Rare Disease Diagnosis

Explore similar work

FaceKit: a Toolkit for Interpretable Facial Phenotyping, Synthetic Image Generation and Privacy Analysis in Rare Diseases

GestaltMML: Enhancing Rare Genetic Disease Diagnosis through Multimodal Machine Learning Combining Facial Images and Clinical Text

Multi-Level Evidence Aggregation for Robust Facial Phenotype Retrieval in Rare Genetic Disorder Prioritization