Authors: Alec K. Peltekian, Gorkem Durak, Halil Ertugrul Aktas, Carrie Lynn Richardson, Mary Carns, Kathleen Aren, GR Scott Budinger, Anthony J. Esposito, Alexander Misharin, Alok Nidhi Choudhary, Ankit Agrawal, Ulas Bagci
Organizations: Department of Computer Science, Northwestern University McCormick School of Engineering and Applied Science, Chicago, IL, USA · Department of Radiology, Machine and Hybrid Intelligence Laboratory, Northwestern University Feinberg School of Medicine, Chicago, IL, USA · Division of Rheumatology, Northwestern University Feinberg School of Medicine, Chicago, IL, USA · Division of Pulmonary and Critical Care, Northwestern University Feinberg School of Medicine, Chicago, IL, USA · Simpson Querrey Lung Institute for Translational Science, Northwestern University Feinberg School of Medicine, Chicago, IL, USA · Department of Electrical and Computer Engineering, Northwestern University McCormick School of Engineering and Applied Science, Chicago, IL, USA
Mixture-of-experts (MoE) models combine specialized predictors under learned routing, offering a principled mechanism for leveraging heterogeneity in medical data. We present a hierarchical multimodal MoE for interstitial lung disease (ILD) classification that integrates a frozen, pre-trained imaging expert with structured electronic health records (EHR) via two-stage gating. A modality-level gate assigns patient-specific weights to imaging and EHR predictions, while a sub-gating module decomposes the EHR branch into clinically defined feature groups with learned, group-specific contributions. This design preserves stable imaging representations while enabling input-dependent clinical weighting and explicit EHR specialization. Under strict patient-level cross-validation, the model achieved the highest mean AUC among the evaluated methods (0.8750 +- 0.0443), compared with 0.8646 for imaging-only REN and 0.7685 for SwinUNETR. The framework extends interpretability across anatomical regions, imaging--EHR utilization, and clinically defined EHR feature groups.
Figures & tables
Figure 1: Overview of the proposed hierarchical multimodal MoE framework for ILD diagnosis. (A) CT pipeline: volumes are preprocessed and segmented into five lung lobes; five lobe-specific imaging experts operate on lobe-masked volumes; fixed radiomics-derived lobe-importance weights provide a fold-specific prior for aggregating expert representations. (B) EHR pipeline: structured variables are partitioned into clinically defined feature groups and processed by group-specific sub-experts; an EHR sub-gate aggregates sub-expert outputs into a single clinical representation. We additionally evaluate a selective EHR configuration via expert-subset selection (restricting to a targeted subset of feature groups) prior to EHR sub-gating. (C) Multimodal fusion: a modality gate adaptively weights imaging and clinical representations, followed by a classifier for ILD prediction.
Method
AUC ± SD [95% CI]
Change vs. SwinUNETR
p-value
Imaging-Only Baselines
SwinUNETR (Baseline)
0.7685±0.0759 [0.674, 0.863]
–
–
CNN (Baseline)
0.7584±0.0919 [0.667, 0.850]
−1.3%
–
Mamba (Baseline)
0.6775±0.0454 [0.632, 0.723]
−11.8%
–
ViT (Baseline)
0.6535±0.0356 [0.618, 0.689]
−15.0%
–
MoE Anatomically-Guided Imaging-Only (REN)
Table 1: Performance comparison using 5-fold patient-level cross-validation. Values are reported as mean ± SD across folds. Table p -values are from two-sided paired t -tests against SwinUNETR. The direct comparison between the selective hierarchical MoE and imaging-only REN is reported in the text.
Figure 2: Interpretability analysis of hierarchical MoE gating behavior. (A) Modality-level gate activations for the full hierarchical EHR configuration, stratified by ILD status. (B) Mean EHR sub-expert gate weights across clinical feature groups for the full hierarchy. (C) Relationship between modality-level gate activations and prediction confidence for the full hierarchy. (D-F) Corresponding analyses for the selective hierarchical EHR configuration.
Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM) and TUM University Hospital, Munich, Germany · Hasso Plattner Institute for Digital Engineering, University of Potsdam, Potsdam, Germany · Department of Computing, Imperial College London, UK