Organizations: University of Texas, Austin, TX, USA · Dell Children’s Medical Center, Austin, TX, USA · University of North Carolina at Chapel Hill, NC, USA · University of Nevada, Reno, NV, USA
Accurate classification of pediatric central nervous system tumors remains challenging due to histological complexity and limited training data. While pathology foundation models have advanced whole-slide image (WSI) analysis, they often fail to leverage the rich, complementary information found in clinical text and tissue microarchitecture. To this end, we propose PathMoE, an interpretable multimodal framework that integrates H&E slides, pathology reports, and nuclei-level cell graphs via an interaction-aware mixture-of-experts architecture built on state-of-the-art foundation models for each modality. By training specialized experts to capture modality uniqueness, redundancy, and synergy, PathMoE employs an input-dependent gating mechanism that dynamically weights these interactions, providing sample-level interpretability. We evaluate our framework on two dataset-specific classification tasks on an internal pediatric brain tumor dataset (PBT) and external TCGA datasets. PathMoE improves macro-F1 from 0.762 to 0.799 (+0.037) on PBT when integrating WSI, text, and graph modalities; on TCGA, augmenting WSI with graph knowledge improves macro-F1 from 0.668 to 0.709 (+0.041). These results demonstrate significant performance gains over state-of-the-art image-only baselines while revealing the specific modality interactions driving individual predictions. This interpretability is particularly critical for rare tumor subtypes, where transparent model reasoning is essential for clinical trust and diagnostic validation.
Figures & tables
Figure 1: Overview of PathMoE . H&E WSIs, pathology reports, and nuclei graphs are encoded and fused via an interaction-aware mixture-of-experts module. An input-dependent gating network computes sample-specific weights to combine expert predictions into the final tumor classification. A vanilla fusion method is also depicted.
High cell density and hemorrhage mimic HGs; lacks nuclear atypia/mitoses and shows pituitary morphology.
Case_3
NGT
LG-CNS
NGT
0.211
0.176
0.210
Mesenchymal non-glial tumor with spindled architecture; overlaps with low-grade gliomas but distinguishable by growth pattern.
Case_4
LG-CNS
HG-CNS
LG-CNS
0.233
0.181
0.144
Low-grade glioneuronal tumor with calcifications and mild nuclear density increase; lacks atypia, mitoses, or necrosis of high-grade tumors.
Table 3: Qualitative error analysis of the baseline and PathMoE (PM) across image–text (WT) and image–text–graph (WTG) settings. Each case shows the ground-truth label, model predictions, modality interaction weights ( wW , wT , wG ), and a brief clinical rationale for the correct diagnosis. Nodes and edges (light blue) often aggregate in regions of cellular density learned by the GNN.
Healthcare models are transitioning from unimodal prediction toward multimodal reasoning over heterogeneous diagnostic inputs. In computational pathology, for complex tumor subtypes where morphology alone can be challenging to distinguish, pathology reports and molecular measurements may provide additional diagnostic evidence alongside whole-slide images, yet existing models often fail to clarify how diverse signals assemble into recognizable diagnostic concepts. We propose ConceptM3oE (Concept Multimodal MoE), which embeds concept formation directly within interaction-aware mixture-of-experts (MoE) pathways. The architecture decomposes evidence into modality-specific, redundant, and synergistic experts, which are then projected into structured concept bottlenecks mapping latent features to a hierarchy of morphology and biomarker concepts. To prevent the information loss typical of interpretable bottlenecks, we utilize residual pathways within each expert to allow task-relevant signals to flow both through the concepts and directly to the final task prediction, so that high performance is maintained alongside interpretability. Across an institutional pediatric brain tumor cohort and a public glioma cohort, the framework delivers competitive performance to unconstrained models while producing reasoning traces validated by an independent neuropathologist. In data-limited regimes, ConceptM3oE improves limited-data performance, increasing macro-F1 from 56.41% to 66.70% at small training sizes compared to non-concept-informed baselines, while also showing faster training convergence consistent with the regularizing effect of concept learning. This work offers a scalable path toward high-performance medical AI that is inherently verifiable and better aligned with the complex decision-making of clinical practice.
Xuan Wang, Zhongling Xu, Gopi Kannedhara +13
University of Texas at Austin, Austin, TX, USA · University of North Carolina at Chapel Hill, NC, USA · Dell Children’s Medical Center, Austin, TX, USA +2
Accurate brain tumor typing requires integrating heterogeneous clinical evidence, including magnetic resonance imaging (MRI), histopathology, and pathology reports, which are often incomplete at the time of diagnosis. We introduce CoRe-BT, a cross-modal radiology-pathology-text benchmark for brain tumor typing, designed to study robust multimodal learning under missing modality conditions. The dataset comprises 310 patients with multi-sequence brain MRI (T1, T1c, T2, FLAIR), including 95 cases with paired H&E-stained whole-slide pathology images and pathology reports. All cases are annotated with tumor type and grade, and MRI volumes include expert-annotated tumor masks, enabling both region-aware modeling and auxiliary learning tasks. Tumors are categorized into six clinically relevant classes capturing the heterogeneity of common and rare glioma subtypes. We evaluate tumor typing under variable modality availability by comparing MRI-only models with multimodal approaches that incorporate pathology information when present. Baseline experiments demonstrate the feasibility of multimodal fusion and highlight complementary modality contributions across clinically relevant typing tasks. CoRe-BT provides a grounded testbed for advancing multimodal glioma typing and representation learning in realistic scenarios with incomplete clinical data.
Juampablo E. Heras Rivera, Daniel K. Low, Xavier Xiong +5
University of Washington, Seattle, Washington 98105, USA · University of Washington School of Medicine, Seattle, Washington 98105, USA · Microsoft Health AI, Redmond, Washington 98052-6399, USA
Peripheral neuroblastic tumors (pNTs) are among the most common extracranial solid tumors in children, and accurate pathological subtyping is important for risk stratification and treatment planning. However, pNT subtyping on hematoxylin-eosin whole-slide images (WSIs) remains challenging because of limited pediatric tumor cohorts, marked histological heterogeneity, inter-observer variability, and the computational burden of existing WSI classifiers. To address these challenges, we propose CoPath, a framework consisting of CoHisNet and PathVote. CoHisNet is a lightweight multi-scale feature-fusion network for patch-level histopathological classification. By replacing the multilayer perceptron components in Swin Transformer blocks and the classification head with Kolmogorov-Arnold Network layers, CoHisNet improves nonlinear feature modeling under a compact architecture. Its multi-scale interaction and contrast-driven feature-enhancement design enables the model to capture both tissue-level structures and fine-grained cellular morphology. PathVote further incorporates pathology-informed tissue-component priors to aggregate patch-level predictions into WSI-level decisions. We validated CoPath on a private two-branch PpNTs cohort and the public BreakHis breast cancer histopathology dataset. Experimental results show that CoPath achieves competitive or superior performance compared with general image classifiers, pathology foundation models under linear probing, and pathology-specific classification models, while maintaining substantially lower computational complexity. The source code is available at https://github.com/JSLiam94/CoPath.
Zhu Zhu, Shuo Jiang, Jingyuan Zheng +7
Children’s Hospital, Zhejiang University School of Medicine, Hangzhou, 310000, China · Hangzhou Dianzi University, Hangzhou 310018, China · Chinese University of Hong Kong, Hong Kong SAR, China +1