PathMoE: Interpretable Multimodal Interaction Experts for Pediatric Brain Tumor Classification

Jian Yu; Joakim Nguyen; Jinrui Fang; Awais Naeem; Zeyuan Cao; Sanjay Krishnan; Nicholas Konz; Tianlong Chen

cs.CVMar 2, 2026

PathMoE: Interpretable Multimodal Interaction Experts for Pediatric Brain Tumor Classification

Authors: Jian Yu, Joakim Nguyen, Jinrui Fang, Awais Naeem, Zeyuan Cao, Sanjay Krishnan, Nicholas Konz, Tianlong Chen,

Organizations: University of Texas, Austin, TX, USA · Dell Children’s Medical Center, Austin, TX, USA · University of North Carolina at Chapel Hill, NC, USA · University of Nevada, Reno, NV, USA

Abstract

Accurate classification of pediatric central nervous system tumors remains challenging due to histological complexity and limited training data. While pathology foundation models have advanced whole-slide image (WSI) analysis, they often fail to leverage the rich, complementary information found in clinical text and tissue microarchitecture. To this end, we propose PathMoE, an interpretable multimodal framework that integrates H&E slides, pathology reports, and nuclei-level cell graphs via an interaction-aware mixture-of-experts architecture built on state-of-the-art foundation models for each modality. By training specialized experts to capture modality uniqueness, redundancy, and synergy, PathMoE employs an input-dependent gating mechanism that dynamically weights these interactions, providing sample-level interpretability. We evaluate our framework on two dataset-specific classification tasks on an internal pediatric brain tumor dataset (PBT) and external TCGA datasets. PathMoE improves macro-F1 from 0.762 to 0.799 (+0.037) on PBT when integrating WSI, text, and graph modalities; on TCGA, augmenting WSI with graph knowledge improves macro-F1 from 0.668 to 0.709 (+0.041). These results demonstrate significant performance gains over state-of-the-art image-only baselines while revealing the specific modality interactions driving individual predictions. This interpretability is particularly critical for rare tumor subtypes, where transparent model reasoning is essential for clinical trust and diagnostic validation.

Figures & tables

Figure 1: Overview of PathMoE . H&E WSIs, pathology reports, and nuclei graphs are encoded and fused via an interaction-aware mixture-of-experts module. An input-dependent gating network computes sample-specific weights to combine expert predictions into the final tumor classification. A vanilla fusion method is also depicted.

	Case	Label	Baseline	$\textbf{PM}_{\textbf{WTG}}$	$w_{W}$	$w_{T}$	$w_{G}$	Clinical Rationale
Case_1		HG-CNS	NGT	HG-CNS	0.232	0.167	0.173	Polyphenotypic tumor mimicking non-glial types; rhabdoid cells and monotonous morphology favor HG despite overlap.
Case_2		NGT	HG-CNS	NGT	0.208	0.184	0.205	High cell density and hemorrhage mimic HGs; lacks nuclear atypia/mitoses and shows pituitary morphology.
Case_3		NGT	LG-CNS	NGT	0.211	0.176	0.210	Mesenchymal non-glial tumor with spindled architecture; overlaps with low-grade gliomas but distinguishable by growth pattern.
Case_4		LG-CNS	HG-CNS	LG-CNS	0.233	0.181	0.144	Low-grade glioneuronal tumor with calcifications and mild nuclear density increase; lacks atypia, mitoses, or necrosis of high-grade tumors.

Table 3: Qualitative error analysis of the baseline and PathMoE (PM) across image–text (WT) and image–text–graph (WTG) settings. Each case shows the ground-truth label, model predictions, modality interaction weights (

w_{W}

w_{T}

w_{G}

), and a brief clinical rationale for the correct diagnosis. Nodes and edges (light blue) often aggregate in regions of cellular density learned by the GNN.

Brain Tumor Classification

Brain tumor classification uses medical images and molecular data to identify tumor types, grades, and subtypes for diagnosis and treatment planning.

See all 51 papers

ConceptM $^3$ oE: Concept-Guided Multimodal Mixture of Experts for Interpretable Computational Pathology
May 23, 2026Xuan Wang, Zhongling Xu, Gopi Kannedhara +13Computational Pathology Concept Bottleneck Models
University of Texas at Austin, Austin, TX, USA · University of North Carolina at Chapel Hill, NC, USA · Dell Children’s Medical Center, Austin, TX, USA +2
CoRe-BT: A Multimodal Radiology-Pathology-Text Benchmark for Robust Brain Tumor Typing
Mar 4, 2026Juampablo E. Heras Rivera, Daniel K. Low, Xavier Xiong +5Brain Tumor Classification Medical Image Benchmarks
University of Washington, Seattle, Washington 98105, USA · University of Washington School of Medicine, Seattle, Washington 98105, USA · Microsoft Health AI, Redmond, Washington 98052-6399, USA
Towards Accurate and Lightweight Peripheral Neuroblastic Tumor Diagnosis via Contrastive Multi-scale Pathological Image Analysis
Apr 18, 2025Zhu Zhu, Shuo Jiang, Jingyuan Zheng +7Computational Pathology Oncology
Children’s Hospital, Zhejiang University School of Medicine, Hangzhou, 310000, China · Hangzhou Dianzi University, Hangzhou 310018, China · Chinese University of Hong Kong, Hong Kong SAR, China +1

May 23, 2026J/K move · Enter open · S save

ConceptM $^3$ oE: Concept-Guided Multimodal Mixture of Experts for Interpretable Computational Pathology

Xuan Wang, Zhongling Xu, Gopi Kannedhara, Joakim Nguyen+12

University of Texas at Austin, Austin, TX, USA · University of North Carolina at Chapel Hill, NC, USA · Dell Children’s Medical Center, Austin, TX, USA+2

Computational Pathology Concept Bottleneck Models Healthcare Interpretable ML Multimodal Classification Oncology+4

Open paper AskPDF arXiv

Healthcare models are transitioning from unimodal prediction toward multimodal reasoning over heterogeneous diagnostic inputs. In computational pathology, for complex tumor subtypes where morphology alone can be challenging to distinguish, pathology reports and molecular measurements may provide additional diagnostic evidence alongside whole-slide images, yet existing models often fail to clarify how diverse signals assemble into recognizable diagnostic concepts. We propose ConceptM

^3

oE (Concept Multimodal MoE), which embeds concept formation directly within interaction-aware mixture-of-experts (MoE) pathways. The architecture decomposes evidence into modality-specific, redundant, and synergistic experts, which are then projected into structured concept bottlenecks mapping latent features to a hierarchy of morphology and biomarker concepts. To prevent the information loss typical of interpretable bottlenecks, we utilize residual pathways within each expert to allow task-relevant signals to flow both through the concepts and directly to the final task prediction, so that high performance is maintained alongside interpretability. Across an institutional pediatric brain tumor cohort and a public glioma cohort, the framework delivers competitive performance to unconstrained models while producing reasoning traces validated by an independent neuropathologist. In data-limited regimes, ConceptM

^3

oE improves limited-data performance, increasing macro-F1 from 56.41% to 66.70% at small training sizes compared to non-concept-informed baselines, while also showing faster training convergence consistent with the regularizing effect of concept learning. This work offers a scalable path toward high-performance medical AI that is inherently verifiable and better aligned with the complex decision-making of clinical practice.

Abstract

Brain Tumor Classification

Explore similar work

ConceptM3^33oE: Concept-Guided Multimodal Mixture of Experts for Interpretable Computational Pathology

CoRe-BT: A Multimodal Radiology-Pathology-Text Benchmark for Robust Brain Tumor Typing

Towards Accurate and Lightweight Peripheral Neuroblastic Tumor Diagnosis via Contrastive Multi-scale Pathological Image Analysis

ConceptM $^3$ oE: Concept-Guided Multimodal Mixture of Experts for Interpretable Computational Pathology