eess.IVSep 3, 2026

Explainable Multimodal Deep Learning Integrating Imaging and Clinical Data for Oral Potentially Malignant Disorder Detection

Authors: Ruilin You, Yihan Wang, Jiabin Chen, Cherie Wink, Petra Wilder-Smith, Rongguang Liang, Bofan Song

Organizations: Wyant College of Optical Sciences, University of Arizona, Tucson, AZ, USA · Beckman Laser Institute & Medical Clinic, University of California Irvine, Irvine, CA, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Oral potentially malignant disorders (OPMDs) are critical precursors to oral cancer, yet clinical detection remains challenging because of substantial phenotypic heterogeneity and overlap with benign conditions. Although image-based deep learning shows promise for automated screening, visual information alone may be insufficient in real-world settings, where diagnostic decisions also rely on patient-specific risk factors. We developed M2-OPMDNet, a multimodal deep learning framework that integrates co-registered white-light and autofluorescence intraoral images with structured clinical information for OPMD detection. A customized questionnaire was designed to capture clinically relevant risk factors and symptoms in a standardized, reproducible format for integration with image-derived features. Multiple image encoders, including conventional convolutional neural networks and foundation model-based architectures, were evaluated using a prospectively collected dataset reflecting real-world screening conditions. Model interpretability was assessed using SHapley Additive exPlanations (SHAP) to quantify feature- and modality-level contributions. M2-OPMDNet achieved an AUC of 0.952, outperforming unimodal approaches and showing improved performance for visually subtle lesions. SHAP analysis demonstrated that structured clinical variables contributed substantially to risk estimation and complemented imaging features. These results demonstrate that explainable multimodal learning combining white-light and autofluorescence imaging with structured clinical data can provide accurate, transparent, and clinically grounded OPMD detection. M2-OPMDNet offers a scalable framework for real-world oral cancer screening and decision support.

Figures & tables

Figure 1: Overview of Multi-Modal Oral Potentially Malignant Disorders (OPMDs) Detection Nerual Network (M2-OPMDNet). a) The multi-modal oral image network captures both white-light and fluorescent images of the exact same intraoral region. These images are passed into feature extractors to generate embeddings, which are then concatenated with questionnaire embeddings for final classification. b) The top 5 questionnaire items with the greatest impact on the final results are shown. The color indicates the patient’s answer: yes (orange) or no (black). The x-axis represents how each question influences the classification decision, either pushing it toward a more positive or negative outcome. Each point corresponds to a patient case. All 23 questions are listed in the appendix. c) We compared classification accuracy across different backbone structures for the feature extractors, specifically between convolutional neural networks and attention-based transformers.
Figure 2: Comparison between white-light and fluorescent images across various oral cavity regions. The left panel shows positive cases, while the right panel shows negative cases.
Figure 3: ROC curve comparison of M2-OPMDNet with and without questionnaire input. All results are obtained using five-fold cross-validation, with the mean shown as a solid line and the variance indicated by the shaded error region. The area under the curve (AUC) for each image-extractor backbone is reported in the legend. (a) and (b) present results using only dual-modality oral images as input. (c) and (d) show results from the multi-modal neural network that incorporates both dual-modality oral images and questionnaire data.
Figure 4: SHAP case studies on the impact of all modalities. Panels (a–c) show three individual cases from a trained M2-OPMDNet using VGG-19 as the image feature extractor, illustrating how each modality influences the final prediction. These explanations provide clinicians with additional insight before making a final OPMD diagnosis when using a neural-network classifier.The full list of the questions is provided in the appendix. Table 1
Figure 5: Ranking of the most impactful clinical questions (n=23) on neural network outputs via SHAP. We group each question’s effect into four relationships: positive correlation, negative correlation, unidirectional positive correlation, and asymmetric conditional effect. The full list of OPMD-related clinical questions is provided in the appendix Table 1
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

#Question
1Has the patient had a previous history of oral squamous cell carcinoma, severe dysplasia/carcinoma-in-situ, or mild/moderate oral dysplasia?
2Has the patient had previous head/neck radiation therapy for a non-squamous cell carcinoma tumor?
3Does the patient have oral leukoplakia (white plaque that cannot be wiped off and cannot be diagnosed as any other condition or disease)?
4Does the patient have oral erythroplakia (red plaque that cannot be diagnosed as any other condition or disease)?
5Does the patient have a non-healing oral ulcer (greater than two weeks)?
6Does the patient have any unexplained oral swelling, oral numbness, or tingling sensation?
Appendix
Table 1: Oral Cancer/OPMD Risk and Symptom Questionnaire
#Question
13Does the patient use smokeless tobacco (e.g., snus, moist snuff or chewing tobacco)?
14Is the patient a moderate to heavy alcohol drinker (more than two standard drinks per day)?
15Does the patient have any other habits such as smoking bidis, kreteks (clove cigarettes), marijuana, use of betel quid (paan) or areca nut?
16Does the patient experience frequent sunburns of the lip?
17Does the patient have poor nutrition?
18Does the patient have oral human papillomavirus colonization (high-risk subtypes 16/18 as confirmed by salivary diagnostics)?
Appendix
Table 1: Oral Cancer/OPMD Risk and Symptom Questionnaire (continued)
More in this topic

Multimodal Classification

Multimodal classification combines images, text, audio, signals, metadata, or other inputs to assign labels more accurately and reliably.

See all 200 papers

Explore similar work

CardsList
Jun 17, 2026

An approach with Visual and Tabular Mamba to multimodal medical data using Mixed Fusion

This article presents a complementary approach for integrating multimodal medical data in cancer classification, based on state space models represented by the Mamba architecture. To this end, a mixed multimodal fusion architecture, called Mixed Fusion, was employed and developed to enhance the interpretability of the decision-making process. The proposed approach explores two variants of Mamba: one dedicated to visual processing, responsible for classifying the lesion image and generating probabilities associated with the target classes, and another focused on tabular processing, which uses these probabilities together with clinical and/or sociodemographic data to produce the final diagnosis. The experiments were conducted on two medical datasets: PAD-UFES-20, composed of clinical images and information associated with skin lesions, and NDB-UFES, consisting of histopathological images and sociodemographic data related to oral cancer. The results indicate slightly lower performance in balanced accuracy, compared with Transformer-based approaches, on PAD-UFES-20, and superior performance on NDB-UFES. Additionally, substantial gains were observed in the recall metric. Furthermore, the adoption of the Mixed Fusion architecture enables the application of the Shapley Additive Explanations (SHAP) method, increasing the interpretability of the results. These findings indicate that Mamba-based models constitute a suitable alternative for multimodal classification in medical data, especially in scenarios in which sensitivity is a relevant requirement.

Matheus B. Rocha, Gustavo B. Dettogni, Renato A. Krohling

Labcin - Nature Inspired Computing Lab, Federal University of Espírito Santo, Vitória, Brazil · PPGI - Graduate Program in Computer Science, Federal University of Espírito Santo, Vitória, Brazil

MambaMultimodal ClassificationOncologyMultimodal Model InterpretabilityMultimodal Fusion
AskPDF
Aug 3, 2025

GlaBoost: A Multimodal Structured Framework for Glaucoma Risk Stratification

Early and accurate glaucoma detection is critical to prevent irreversible vision loss, yet existing AI methods often rely on unimodal inputs and lack interpretability. We present GlaBoost, a multimodal gradient boosting framework that unifies three complementary signals for glaucoma risk prediction: fundus image embeddings from a pretrained convolutional encoder,free-text neuroretinal rim assessments encoded by a transformer-based language model, and structured ophthalmic biomarkers. These modalities are fused into a single representation and classified by an enhanced XGBoost model.On two real-world annotated datasets, GlaBoost consistently outperforms unimodal and generic multimodal baselines. Feature importance analysis highlights the cup-to-disc ratio, rim thinning, and the ISNT rule as the dominant predictors, yielding clinically consistent and interpretable decisions. GlaBoost offers a transparent and scalable foundation for multimodal decision support in ophthalmology.

Cheng Huang, Zeyu Han, Weizheng Xie +4

Department of Computer Science, Southern Methodist University, Dallas, TX 75205, USA · Department of Ophthalmology, UT Southwestern Medical Center, Dallas, TX 75390, USA · High Performance Computing Program, National Aeronautics and Space Administration, Washington, DC 20546, USA

Multimodal FusionClinical Risk PredictionGlaucoma Detection
AskPDF
Mar 2, 2026

Bridging the gap between Performance and Interpretability: An Explainable Disentangled Multimodal Framework for Cancer Survival Prediction

While multimodal survival prediction models are increasingly more accurate, their complexity often reduces interpretability, limiting insight into how different data sources influence predictions. To address this, we introduce DIMAFx, an explainable multimodal framework for cancer survival prediction that produces disentangled, interpretable modality-specific and modality-shared representations from histopathology whole-slide images and transcriptomics data. Across multiple cancer cohorts, DIMAFx achieves state-of-the-art performance and improved representation disentanglement. Leveraging its interpretable design and SHapley Additive exPlanations, DIMAFx systematically reveals key multimodal interactions and the biological information encoded in the disentangled representations. In breast cancer survival prediction, the most predictive features contain modality-shared information, including one capturing solid tumor morphology contextualized primarily by late estrogen response, where higher-grade morphology aligned with pathway upregulation and increased risk, consistent with known breast cancer biology. Key modality-specific features capture microenvironmental signals from interacting adipose and stromal morphologies. These results show that multimodal models can overcome the traditional trade-off between performance and explainability, supporting their application in precision medicine.

Aniek Eijpe, Soufyan Lakbir, Melis Erdal Cesur +4

AI Technology for Life, Department of Information and Computing Sciences, Department of Biology, Utrecht University, Utrecht, The Netherlands · Department of Metabolic Diseases, Wilhelmina Children’s Hospital, University Medical Center Utrecht, Utrecht, The Netherlands · Regenerative Medicine Center Utrecht, Utrecht, The Netherlands +3

Cancer Survival PredictionMultimodal Survival AnalysisMultimodal Model InterpretabilityMultimodal Disentangled Representation LearningComputational Pathology
AskPDF