Explainable Multimodal Deep Learning Integrating Imaging and Clinical Data for Oral Potentially Malignant Disorder Detection

Ruilin You; Yihan Wang; Jiabin Chen; Cherie Wink; Petra Wilder-Smith; Rongguang Liang; Bofan Song

eess.IVSep 3, 2026

Explainable Multimodal Deep Learning Integrating Imaging and Clinical Data for Oral Potentially Malignant Disorder Detection

Authors: Ruilin You, Yihan Wang, Jiabin Chen, Cherie Wink, Petra Wilder-Smith, Rongguang Liang, Bofan Song

Organizations: Wyant College of Optical Sciences, University of Arizona, Tucson, AZ, USA · Beckman Laser Institute & Medical Clinic, University of California Irvine, Irvine, CA, USA

Abstract

Oral potentially malignant disorders (OPMDs) are critical precursors to oral cancer, yet clinical detection remains challenging because of substantial phenotypic heterogeneity and overlap with benign conditions. Although image-based deep learning shows promise for automated screening, visual information alone may be insufficient in real-world settings, where diagnostic decisions also rely on patient-specific risk factors. We developed M2-OPMDNet, a multimodal deep learning framework that integrates co-registered white-light and autofluorescence intraoral images with structured clinical information for OPMD detection. A customized questionnaire was designed to capture clinically relevant risk factors and symptoms in a standardized, reproducible format for integration with image-derived features. Multiple image encoders, including conventional convolutional neural networks and foundation model-based architectures, were evaluated using a prospectively collected dataset reflecting real-world screening conditions. Model interpretability was assessed using SHapley Additive exPlanations (SHAP) to quantify feature- and modality-level contributions. M2-OPMDNet achieved an AUC of 0.952, outperforming unimodal approaches and showing improved performance for visually subtle lesions. SHAP analysis demonstrated that structured clinical variables contributed substantially to risk estimation and complemented imaging features. These results demonstrate that explainable multimodal learning combining white-light and autofluorescence imaging with structured clinical data can provide accurate, transparent, and clinically grounded OPMD detection. M2-OPMDNet offers a scalable framework for real-world oral cancer screening and decision support.

Figures & tables

Figure 1: Overview of Multi-Modal Oral Potentially Malignant Disorders (OPMDs) Detection Nerual Network (M2-OPMDNet). a) The multi-modal oral image network captures both white-light and fluorescent images of the exact same intraoral region. These images are passed into feature extractors to generate embeddings, which are then concatenated with questionnaire embeddings for final classification. b) The top 5 questionnaire items with the greatest impact on the final results are shown. The color indicates the patient’s answer: yes (orange) or no (black). The x-axis represents how each question influences the classification decision, either pushing it toward a more positive or negative outcome. Each point corresponds to a patient case. All 23 questions are listed in the appendix. c) We compared classification accuracy across different backbone structures for the feature extractors, specifically between convolutional neural networks and attention-based transformers.

Figure 2: Comparison between white-light and fluorescent images across various oral cavity regions. The left panel shows positive cases, while the right panel shows negative cases.

Figure 3: ROC curve comparison of M2-OPMDNet with and without questionnaire input. All results are obtained using five-fold cross-validation, with the mean shown as a solid line and the variance indicated by the shaded error region. The area under the curve (AUC) for each image-extractor backbone is reported in the legend. (a) and (b) present results using only dual-modality oral images as input. (c) and (d) show results from the multi-modal neural network that incorporates both dual-modality oral images and questionnaire data.

Figure 4: SHAP case studies on the impact of all modalities. Panels (a–c) show three individual cases from a trained M2-OPMDNet using VGG-19 as the image feature extractor, illustrating how each modality influences the final prediction. These explanations provide clinicians with additional insight before making a final OPMD diagnosis when using a neural-network classifier.The full list of the questions is provided in the appendix. Table 1

Figure 5: Ranking of the most impactful clinical questions (n=23) on neural network outputs via SHAP. We group each question’s effect into four relationships: positive correlation, negative correlation, unidirectional positive correlation, and asymmetric conditional effect. The full list of OPMD-related clinical questions is provided in the appendix Table 1

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

#	Question
1	Has the patient had a previous history of oral squamous cell carcinoma, severe dysplasia/carcinoma-in-situ, or mild/moderate oral dysplasia?
2	Has the patient had previous head/neck radiation therapy for a non-squamous cell carcinoma tumor?
3	Does the patient have oral leukoplakia (white plaque that cannot be wiped off and cannot be diagnosed as any other condition or disease)?
4	Does the patient have oral erythroplakia (red plaque that cannot be diagnosed as any other condition or disease)?
5	Does the patient have a non-healing oral ulcer (greater than two weeks)?
6	Does the patient have any unexplained oral swelling, oral numbness, or tingling sensation?

Appendix

Table 1: Oral Cancer/OPMD Risk and Symptom Questionnaire

#	Question
13	Does the patient use smokeless tobacco (e.g., snus, moist snuff or chewing tobacco)?
14	Is the patient a moderate to heavy alcohol drinker (more than two standard drinks per day)?
15	Does the patient have any other habits such as smoking bidis, kreteks (clove cigarettes), marijuana, use of betel quid (paan) or areca nut?
16	Does the patient experience frequent sunburns of the lip?
17	Does the patient have poor nutrition?
18	Does the patient have oral human papillomavirus colonization (high-risk subtypes 16/18 as confirmed by salivary diagnostics)?

Appendix

Table 1: Oral Cancer/OPMD Risk and Symptom Questionnaire (continued)

Multimodal Classification

Multimodal classification combines images, text, audio, signals, metadata, or other inputs to assign labels more accurately and reliably.

See all 201 papers

An approach with Visual and Tabular Mamba to multimodal medical data using Mixed Fusion
Jun 17, 2026Matheus B. Rocha, Gustavo B. Dettogni, Renato A. KrohlingMamba Multimodal Classification
Labcin - Nature Inspired Computing Lab, Federal University of Espírito Santo, Vitória, Brazil · PPGI - Graduate Program in Computer Science, Federal University of Espírito Santo, Vitória, Brazil
GlaBoost: A Multimodal Structured Framework for Glaucoma Risk Stratification
Aug 3, 2025Cheng Huang, Zeyu Han, Weizheng Xie +4Multimodal Fusion Clinical Risk Prediction
Department of Computer Science, Southern Methodist University, Dallas, TX 75205, USA · Department of Ophthalmology, UT Southwestern Medical Center, Dallas, TX 75390, USA · High Performance Computing Program, National Aeronautics and Space Administration, Washington, DC 20546, USA
Bridging the gap between Performance and Interpretability: An Explainable Disentangled Multimodal Framework for Cancer Survival Prediction
Mar 2, 2026Aniek Eijpe, Soufyan Lakbir, Melis Erdal Cesur +4Cancer Survival Prediction Multimodal Survival Analysis
AI Technology for Life, Department of Information and Computing Sciences, Department of Biology, Utrecht University, Utrecht, The Netherlands · Department of Metabolic Diseases, Wilhelmina Children’s Hospital, University Medical Center Utrecht, Utrecht, The Netherlands · Regenerative Medicine Center Utrecht, Utrecht, The Netherlands +3

Jun 17, 2026J/K move · Enter open · S save

An approach with Visual and Tabular Mamba to multimodal medical data using Mixed Fusion

Matheus B. Rocha, Gustavo B. Dettogni, Renato A. Krohling

Labcin - Nature Inspired Computing Lab, Federal University of Espírito Santo, Vitória, Brazil · PPGI - Graduate Program in Computer Science, Federal University of Espírito Santo, Vitória, Brazil

Mamba Multimodal Classification Oncology Multimodal Model Interpretability Multimodal Fusion

Open paper AskPDF arXiv

This article presents a complementary approach for integrating multimodal medical data in cancer classification, based on state space models represented by the Mamba architecture. To this end, a mixed multimodal fusion architecture, called Mixed Fusion, was employed and developed to enhance the interpretability of the decision-making process. The proposed approach explores two variants of Mamba: one dedicated to visual processing, responsible for classifying the lesion image and generating probabilities associated with the target classes, and another focused on tabular processing, which uses these probabilities together with clinical and/or sociodemographic data to produce the final diagnosis. The experiments were conducted on two medical datasets: PAD-UFES-20, composed of clinical images and information associated with skin lesions, and NDB-UFES, consisting of histopathological images and sociodemographic data related to oral cancer. The results indicate slightly lower performance in balanced accuracy, compared with Transformer-based approaches, on PAD-UFES-20, and superior performance on NDB-UFES. Additionally, substantial gains were observed in the recall metric. Furthermore, the adoption of the Mixed Fusion architecture enables the application of the Shapley Additive Explanations (SHAP) method, increasing the interpretability of the results. These findings indicate that Mamba-based models constitute a suitable alternative for multimodal classification in medical data, especially in scenarios in which sensitivity is a relevant requirement.

Abstract

Multimodal Classification

Explore similar work

An approach with Visual and Tabular Mamba to multimodal medical data using Mixed Fusion

GlaBoost: A Multimodal Structured Framework for Glaucoma Risk Stratification

Bridging the gap between Performance and Interpretability: An Explainable Disentangled Multimodal Framework for Cancer Survival Prediction