OraRefer: Segmentation-Guided Multi-Photograph Fusion for Oral Cancer Screening with Guaranteed Patient-Level Sensitivity

Rupam Mukherjee; Mehakdeep Singh; Shirin Dasgupta; Subhamoy Mandal

cs.CVNov 5, 2026

OraRefer: Segmentation-Guided Multi-Photograph Fusion for Oral Cancer Screening with Guaranteed Patient-Level Sensitivity

Authors: Rupam Mukherjee, Mehakdeep Singh, Shirin Dasgupta, Subhamoy Mandal

Organizations: Department of Medical Science and Technology Indian Institute of Technology Kharagpur, India · Dr. B. C. Roy Multi-Speciality Medical Research Centre Indian Institute of Technology Kharagpur, India

Abstract

Oral cancer and the oral potentially malignant disorders (OPMD) that can precede it are visible on the oral mucosa, which makes smartphone photographs a promising tool for screening. Deep learning models for this task classify individual photographs. However, screening refers patients, not photographs: a patient is usually photographed several times, and the referral decision must miss no more than a set share of patients with disease. We propose OraRefer, a patient-level screening model that reads each photograph with a segmentation-guided foundation model that conditions its class decision on the predicted lesion, combines all photographs of a patient with risk factors, and refers through a threshold with a finite-sample sensitivity guarantee. The model localizes the mouth itself, so inference needs no annotation. On 3000 photographs of 714 patients, patient-level fusion reaches an AUROC of 0.919 and outperforms attention-based multiple-instance learning. At a 90% sensitivity target, OraRefer detects 95% of patients with OPMD or cancer and misses 3 of 61 patients with cancer, at a specificity of 0.610. Because the threshold is calibrated without retraining, each clinic can set its own guaranteed operating point on a local calibration sample.

Figures & tables

Figure 1: OraRefer needs no annotation at inference, fuses a patient’s photographs with risk factors, and refers with guaranteed sensitivity. (a) Pass 1 predicts the mouth box (dashed); pass 2 segments the crop ( $\hat{S}$ ) and pools its tokens with the lesion map $\hat{S}_{\ell}$ . Shown: a held-out patient with OPMD [ 15 ] . (b) Per-photograph outputs are aggregated per patient. (c) The threshold $t$ on the patient score $s$ and Mondrian triage ( $C$ : OPMD or cancer, $N$ : neither) are calibrated on held-out patients without refitting.

	Macro AUROC	$\Delta$ [95% interval]	Folds lower	Cancer AUROC	Macro-F1	Lesion Dice
OraRefer	0.893	–	–	0.942	0.704	0.580
w/o segmentation guidance	0.869	$-$ 0.023 [ $-$ 0.033, $-$ 0.015]	5	0.923	0.688	–
w/o lesion-guided pooling	0.884	$-$ 0.009 [ $-$ 0.016, $-$ 0.002]	4	0.919	0.703	0.587
w/o region-affinity attention	0.893	0.001 [ $-$ 0.005, $-$ 0.007]	1	0.943	0.703	0.561
w/o supervised contrastive loss	0.885	$-$ 0.007 [ $-$ 0.014, $-$ 0.002]	4	0.934	0.705	0.581
with annotated mouth box	0.891	$-$ 0.001 [ $-$ 0.003, $-$ 0.000]	3	0.940	0.707	0.583

Table 1: Segmentation guidance drives the image model, and self-localization matches annotated crops.

\Delta

: change in macro AUROC from OraRefer; folds lower: number of the five folds in which it falls. A second training seed changes macro AUROC by

-

0.005.

	AUROC	Sens.	Spec.	Ref.
Risk factors only	0.760 [0.724, 0.795]	0.958	0.201	88%
Image only	0.909 [0.888, 0.930]	0.958	0.562	72%
Max of $q_{ij}$	0.878 [0.853, 0.903]	0.950	0.538	72%
Mean of $q_{ij}$	0.889 [0.865, 0.914]	0.958	0.502	74%
Noisy-OR	0.851 [0.824, 0.879]	0.961	0.387	80%
ABMIL	0.890 [0.867, 0.914]	0.953	0.508	74%

Table 2: OraRefer has the highest patient-level AUROC. Guaranteed-sensitivity operating points (

\tau=0.90

\delta=0.05

); Ref.: referral rate. ABMIL and CAR-MIL AUROCs differ only in the fourth decimal.

Figure 2: The guaranteed-sensitivity threshold exceeds every target, and Mondrian triage sends few referable patients to routine care. (a) Patient-level ROC curves (AUROC) with the operating points of both thresholds at

\tau=0.90

. (b) Sensitivity (left; dashed: sensitivity

=\tau

) and specificity (right) of both thresholds against the target

\tau

. (c) Triage (

\alpha=0.05

) of patients grouped by their most severe photograph label, mean over 20 calibration re-splits.

Oncology

Oncology studies cancer biology, prevention, diagnosis, treatment, and data-driven ways to improve patient care.

See all 265 papers

Explainable Multimodal Deep Learning Integrating Imaging and Clinical Data for Oral Potentially Malignant Disorder Detection
Sep 3, 2026Ruilin You, Yihan Wang, Jiabin Chen +4Multimodal Classification Oncology
Wyant College of Optical Sciences, University of Arizona, Tucson, AZ, USA · Beckman Laser Institute & Medical Clinic, University of California Irvine, Irvine, CA, USA
Robust Lightweight Deep Learning Models for Oral Cancer Screening
Aug 21, 2026Siddhant Bharadwaj, Aakash Shedsale, Tejashree Subramanya +6Medical Image Classification Lightweight CNNs
Indian Institute of Science · KLES’ Institute of Dental Sciences · Indian Institue of Science +1
TLNM: Externally Validated Tooth Detection, Numbering and Segmentation from Smartphone Photographs Using Mask R-CNN
Aug 6, 2026Arash Nedaei, Henna Tiensuu, Elina Väyrynen +2Medical Imaging Instance Segmentation
Biomimetics and Intelligent Systems Group, Faculty of Information Technology and Electrical Engineering, University of Oulu, Oulu, Finland · Research Unit of Population Health, Faculty of Medicine, University of Oulu, Oulu, Finland

Sep 3, 2026J/K move · Enter open · S save

Explainable Multimodal Deep Learning Integrating Imaging and Clinical Data for Oral Potentially Malignant Disorder Detection

Ruilin You, Yihan Wang, Jiabin Chen, Cherie Wink+3

Wyant College of Optical Sciences, University of Arizona, Tucson, AZ, USA · Beckman Laser Institute & Medical Clinic, University of California Irvine, Irvine, CA, USA

Multimodal Classification Oncology Explainable Medical Image Analysis

Open paper AskPDF arXiv

Oral potentially malignant disorders (OPMDs) are critical precursors to oral cancer, yet clinical detection remains challenging because of substantial phenotypic heterogeneity and overlap with benign conditions. Although image-based deep learning shows promise for automated screening, visual information alone may be insufficient in real-world settings, where diagnostic decisions also rely on patient-specific risk factors. We developed M2-OPMDNet, a multimodal deep learning framework that integrates co-registered white-light and autofluorescence intraoral images with structured clinical information for OPMD detection. A customized questionnaire was designed to capture clinically relevant risk factors and symptoms in a standardized, reproducible format for integration with image-derived features. Multiple image encoders, including conventional convolutional neural networks and foundation model-based architectures, were evaluated using a prospectively collected dataset reflecting real-world screening conditions. Model interpretability was assessed using SHapley Additive exPlanations (SHAP) to quantify feature- and modality-level contributions. M2-OPMDNet achieved an AUC of 0.952, outperforming unimodal approaches and showing improved performance for visually subtle lesions. SHAP analysis demonstrated that structured clinical variables contributed substantially to risk estimation and complemented imaging features. These results demonstrate that explainable multimodal learning combining white-light and autofluorescence imaging with structured clinical data can provide accurate, transparent, and clinically grounded OPMD detection. M2-OPMDNet offers a scalable framework for real-world oral cancer screening and decision support.

Abstract

Oncology

Explore similar work

Explainable Multimodal Deep Learning Integrating Imaging and Clinical Data for Oral Potentially Malignant Disorder Detection

Robust Lightweight Deep Learning Models for Oral Cancer Screening

TLNM: Externally Validated Tooth Detection, Numbering and Segmentation from Smartphone Photographs Using Mask R-CNN