Smartphone clinical images provide a low-cost data source for computer-aided skin lesion classification in resource-constrained settings, but images alone cannot fully capture patients' medical histories and lesion symptoms. To address the lack of structured organization among heterogeneous clinical variables, the ambiguity between observed and imputed values, and the risk that clinical information may interfere with visual evidence, we propose GAMF-Net, a grouped availability-aware multimodal fusion network. Ten clinical variables are divided into three semantic groups---demographics, history-related risk, and lesion presentation---and each group jointly encodes field values and field-level availability masks. The image branch produces the base classification logits, while each clinical group generates image-conditioned class-level residual evidence. Only available groups participate in availability-normalized equal-weight fusion; when all clinical information is unavailable, the model reduces exactly to the image-only pathway. We evaluate the method on PAD-UFES-20 under a patient-disjoint five-fold protocol with separate training, validation, and test partitions. GAMF-Net achieves an Accuracy of 74.17±2.85%, a balanced accuracy (BACC) of 70.11±2.11%, a Macro-F1 of 68.85±0.85%, and a Macro-AUROC of 93.42±1.20%, outperforming all image-only and multimodal baselines reimplemented under a unified protocol. Compared with the strongest baseline, MetaBlock, the corresponding improvements are 2.37, 2.59, 4.14, and 1.02 percentage points, respectively. Ablation results show that clinical information substantially improves class-balanced performance, constrained equal-weight fusion is more effective than dynamic fusion, and the lesion-presentation group provides the most informative clinical complement.
Figures & tables
Figure 1: Overview of GAMF-Net. Overview of GAMF-Net: image-anchored prediction with availability-aware grouped clinical residuals.
Figure 2: Representative smartphone clinical images of the five skin lesion classes in PAD-UFES-20.
Model
Accuracy
BACC
Macro-F1
Macro-AUROC
Image-only ResNet18
64.93±2.67
61.53±3.11
58.06±2.69
88.69±1.20
Direct Concat
62.88±4.10
61.65±4.42
55.89±3.49
87.84±1.65
MetaNet
68.84±4.97
64.32±2.22
62.03±3.13
90.71±1.33
Dynamic Feature Modulation
68.07±3.49
62.89±2.22
60.94±2.39
89.75±1.38
MetaBlock-SE
66.56±1.49
62.12±3.86
58.86±3.00
88.92±1.62
MetaBlock
71.80±1.74
67.52±2.64
64.71±3.06
92.40±0.95
Table 1: Patient-level five-fold test results (mean ± sample standard deviation, %). Bold values indicate the highest mean under the unified protocol.
Configuration
Accuracy
BACC
Macro-F1
Macro-AUROC
Fusion and Input Selection
Dynamic Fusion
70.04±4.68
67.99±3.42
66.28±4.74
92.13±1.49
Full-21 Fields
72.99±3.53
70.61±1.43
68.71±1.81
93.70±1.02
Full Model
74.17±2.85
70.11±2.11
68.85±0.85
93.42±1.20
Clinical Semantic Group Contributions
D Only
65.09±1.75
64.34±2.21
61.80±1.46
89.96±1.49
Table 2: Patient-level five-fold ablation results (mean ± sample standard deviation, %). The full model uses the same run as the main comparison. Bold values indicate the highest mean within each block; the Full Model row is repeated for reference.
Figure 3: Five-fold fusion/input and clinical-group comparisons. Error bars denote sample standard deviations. Lines are visual guides across discrete configurations. Image-only in (a) is a reference baseline.
Comparison
Δ BACC [95% CI]
BACC wins
Δ mF1 [95% CI]
mF1 wins
Full vs. MetaBlock
+2.60[−1.52,6.83]
4/5
+4.13[0.69,7.73]
5/5
Table 3: Patient-clustered paired bootstrap comparisons. Differences are Full Model minus the comparison model, in percentage points. Wins count folds in which the Full Model has the higher metric.
Figure 4: Fold-wise and class-wise analysis. (a) BACC across the five patient-disjoint test folds. (b) Row-normalized confusion matrix from the aggregated out-of-fold predictions of GAMF-Net.
Skin lesion classification is essential for early dermatological diagnosis, yet many existing computer-aided systems rely primarily on dermoscopic images and underutilize the multimodal evidence routinely available in clinical practice. To address this gap, we propose JI-ADF, a trimodal deep learning framework that integrates dermoscopic images, clinical photographs, and structured patient metadata for clinically grounded skin lesion classification. The proposed architecture combines joint multimodal representation learning with modality-specific auxiliary supervision and an adaptive decision fusion mechanism that dynamically calibrates modality contributions on a per-sample basis. To enhance cross-modal reasoning while preserving modality-specific evidence, we further introduce a multimodal fusion attention (MMFA) module. We evaluate JI-ADF on the large-scale MILK10k benchmark, which reflects real-world clinical acquisition conditions and severe class imbalance. The proposed method demonstrates strong and well-balanced performance across lesion categories, improving sensitivity and Dice score while maintaining high specificity and good calibration. Extensive analyses, including modality ablation, calibration evaluation, and Grad-CAM visualization, further confirm the robustness and clinically meaningful behavior of the model. These results indicate that JI-ADF provides a reliable and practical foundation for multimodal skin lesion classification in real-world clinical settings.
Phan Nguyen, Dat Cao, Hien Kha +4
1KAIST · 2Taipei Medical University · 3Yale University
Skin lesion classification plays an important role in supporting the early diagnosis of skin cancer. However, automated analysis remains challenging due to class imbalance, inter-class similarity, and intra-class variability in dermoscopic images. This paper proposes a multimodal classification framework that combines Swin Transformer-based image features with structured clinical metadata to improve diagnostic performance through integrated visual-context learning. Experiments on a publicly available dataset show that the proposed model achieves a test accuracy of 92.55% and a macro F1-score of 91.33%, with strong performance across minority classes. Temperature scaling is applied as a post-hoc calibration method, resulting in a reduction in expected calibration error and improving prediction reliability, while uncertainty estimation is incorporated to further assess the confidence of model predictions. Qualitative explainability analysis further shows that the model focuses on lesion regions during inference. Therefore, the results demonstrate that multimodal fusion, combined with calibration and interpretability analysis, provides an effective and trustworthy approach for automated skin lesion classification.
Nethmi Pathirana, Isuru Munasinghe, Dileeka Alwis
School of Computing, Informatics Institute of Technology, Colombo, Sri Lanka · Department of Electronic and Telecommunication Engineering, University of Moratuwa, Katubedda, Sri Lanka
Automated skin cancer classification from dermoscopic images remains challenging due to heterogeneous lesion structure, strong intra-class variability, and subtle visual differences between benign and malignant cases. Existing CNN/ViT pipelines typically rely on global or patch-level features and often combine patient metadata via late fusion, which limits spatially grounded multimodal reasoning. We present a novel region-based graph learning framework that explicitly models lesions as graphs of spatially coherent superpixel regions represented as frozen CNN features. To capture fine-grained lesion arrangements, we encode inter-regional geometry as edge attributes and introduce a dedicated metadata context node connected to all regions, providing structured integration of demographic/clinical variables within the same relational space. Node representations are updated using our edge-aware graph transformer followed by attention-driven propagation, and a final graph-level embedding for benign-malignant classification. Experiments on four public benchmarks demonstrate that explicit region-level relational modeling and graph-native multimodal fusion yield consistent gains over the state-of-the-art. Consequently, we establish a new graph-centric perspective in which CNN features are modeled as relational nodes and improved through contextual integration, yielding more expressive and robust classifications.
Muhammad Azeem, Tanveer Hussain, Amr Ahmed +1
Edge Hill University, Ormskirk, Lancashire, L39 4QP, United Kingdom