MIFR: A Modality-Invariant and Fair Representation Framework for Skin Disease Classification

Asonyu Senge Njih; Yvan Guifo Fodjo; Vianney Kengne Tchendji; Jerry Lacmou Zeutouo; Kerol Djoumessi

cs.CVAug 19, 2026

MIFR: A Modality-Invariant and Fair Representation Framework for Skin Disease Classification

Authors: Asonyu Senge Njih, Yvan Guifo Fodjo, Vianney Kengne Tchendji, Jerry Lacmou Zeutouo, Kerol Djoumessi

Organizations: Department of Mathematics and Computer Science, University of Dschang, Cameroon · EFREI Research Lab, Université Paris-Panthéon-Assas, France · MIS, Université de Picardie Jules Verne, France · Hertie Institute for AI in Brain Health, University of Tübingen, Germany

Abstract

Skin diseases represent a major global public health burden, yet machine learning tools developed to assist in their diagnosis suffer from two critical limitations: reliance on only one modality for diagnosis and systematic performance disparities across skin tones. While existing approaches address each challenge separately, this work proposes a modality-invariant framework with fair representation (MIFR) for skin disease classification. The architecture pairs clinical photographs with dermoscopic images using ViT-based encoders, projecting each input into a high-dimensional embedding space via modality-specific projection heads. The resulting model is trained with a five-component multi-objective loss including weighted cross-entropy for classification, confusion and skin-type classification losses for fairness, per-modality supervised contrastive loss for class alignment, and a modality-invariance loss for clinical and dermoscopic modality alignment. Experiments on the HIBA+Derm7pt paired dataset and the external PAD-UFES-20 and ISIC 2019 datasets showed that modality-invariant representation learning provides competitive predictive performance compare to relevant baseline models and competitive fairness on the internal dataset. t-SNE visualizations confirmed that clinical and dermoscopic embeddings of the same disease are geometrically aligned, validating the joint objectives.

Figures & tables

Figure 1: Overview of the proposed architecture . Clinical and dermoscopic streams are processed by independent encoders and projected into a shared embedding space via modality-specific projection heads. Four training branches jointly optimize classification ( $L_{\text{cls}}$ ), skin-tone fairness via adversarial confusion and skin-tone losses ( $L_{\text{conf}}$ , $L_{s}$ ), cross-modality contrastive alignment ( $L_{\text{con}}$ ), and paired modality invariance ( $L_{\text{MI}}$ ).

Dataset	# Samples	Retained	Melanoma	Nevus	BCC
HIBA [ 15 ]	1,616	1,195	253	602	340
Derm7pt [ 9 ]	2,022	1,738	504	1,150	84
PAD-UFES-20 [ 14 ]	2,098	1,141	52	244	845
ISIC 2019 [ 3 ]	33,769	25,517	5,849	15,370	4,298

Table 1: Dataset statistics after filtering to the target classes (MEL, NV, BCC).

Dataset	Model	Performance			Fairness
Dataset	Model	Acc $\uparrow$	AUC $\uparrow$	F1-Score $\uparrow$	$\Delta$ Acc $\downarrow$	PQD $\uparrow$
HIBA+Derm7pt	BASE	.9313	.9738	.9165	.0691	.6667
	BASE+SA	.9352	.9813	.9226	.0562	.6667
	BASE+Contr	.9284	.9683	.9142	.0887	.8298
	BASE+MI	.9188	.9619	.9001	.0394	.8723
	BASE+Contr+MI	.9043	.9664	.8883	.0265	.8511

Table 2: Results on internal and external datasets: ablations and state-of-the-art comparison. The best values are in bold , and the second are underlined

Figure 2: Left: t-SNE of the full model colored by imaging modality, showing alignment of clinical and dermoscopic embeddings. Right: t-SNE colored by Fitzpatrick skin type, showing near-perfect skin-tone mixing within disease clusters.

Explore similar work

JI-ADF: Joint-Individual Learning with Adaptive Decision Fusion for Multimodal Skin Lesion Classification
Apr 30, 2026Phan Nguyen, Dat Cao, Hien Kha +4Skin Lesion Classification Medical Diagnosis
1KAIST · 2Taipei Medical University · 3Yale University
Multimodal Skin Lesion Classification with Swin Transformer and Clinical Metadata Fusion
Aug 4, 2026Nethmi Pathirana, Isuru Munasinghe, Dileeka AlwisSkin Lesion Classification Multimodal Classification
School of Computing, Informatics Institute of Technology, Colombo, Sri Lanka · Department of Electronic and Telecommunication Engineering, University of Moratuwa, Katubedda, Sri Lanka
SkinCLIP-VL: Consistency-Aware Vision-Language Learning for Multimodal Skin Cancer Diagnosis
Mar 22, 2026Zhixiang Lu, Shijie Xu, Kaicheng Yan +6Medical VLMs VLM Adaptation
Xi’an Jiaotong-Liverpool University, China · University of Liverpool, United Kingdom

Apr 30, 2026J/K move · Enter open · S save

JI-ADF: Joint-Individual Learning with Adaptive Decision Fusion for Multimodal Skin Lesion Classification

Phan Nguyen, Dat Cao, Hien Kha, Hien Chu+3

1KAIST · 2Taipei Medical University · 3Yale University

Skin Lesion Classification Medical Diagnosis Multimodal Classification Medical Image Classification Multimodal Fusion

Open paper AskPDF arXiv

Skin lesion classification is essential for early dermatological diagnosis, yet many existing computer-aided systems rely primarily on dermoscopic images and underutilize the multimodal evidence routinely available in clinical practice. To address this gap, we propose JI-ADF, a trimodal deep learning framework that integrates dermoscopic images, clinical photographs, and structured patient metadata for clinically grounded skin lesion classification. The proposed architecture combines joint multimodal representation learning with modality-specific auxiliary supervision and an adaptive decision fusion mechanism that dynamically calibrates modality contributions on a per-sample basis. To enhance cross-modal reasoning while preserving modality-specific evidence, we further introduce a multimodal fusion attention (MMFA) module. We evaluate JI-ADF on the large-scale MILK10k benchmark, which reflects real-world clinical acquisition conditions and severe class imbalance. The proposed method demonstrates strong and well-balanced performance across lesion categories, improving sensitivity and Dice score while maintaining high specificity and good calibration. Extensive analyses, including modality ablation, calibration evaluation, and Grad-CAM visualization, further confirm the robustness and clinically meaningful behavior of the model. These results indicate that JI-ADF provides a reliable and practical foundation for multimodal skin lesion classification in real-world clinical settings.

Abstract

Explore similar work

JI-ADF: Joint-Individual Learning with Adaptive Decision Fusion for Multimodal Skin Lesion Classification

Multimodal Skin Lesion Classification with Swin Transformer and Clinical Metadata Fusion

SkinCLIP-VL: Consistency-Aware Vision-Language Learning for Multimodal Skin Cancer Diagnosis