Contrastive Adapter Training and Consensus Knowledge Distillation for Multi-Source-Free Domain Adaptation in Skin Cancer Diagnosis

Ali Abedi; Q. M. Jonathan Wu; Ning Zhang; Farhad Pourpanah

cs.CVNov 2, 2026

Contrastive Adapter Training and Consensus Knowledge Distillation for Multi-Source-Free Domain Adaptation in Skin Cancer Diagnosis

Authors: Ali Abedi, Q. M. Jonathan Wu, Ning Zhang, Farhad Pourpanah

Organizations: University of Windsor, Department of Electrical and Computer Engineering · Queen’s University, Department of Electrical and Computer Engineering

Abstract

Skin cancer diagnosis, particularly differentiating melanoma from benign nevi, is challenging due to the visual similarity between lesions. Although convolutional neural networks (CNNs) and vision transformers (ViTs) have shown promising performance, their effectiveness often deteriorates on data from heterogeneous clinical sources. Conventional domain adaptation methods address domain shift but require access to source data during adaptation, which is often infeasible due to privacy regulations. Multi-source-free unsupervised domain adaptation (MSFDA) addresses this limitation by leveraging multiple labeled source domains to generalize to an unlabeled target domain without requiring access to source data. However, existing MSFDA methods rely on full backbone fine-tuning, leading to catastrophic forgetting and overfitting on small clinical datasets, and address domain shift at the aggregation stage without establishing a shared domain-invariant feature space. Furthermore, their reliance on hard pseudo-labels or confidence-weighted aggregation introduces noisy supervision signals under domain shift. To address these limitations, we propose CAT-CKD, consisting of two components: (1) contrastive adapter training (CAT), which trains lightweight ConvPass adapters within a frozen ViT backbone using supervised contrastive learning (SCL) to establish a shared domain-invariant feature space before source-specific model training, and (2) consensus knowledge distillation (CKD), which aggregates logits from multiple source models into a consensus supervisory signal and adapts a student model on unlabeled target data using KL divergence. Experiments on five publicly available skin lesion datasets show that CAT-CKD achieves an average AUROC of 86.1%, outperforming existing MSFDA methods while requiring only 4.3M trainable parameters. The code is available at https://github.com/A-Abedi/CAT_CKD.

Figures & tables

Figure 1: Overview of the proposed CAT-CKD framework. (a) CAT: ConvPass adapters are trained on pooled source-domain data using SCL, while the ViT backbone remains frozen to form a shared domain-invariant feature space. Domain and class imbalance are addressed via weighted sampling. (b) Source model training: Each domain independently trains a bottleneck and classification head on the frozen backbone with binary CE loss. (c) CKD: Source model logits on unlabeled target samples are averaged to form a consensus signal, which supervises the student model through KL divergence. Samples with the same border color belong to the same source domain. 2 2 2 Distillation icon created by Sir.Vector - Flaticon.

Dataset	Melanoma	Nevus
ISIC 2018 [ 7 , 44 ]	1113	6705
ISIC 2020 [ 41 ]	584	5193
PAD-UFES-20 [ 36 ]	52	244
Derm7pt-Derm [ 26 ]	90	256
Derm7pt-Clinic [ 26 ]	90	256

Table 1: Summary of the number of samples in skin lesion datasets used in the study.

Figure 2: Pairwise Fréchet Distance between feature distributions of the five skin lesion datasets, where higher values indicate greater domain shift. The substantial distances observed across dataset pairs confirm the presence of significant inter-domain variability. This result motivates the use of domain adaptation.

Figure 3: Demonstration of data augmentations applied to a skin lesion images in our proposed method.

Model	MS	SF	ISIC 2018	ISIC 2020	PAD-UFES-20	Derm7pt-Derm	Derm7pt-Clinic	Avg.
Source Only	$\times$	$\times$	$78.3\pm 4.8$	$54.1\pm 6.5$	$45.8\pm 7.4$	$50.8\pm 5.1$	$48.6\pm 2.3$	$55.5\pm 5.2$
ADDA [ 45 ]	$\times$	$\times$	$92.1\pm 0.8$	$83.0\pm 0.4$	$72.5\pm 0.7$	$77.2\pm 0.3$	$72.2\pm 0.6$	$79.4\pm 0.5$
DANN [ 12 ]	$\times$	$\times$	$88.7\pm 0.6$	$81.7\pm 0.6$	$79.9\pm 0.7$	$80.2\pm 0.4$	$78.3\pm 0.5$	$81.8\pm 0.6$
MDAN [ 60 ]	$\checkmark$	$\times$	$84.9\pm 0.7$	$75.1\pm 0.6$	$79.8\pm 0.9$	$83.7\pm 0.5$	$77.1\pm 0.7$	$80.1\pm 0.7$
DECISION [ 1 ]	$\checkmark$	$\checkmark$	$76.3\pm 1.5$	$58.7\pm 0.7$	$77.8\pm 1.1$	$79.1\pm 0.6$	$56.2\pm 0.8$	$69.6\pm 0.9$
EAAF [ 38 ]	$\checkmark$	$\checkmark$	$80.8\pm 0.7$	$72.5\pm 0.6$	$78.5\pm 0.5$	$82.4\pm 0.7$	$79.7\pm 0.4$	$78.8\pm 0.6$

Table 2: AUROC comparison of domain adaptation models on skin lesion datasets. Best results are shown in bold.

Figure 4: t-SNE visualization of target-domain features before and after adaptation with CAT-CKD.

Model Configuration	# Parameters
EAAF [ 38 ]	98.2M
DECISION [ 1 ]	96.1M
DANN [ 12 ]	53.0M
ADDA [ 45 ]	26.0M
MDAN [ 60 ]	11.1M
Bi-ATEN [ 30 ]	10.6M

Table 3: Comparison of the trainable parameters between our proposed model and baseline models is presented.

Model	ISIC 2018	ISIC 2020	PAD-UFES-20	Derm7pt-Derm	Derm7pt-Clinic	Avg.
Ours	90.8	90.5	75.8	89.7	84.0	86.1
Ours w/o CAT	87.5	90.1	72.3	78.6	78.0	81.3
Ours w/o CAT, w/ Full backbone finetuning	78.6	90.1	64.5	65.9	75.4	74.9
Ours w/o CKD	86.0	59.5	53.4	73.2	74.2	69.3
Ours w/o KL Divergence	85.7	61.2	53.3	72.1	73.5	69.2

Table 4: Ablation study evaluating the impact of key components in our framework. We report AUROC scores across five target datasets and the average. Best results are shown in bold.

Figure 5: Comparison of attention maps generated by the ViT backbone with and without CAT.

Model	ISIC 2018	ISIC 2020	PAD-UFES-20	Derm7pt-Derm	Derm7pt-Clinic	Avg.
Hard-label CKD	87.4	86.9	71.3	87.7	83.7	83.4
Soft-label CKD (Ours)	90.8	90.5	75.8	89.7	84.0	86.1

Table 5: Comparison between soft-label and hard-label knowledge distillation.

Model	ISIC 2018	ISIC 2020	PAD-UFES-20	Derm7pt-Derm	Derm7pt-Clinic	Avg.
CAT(w/ Houlsby [ 18 ] )-CKD	90.7	73.7	68.0	91.2	84.6	81.6
CAT(w/ Pfeiffer [ 39 ] )-CKD	89.7	77.9	74.1	91.7	81.0	82.8
CAT(w/ Convpass [ 25 ] )-CKD (Ours)	90.8	90.5	75.8	89.7	84.0	86.1

Table 6: Comparison between different adapter models.

Model	ISIC 2018	ISIC 2020	PAD-UFES-20	Derm7pt-Derm	Derm7pt-Clinic	Avg.
Confidence-weighted averaging [ 6 ]	91.2	75.7	70.8	90.7	86.2	82.9
Margin-based weighting [ 42 ]	90.8	75.0	73.1	90.7	85.3	83.0
Entropy-weighted averaging [ 31 ]	89.1	84.4	72.2	89.6	84.1	83.9
CKD (Ours)	90.8	90.5	75.8	89.7	84.0	86.1

Table 7: Comparison between different consensus label generation strategies.

Figure 6: AUROC performance comparison for different temperature values.

Source-Free Domain Adaptation

Source-Free Domain Adaptation transfers models to new data distributions without access to the original training examples.

See all 73 papers

May 29, 2026

CoFiDA-M: Concept-Aware Feature Modulation for Cross-Domain Adaptation with Image-Only Inference

Models for AI-based skin cancer screening suffer a severe performance drop when shifting from expert dermoscopic (source) images to consumer-grade clinical (target) images, hindering real-world deployment. Existing domain adaptation methods often ignore crucial semantic invariants, such as clinical concepts. While new foundation models like MONET can provide this semantic information as dense, probabilistic scores, this metadata is unavailable at test time, creating a deployment paradox for practical image-only screening tools. We address this gap by proposing CoFiDA-M, a privileged information framework that learns from concepts at training time but deploys as an image-only model. Our method trains a teacher network that uses MONET concept probabilities to guide a FiLM modulator, transforming visual features into a semantically

edited" feature space. A lightweight, image-only student is then trained to reproduce this edited representation, not just the teacher's final predictions. This distillation

bakes" the clinical reasoning into the student's weights. On a challenging multi-dataset benchmark, our image-only student significantly outperforms state-of-the-art approaches, especially in melanoma recall. Our work provides a practical and generalizable framework for leveraging noisy, probabilistic metadata as privileged information, demonstrating strong cross-dataset robustness and potential for real-world deployment beyond dermatology. Implementation code is available at: https://github.com/mmu-dermatology-research/CoFiDA.git

Nurjahan Sultana, Moi Hoon Yap, Xinqi Fan +1

Department of Computing and Mathematics, Manchester Metropolitan University Dalton Building, Chester Street, M1 5GD Manchester, UK

Skin Lesion Classification Medical Image Classification Cross-Domain Transfer Learning Learning Using Privileged Information Feature-Based Knowledge Distillation Dermatology

Ask PDF

Feb 23, 2026

Contrastive meta-domain adaptation for robust skin lesion classification across clinical and acquisition conditions

Deep learning models for dermatological image analysis remain sensitive to acquisition variability and domain-specific visual characteristics, leading to performance degradation when deployed in clinical settings. We investigate how visual artifacts and domain shifts affect deep learning-based skin lesion classification. We propose an adaptation strategy, grounded in the idea of visual meta-domains, that transfers visual representations from larger dermoscopic datasets into clinical image domains, thereby improving generalization robustness. Experiments across multiple dermatology datasets show consistent gains in classification performance and reduced gaps between dermoscopic and clinical images. These results emphasize the importance of domain-aware training for deployable systems.

Rodrigo Mota, Kelvin Cunha, Emanoel dos Santos +6

Centro de Informática, Universidade Federal de Pernambuco, Brazil

Skin Lesion Classification Domain Adaptation Dermatology

Ask PDF

Mar 5, 2026

Tell2Adapt: A Unified Framework for Source Free Unsupervised Domain Adaptation via Vision Foundation Model

Source Free Unsupervised Domain Adaptation (SFUDA) is critical for deploying deep learning models across diverse clinical settings. However, existing methods are typically designed for low-gap, specific domain shifts and cannot generalize into a unified, multi-modalities, and multi-target framework, which presents a major barrier to real-world application. To overcome this issue, we introduce Tell2Adapt, a novel SFUDA framework that harnesses the vast, generalizable knowledge of the Vision Foundation Model (VFM). Our approach ensures high-fidelity VFM prompts through Context-Aware Prompts Regularization (CAPR), which robustly translates varied text prompts into canonical instructions. This enables the generation of high-quality pseudo-labels for efficiently adapting the lightweight student model to target domain. To guarantee clinical reliability, the framework incorporates Visual Plausibility Refinement (VPR), which leverages the VFM's anatomical knowledge to re-ground the adapted model's predictions in target image's low-level visual features, effectively removing noise and false positives. We conduct one of the most extensive SFUDA evaluations to date, validating our framework across 10 domain adaptation directions and 22 anatomical targets, including brain, cardiac, polyp, and abdominal targets. Our results demonstrate that Tell2Adapt consistently outperforms existing approaches, achieving SOTA for a unified SFUDA framework in medical image segmentation. Code are avaliable at https://github.com/derekshiii/Tell2Adapt.

Yulong Shi, Shijie Li, Ziyi Li +1

College of Medicine and Biological Information Engineering, Northeastern University · Engineering Research Center of Medical Imaging and Intelligent Analysis, Ministry of Education · Key Laboratory of Medical Image Computing, Ministry of Education, Northeastern University

Source-Free Domain Adaptation Medical Image Segmentation Vision Foundation Model Adaptation

Ask PDF

Abstract

Source-Free Domain Adaptation

Explore similar work

CoFiDA-M: Concept-Aware Feature Modulation for Cross-Domain Adaptation with Image-Only Inference

Contrastive meta-domain adaptation for robust skin lesion classification across clinical and acquisition conditions

Tell2Adapt: A Unified Framework for Source Free Unsupervised Domain Adaptation via Vision Foundation Model