SkinCLIP-VL: Consistency-Aware Vision-Language Learning for Multimodal Skin Cancer Diagnosis

Zhixiang Lu; Shijie Xu; Kaicheng Yan; Xuyue Cai; Chong Zhang; Yulong Li; Angelos Stefanidis; Anh Nguyen

cs.CVMar 22, 2026

SkinCLIP-VL: Consistency-Aware Vision-Language Learning for Multimodal Skin Cancer Diagnosis

Authors: Zhixiang Lu, Shijie Xu, Kaicheng Yan, Xuyue Cai, Chong Zhang, Yulong Li, Angelos Stefanidis, Anh Nguyen, +1 more

Organizations: Xi’an Jiaotong-Liverpool University, China · University of Liverpool, United Kingdom

Abstract

The deployment of vision-language models (VLMs) in dermatology is hindered by the trilemma of high computational costs, extreme data scarcity, and the black-box nature of deep learning. To address these challenges, we present SkinCLIP-VL, a resource-efficient framework that adapts foundation models for trustworthy skin cancer diagnosis. Adopting a frozen perception, adaptive reasoning paradigm, we integrate a frozen CLIP encoder with a lightweight, quantized Qwen2.5-VL via low-rank adaptation (LoRA). To strictly align visual regions with clinical semantics under long-tailed distributions, we propose the Consistency-aware Focal Alignment (CFA) Loss. This objective synergizes focal re-weighting, semantic alignment, and calibration. On ISIC and Derm7pt benchmarks, SkinCLIP-VL surpasses 13B-parameter baselines by 4.3-6.2% in accuracy with 43% fewer parameters. Crucially, blinded expert evaluation and out-of-distribution testing confirm that our visually grounded rationales significantly enhance clinical trust compared to traditional saliency maps.

Figures & tables

Fig. 1: Consistency-Aware Focal Alignment Architecture.

Fig. 2: The overall framework of SkinCLIP-VL. The architecture consists of three key stages: (1) Meta-Data Enhancement : We leverage GPT-4o to expand tabular meta-data into comprehensive clinical descriptions, providing semantic guidance for the visual branch. (2) Parameter-Efficient Encoding : Instead of full fine-tuning, we employ a frozen CLIP visual encoder and a LoRA-adapted Qwen2.5-VL generative decoder to extract and align visual and textual features. (3) Consistency-Aware Focal Alignment (CFA) : To capture fine-grained correlations, the Focal Pooling Layer aggregates the $N\times M$ interaction map into compact focal vectors. These vectors are fused via a Transformer Encoder under the joint supervision of focal and alignment losses ( $\mathcal{L}_{focal},\mathcal{L}_{align}$ ).

Model	Params	ISIC 2019			Derm7pt			ISIC 2024 (OOD)
Model	Params	B-ACC $\uparrow$	AUROC $\uparrow$	ECE $\downarrow$	B-ACC $\uparrow$	AUROC $\uparrow$	ECE $\downarrow$	B-ACC $\uparrow$	AUROC $\uparrow$	ECE $\downarrow$
Multimodal Models
EfficientNet-B4 [ 24 ]	19M	81.1%	0.901	0.084	78.2%	0.899	0.093	79.0%	0.890	0.098
ResNet-34 [ 10 ]	22M	79.5%	0.890	0.095	77.0%	0.885	0.100	78.0%	0.880	0.105
ConvNeXt-Base [ 18 ]	88M	82.1%	0.908	0.080	78.8%	0.905	0.090	80.0%	0.900	0.090
CaFormer-B36 [ 28 ]	152M	81.8%	0.905	0.082	78.5%	0.902	0.091	79.5%	0.895	0.092

TABLE I: Performance comparison on ISIC 2019, Derm7pt, and ISIC 2024.

Fig. 3: Hyperparameter sensitivity analysis.

Model Configuration	B-ACC $\uparrow$	AUROC $\uparrow$	ECE $\downarrow$
Baseline (CE Loss)	71.5%	0.820	0.155
+ $\mathcal{L}_{focal}$	79.2%	0.910	0.095
+ $\mathcal{L}_{focal}$ + $\mathcal{L}_{align}$	82.5%	0.945	0.088
+ $\mathcal{L}_{focal}$ + $\mathcal{L}_{cal}$	80.1%	0.915	0.045
+ $\mathcal{L}_{CFA}$ (Full)	85.0%	0.972	0.033

TABLE II: Ablation of CFA components on ISIC 2024 (OOD).

Model	Train-100%	Train-12%	$\Delta$ Drop
EfficientNet-B4	79.0%	58.3%	-20.7%
SkinGPT-4	81.0%	64.1%	-16.9%
SkinCLIP-VL	85.0%	82.4%	-2.6%

TABLE III: Data Efficiency Analysis on ISIC 2024 .

Metric	SkinGPT-4	Ours
1. I trust this output.	4.5	5.2
2. The rationale is clear .	5.4	5.5
3. Justifies diagnosis.	4.2	5.3

TABLE IV: Blinded expert evaluation (N=20, 7-point Likert).

Fig. 4: Case study of dynamic visual grounding.

Explore similar work

Skin-R1: Clinical Knowledge-Guided Dermatological Diagnosis Using Vision-Language Models
Nov 18, 2025Zehao Liu, Weijieying Ren, Jipeng Zhang +4VLM Robustness Medical Diagnosis
Pennsylvania State University, University Park, PA 16802, USA · Stanford University, Stanford, CA 94305, USA · Hong Kong University of Science and Technology, Hong Kong, China
How Medical VLMs Underutilize Their Vision Encoders: A Dermatology Perspective
Sep 29, 2026Janet Wang, Yunbei Zhang, Xiao Wang +1Medical Diagnosis VLM Interpretability
Tulane University · Oak Ridge National Laboratory
CNNs, Transformers, Hybrid, and Vision Language Models for Skin Cancer Detection
May 25, 2026Durjoy Dey, Yuhong Yan, Hassan HajjdiabHybrid CNN-Transformer Architectures Vision Transformer
Department of Computer Science and Software Engineering, Concordia University, Montreal, Canada · Ebovir Biotechnologie Inc., Montreal, Canada

Nov 18, 2025J/K move · Enter open · S save

Skin-R1: Clinical Knowledge-Guided Dermatological Diagnosis Using Vision-Language Models

Zehao Liu, Weijieying Ren, Jipeng Zhang, Tianxiang Zhao+3

Pennsylvania State University, University Park, PA 16802, USA · Stanford University, Stanford, CA 94305, USA · Hong Kong University of Science and Technology, Hong Kong, China

VLM Robustness Medical Diagnosis Medical VLMs VLM Reasoning RL for VLMs

Open paper AskPDF arXiv

Vision--language models (VLMs) have recently shown promise for assisting clinical reasoning in dermatological diagnosis. However, their trustworthiness and clinical utility remain limited by three key challenges: heterogeneous datasets with inconsistent diagnostic labels and concept annotations, the lack of grounded diagnostic rationales for reliable reasoning supervision, and limited scalability when transferring knowledge from small, densely annotated datasets to large collections with sparse labels. To address these challenges, we propose Skin-R1, a dermatology-oriented VLM that integrates textbook-grounded clinical reasoning supervision with reinforcement learning (RL) to improve the accuracy and robustness of diagnostic prediction. First, we construct a textbook-based reasoning generator that synthesizes hierarchy-aware and differential-diagnosis (DDx) diagnostic trajectories derived from authoritative dermatology knowledge. Second, these trajectories are used for supervised fine-tuning (SFT), establishing a clinically grounded reasoning foundation for the model. Finally, we introduce an RL training framework that incorporates the hierarchical structure of dermatological diseases into the reward design, enabling the model to generalize grounded diagnostic reasoning to large-scale datasets with sparse annotations. Extensive experiments across multiple dermatology benchmarks demonstrate that Skin-R1 consistently improves diagnostic accuracy and robustness compared to state-of-the-art Med-VLM baselines. Ablation studies further highlight the critical role of grounded reasoning supervision introduced during the SFT stage.

Abstract

Explore similar work

Skin-R1: Clinical Knowledge-Guided Dermatological Diagnosis Using Vision-Language Models

How Medical VLMs Underutilize Their Vision Encoders: A Dermatology Perspective

CNNs, Transformers, Hybrid, and Vision Language Models for Skin Cancer Detection