Multi-Dataset Diagnostic Utility of Clinical Visual Concepts in AI Systems for Dermatology

Linda Wermelinger; Simone Lionetti; Fabian Gröger; Nipun Ranasekara; Philippe Gottfrois; Ludovic Amruthalingam; Labelling Consortium; Marc Pouly

cs.CVOct 6, 2026

Multi-Dataset Diagnostic Utility of Clinical Visual Concepts in AI Systems for Dermatology

Authors: Linda Wermelinger, Simone Lionetti, Fabian Gröger, Nipun Ranasekara, Philippe Gottfrois, Ludovic Amruthalingam, Labelling Consortium, Marc Pouly, +1 more

Organizations: University of Basel · Lucerne University of Applied Sciences and Arts · University Hospital of Basel

Abstract

The clinical integration of AI systems in digital dermatology relies heavily on human trust. Clinically interpretable visual concepts can act as intermediate representations enhancing trust and reliability. However, research in this domain is currently limited by scattered, heterogeneous dataset annotations. In this work, we introduce SkinLex, a harmonized dataset of 48 clinical morphological attributes across four public datasets (SkinCon, DermaCon-IN, MM-Skin, and PASSION) for a total of 20,411 records. Supervised nine-partition classification of skin conditions shows that limiting features to specific visual groups, like shapes or colors alone, reduces diagnostic accuracy. Bootstrapped backward elimination reveals that the set of 48 visual concepts has some degree of redundancy for algorithmic nine-partition diagnosis on the examined dataset. This demonstrates that coarse diagnosis on the selected dataset requires a relatively small but varied combination of clinical concepts, and motivates further research to improve concept taxonomy. Results can be translated into clinical benefits by reducing inputs for concept-based models, improving efficiency for annotation and modeling, and further enhancing interpretability. Code and prompt templates are available at https://github.com/Digital-Dermatology/SkinLex.

Figures & tables

Figure 1: Overview of the paper: the harmonized SkinLex dataset and visual concept utility analysis.

Source Dataset	Image Modality	Original	Final	Alignment
SkinCon	Clinical	0 3,886	3,642	Direct match.
DermaCon-IN	Clinical	0 5,450	5,405	Direct match.
MM-Skin	Clinical, Dermatoscopy, Pathology	11,015	7,500	Concept and disease extraction.
PASSION	Clinical	0 5,127	3,864	Concept extraction.
SkinLex	Mixed	25,478	20,411

Table 1: Summary of original and final sample counts across unified sources.

Standardized Disease Category	Sample Count	Percentage (%)
Inflammatory	15,091	73.94
Malignant Epidermal	964	4.72
Genodermatoses	867	4.25
Benign Dermal	865	4.24
Benign Epidermal	825	4.04
Benign Melanocyte	641	3.14

Table 2: Distribution of the aligned disease categories across the unified dataset.

		LightGBM		Logistic Regression
Feature Set	Features	ROC-AUC	AP	ROC-AUC	AP
Full Model	48	$0.829\,\mathopen{}\mathclose{{\left[{{}_{0.812}^{0.845}}\right]}}$	$0.309\,\mathopen{}\mathclose{{\left[{{}_{0.288}^{0.330}}\right]}}$	$0.801\,\mathopen{}\mathclose{{\left[{{}_{0.786}^{0.819}}\right]}}$	$0.274\,\mathopen{}\mathclose{{\left[{{}_{0.255}^{0.299}}\right]}}$
Shapes	16	$0.711\,\mathopen{}\mathclose{{\left[{{}_{0.695}^{0.727}}\right]}}$	$0.174\,\mathopen{}\mathclose{{\left[{{}_{0.163}^{0.183}}\right]}}$	$0.672\,\mathopen{}\mathclose{{\left[{{}_{0.658}^{0.689}}\right]}}$	$0.170\,\mathopen{}\mathclose{{\left[{{}_{0.158}^{0.181}}\right]}}$
Colors	10	$0.690\,\mathopen{}\mathclose{{\left[{{}_{0.671}^{0.713}}\right]}}$	$0.173\,\mathopen{}\mathclose{{\left[{{}_{0.164}^{0.185}}\right]}}$	$0.683\,\mathopen{}\mathclose{{\left[{{}_{0.665}^{0.707}}\right]}}$	$0.177\,\mathopen{}\mathclose{{\left[{{}_{0.165}^{0.191}}\right]}}$
Textures	15	$0.661\,\mathopen{}\mathclose{{\left[{{}_{0.644}^{0.675}}\right]}}$	$0.153\,\mathopen{}\mathclose{{\left[{{}_{0.145}^{0.163}}\right]}}$	$0.659\,\mathopen{}\mathclose{{\left[{{}_{0.643}^{0.675}}\right]}}$	$0.155\,\mathopen{}\mathclose{{\left[{{}_{0.148}^{0.163}}\right]}}$
Others	7	$0.530\,\mathopen{}\mathclose{{\left[{{}_{0.524}^{0.538}}\right]}}$	$0.124\,\mathopen{}\mathclose{{\left[{{}_{0.119}^{0.129}}\right]}}$	$0.532\,\mathopen{}\mathclose{{\left[{{}_{0.526}^{0.541}}\right]}}$	$0.124\,\mathopen{}\mathclose{{\left[{{}_{0.119}^{0.129}}\right]}}$

Table 3: Model performance across full and restricted concept groups. Metrics report median and [95% CI ] estimated with 100 validation bootstraps.

Figure 2: Leave-one-out sensitivity for LightGBM on the validation split. The bars illustrate the mean absolute decrease in macro AP (left) and ROC-AUC (right) when a concept is omitted, with higher values indicating concepts critical for performance. Error bars denote the standard deviation across 100 validation bootstraps.

Figure 3: Median and 95% CI of LightGBM macro AP and ROC-AUC across the number of concepts used, using 100 bootstraps of training and validation sets. The dashed line indicates the highest number of used concepts where a paired one-sided Wilcoxon signed-rank test indicates lower performance than the full set.

Explore similar work

Skin-R1: Clinical Knowledge-Guided Dermatological Diagnosis Using Vision-Language Models
Nov 18, 2025Zehao Liu, Weijieying Ren, Jipeng Zhang +4Skin Lesion Classification Melanoma
Pennsylvania State University, University Park, PA 16802, USA · Stanford University, Stanford, CA 94305, USA · Hong Kong University of Science and Technology, Hong Kong, China
Open-Linguistic Concept Unified Learning for Cross-Site Interpretable Dermatology Image Diagnosis
Aug 4, 2026Chengyu Wu, Junpeng Tan, Wanxiang Luo +3Skin Lesion Classification Medical Vision-Language Models
Zhejiang University Hangzhou, China · Westlake University Hangzhou, China · Harbin Institute of Technology Shenzhen, China +1
DermAgent: A Self-Reflective Agentic System for Dermatological Image Analysis with Multi-Tool Reasoning and Traceable Decision-Making
May 14, 2026Yize Liu, Siyuan Yan, Ming Hu +5Skin Lesion Classification Melanoma
AIM for Health Lab, Faculty of Information Technology, Monash University, Melbourne, Australia · Faculty of Information Technology, Monash University, Melbourne, Australia · University College London, Institute of Ophthalmology, London, United Kingdom

Nov 18, 2025, cs.CVJ/K move · Enter open · S save

Skin-R1: Clinical Knowledge-Guided Dermatological Diagnosis Using Vision-Language Models

Zehao Liu, Weijieying Ren, Jipeng Zhang, Tianxiang Zhao+3

Pennsylvania State University, University Park, PA 16802, USA · Stanford University, Stanford, CA 94305, USA · Hong Kong University of Science and Technology, Hong Kong, China

Skin Lesion Classification Melanoma Medical Vision-Language Models Clinical

Open paper AskPDF arXiv

Vision--language models (VLMs) have recently shown promise for assisting clinical reasoning in dermatological diagnosis. However, their trustworthiness and clinical utility remain limited by three key challenges: heterogeneous datasets with inconsistent diagnostic labels and concept annotations, the lack of grounded diagnostic rationales for reliable reasoning supervision, and limited scalability when transferring knowledge from small, densely annotated datasets to large collections with sparse labels. To address these challenges, we propose Skin-R1, a dermatology-oriented VLM that integrates textbook-grounded clinical reasoning supervision with reinforcement learning (RL) to improve the accuracy and robustness of diagnostic prediction. First, we construct a textbook-based reasoning generator that synthesizes hierarchy-aware and differential-diagnosis (DDx) diagnostic trajectories derived from authoritative dermatology knowledge. Second, these trajectories are used for supervised fine-tuning (SFT), establishing a clinically grounded reasoning foundation for the model. Finally, we introduce an RL training framework that incorporates the hierarchical structure of dermatological diseases into the reward design, enabling the model to generalize grounded diagnostic reasoning to large-scale datasets with sparse annotations. Extensive experiments across multiple dermatology benchmarks demonstrate that Skin-R1 consistently improves diagnostic accuracy and robustness compared to state-of-the-art Med-VLM baselines. Ablation studies further highlight the critical role of grounded reasoning supervision introduced during the SFT stage.

Abstract

Explore similar work

Skin-R1: Clinical Knowledge-Guided Dermatological Diagnosis Using Vision-Language Models

Open-Linguistic Concept Unified Learning for Cross-Site Interpretable Dermatology Image Diagnosis

DermAgent: A Self-Reflective Agentic System for Dermatological Image Analysis with Multi-Tool Reasoning and Traceable Decision-Making