SkinGPT-X: A Self-Evolving Collaborative Multi-Agent System for Transparent and Trustworthy Dermatological Diagnosis

Zhangtianyi Chen; Yuhao Shen; Florensia Widjaja; Yan Xu; Liyuan Sun; Zijian Wang; Hongyi Chen; Wufei Dai

cs.CVMar 27, 2026

SkinGPT-X: A Self-Evolving Collaborative Multi-Agent System for Transparent and Trustworthy Dermatological Diagnosis

Authors: Zhangtianyi Chen, Yuhao Shen, Florensia Widjaja, Yan Xu, Liyuan Sun, Zijian Wang, Hongyi Chen, Wufei Dai, +1 more

Organizations: School of Data Science, The Chinese University of Hong Kong, Shenzhen · Department of Dermatology, Tianjin Institute of Integrative Dermatology, Tianjin Academy of Traditional Chinese Medicine Affiliated Hospital, Tianjin 300120, China · Department of Dermatology, Beijing AnZhen Hospital, Capital Medical University, Beijing 100029, China..

Abstract

While recent advancements in Large Language Models have significantly advanced dermatological diagnosis, monolithic LLMs frequently struggle with fine-grained, large-scale multi-class diagnostic tasks and rare skin disease diagnosis owing to training data sparsity, while also lacking the interpretability and traceability essential for clinical reasoning. Although multi-agent systems can offer more transparent and explainable diagnostics, existing frameworks are primarily concentrated on Visual Question Answering and conversational tasks, and their heavy reliance on static knowledge bases restricts adaptability in complex real-world clinical settings. Here, we present SkinGPT-X, a multimodal collaborative multi-agent system for dermatological diagnosis integrated with a self-evolving dermatological memory mechanism. By simulating the diagnostic workflow of dermatologists and enabling continuous memory evolution, SkinGPT-X delivers transparent and trustworthy diagnostics for the management of complex and rare dermatological cases. To validate the robustness of SkinGPT-X, we design a three-tier comparative experiment. First, we benchmark SkinGPT-X against four state-of-the-art LLMs across four public datasets, demonstrating its state-of-the-art performance with a +9.6% accuracy improvement on DDI31 and +13% weighted F1 gain on Dermnet over the state-of-the-art model. Second, we construct a large-scale multi-class dataset covering 498 distinct dermatological categories to evaluate its fine-grained classification capabilities. Finally, we curate the rare skin disease dataset, the first benchmark to address the scarcity of clinical rare skin diseases which contains 564 clinical samples with eight rare dermatological diseases. On this dataset, SkinGPT-X achieves a +9.8% accuracy improvement, a +7.1% weighted F1 improvement, a +10% Cohen's Kappa improvement.

Figures & tables

Fig. 1: a, The process of clinicians formulating a comprehensive Diagnosis Report : Clinicians formulate a Diagnosis Report by integrating three core pillars of medical intelligence: Personal Clinical Experience, Standard Medical Literature, and the Recall of Similar Historical Cases. This holistic synthesis, further augmented by Online Verification of the latest treatments, enables a validated Treatment Plan. b, The Architecture of the SkinGPT-X System : Upon receiving a disease image and a query, the Vision Agent extracts fine-grained Visual Findings, while the Diagnosis Diagnosis Agent performs an initial Pre-diagnosis Hypotheses. This process is anchored by the RAG, which retrieves Local Medical Knowledge to ensure evidence-based reasoning. To incorporate empirical experience, the system utilizes the self-evolving agent memory, where the Top-5 Similar Cases and Diagnostic Guidelines are retrieved from the Historical Case Graph Database. The visual features, retrieved guidelines, and historical precedents are synthesized by the Case-Review Agent, which conducts a rigorous cross-reference to produce a validated Report. The Summarize Agent ensures self-evolving agent memory by integrating new confirmed cases into the Dynamic Repository, iteratively distill the diagnostic guidelines without retraining.

Fig. 2: a, Performance comparison of SkinGPT-X versus four state-of-the-art models (MedGemma, Hulu-Med, Qwen3-VL, and PanDerm) across four benchmark skin disease datasets. $c$ indicates the class number; $n$ represents the data size. Metrics include ACC, Weighted F1, MCC, and Cohen’s Kappa. The error bars represent 95% CIs, and $P$ values were calculated using a two-sided $t$ -test to indicate statistical significance between SkinGPT-X and the second-best model. b, Case study of diagnostic case reviewing and visual findings generation. The panels display the text-based outputs from MedGemma, Hulu-Med, and SkinGPT-X for representative clinical cases. The results illustrate that SkinGPT-X provides more reliable and transparent diagnostics by reviewing current clinical findings with its self-evolving agent memory.

Fig. 3: a, Data distribution and hierarchical composition of the dataset series. This figure illustrates the incremental expansion of skin disease categories within the Dermnet dataset, scaling from Dermnet to the comprehensive Dermnet498. Each colored block represents a unique disease class. b, Comparative performance metrics on the Dermnet498 dataset. The bar chart presents ACC, MCC, Weighted F1, and Kappa scores for MedGemma, Hulu-Med, Qwen3-VL, PanDerm, and SkinGPT-X. Numerical values and error bars represent the mean and confidence intervals, with statistical significance between the top two models indicated by

P

-values. The radar chart displays the composition of Dermnet498. The legend specifies the top categories and their respective sample counts. c, Performance trends across varying category scales. The line graphs plot the changes in ACC, MCC, Weighted F1, and Kappa for the PanDerm and SkinGPT-X models as the number of disease categories increases.

Fig. 4: a, Construction pipeline of the Rare Skin Disease Dataset. The dataset integrates diverse data sources, including medical libraries, clinical centers, and specialized guidelines to ensure high-quality labels for rare dermatological conditions. The RSDD dataset comprises 8 rare conditions, including Cutaneous Neuroendocrine Carcinoma ( $n=115$ ), Generalized Pustular Psoriasis ( $n=110$ ), and Behcet’s Disease ( $n=81$ ), among others. b, Multi-dimensional performance comparison on the RSDD ( $n=564$ ). The radar chart illustrates the diagnostic efficacy of SkinGPT-X versus MedGemma, Hulu-Med, Qwen3-VL, and PanDerm across five metrics: ACC, MCC, Macro F1, Weighted F1, and Kappa. c, Representative case studies of rare disease diagnosis and differential reasoning. EvoDerma-Mem enhances the capability of the system differentiating the correct diagnosis form rare candidate skin diseases

Fig. 5: a, Performance comparison between SkinGPT-X and the baseline framework (without memory) on Fitzpatrick-17k and Dermnet498 datasets. Data are presented as mean metrics with 95% confidence intervals; statistical significance was determined via two-sided t-tests ( $p<0.001$ ). b, Spatiotemporal evolution trajectory of diagnostic guidelines. The bubble color intensity represent degree of knowledge refinement, where deeper shades indicate more substantial updates to the diagnostic guidelines. c, The standardized Physician Evaluation Form used for blinded clinical review, focusing on Rigorousness of Medical Logic, Validity of Diagnostic guidelines Completeness and Rationality of Clinical Manifestation Refinement. d, Results of physician validation across three critical dimensions.

Explore similar work

May 14, 2026

DermAgent: A Self-Reflective Agentic System for Dermatological Image Analysis with Multi-Tool Reasoning and Traceable Decision-Making

Dermatological diagnosis requires integrating fine-grained visual perception with expert clinical knowledge. Although Multimodal Large Language Models (MLLMs) facilitate interactive medical image analysis, their application in dermatology is hindered by insufficient domain-specific grounding and hallucinations. To address these issues, we propose DermAgent, a collaborative multi-tool agent that orchestrates seven specialized vision and language modules within a Plan-Execute-Reflect framework. DermAgent delivers stepwise, traceable diagnostic reasoning through three core components. First, it employs complementary visual perception tools for comprehensive morphological description, dermoscopic concept annotation, and disease diagnosis. Second, to overcome the lack of domain prior, a dual-modality retrieval module anchors every prediction in external evidence by cross-referencing 413,210 diagnosed image cases and 3,199 clinical guideline chunks. To further mitigate hallucinations, a deterministic critic module conducts strict post-hoc auditing via confidence, coverage, and conflict gates, automatically detecting inter-source disagreements to trigger targeted self-correction. Extensive experiments on five dermatology benchmarks demonstrate that DermAgent consistently outperforms state-of-the-art MLLMs and medical agent baselines across zero-shot fine-grained disease diagnosis, concept annotation, and clinical captioning tasks, exceeding GPT-4o by 17.6% in skin disease diagnostic accuracy and 3.15% in captioning ROUGE-L. Our code is available at https://github.com/YizeezLiu/DermAgent.

Yize Liu, Siyuan Yan, Ming Hu +5

AIM for Health Lab, Faculty of Information Technology, Monash University, Melbourne, Australia · Faculty of Information Technology, Monash University, Melbourne, Australia · University College London, Institute of Ophthalmology, London, United Kingdom

Medical Image Analysis LLM Agent Orchestration Tool-Using Vision-Language Agents Medical VLMs Explainable Medical Image Analysis

Ask PDF

Mar 19, 2026

Mind the Rarities: Can Rare Skin Diseases Be Reliably Diagnosed via Diagnostic Reasoning?

Large vision-language models (LVLMs) demonstrate strong performance in dermatology; however, evaluating diagnostic reasoning for rare conditions remains largely unexplored. Existing benchmarks focus on common diseases and assess only final accuracy, overlooking the clinical reasoning process, which is critical for complex cases. We address this gap by constructing DermCase, a long-context benchmark derived from peer-reviewed case reports. Our dataset contains 26,030 multi-modal image-text pairs and 6,354 clinically challenging cases, each annotated with comprehensive clinical information and step-by-step reasoning chains. To enable reliable evaluation, we establish DermLIP-based similarity metrics that achieve stronger alignment with dermatologists for assessing differential diagnosis quality. Benchmarking 22 leading LVLMs exposes significant deficiencies across diagnosis accuracy, differential diagnosis, and clinical reasoning. Fine-tuning experiments demonstrate that instruction tuning substantially improves performance while Direct Preference Optimization (DPO) yields minimal gains. Systematic error analysis further reveals critical limitations in current models' reasoning capabilities.

Yang Liu, Jiyao Yang, Hongjin Zhao +10

Carnegie Mellon University · University of Alabama at Birmingham · Australian National University +3

Rare Disease Diagnosis Medical VLMs Clinical Reasoning Medical Image Benchmarks VLM Reasoning

Ask PDF

Nov 18, 2025

Skin-R1: Clinical Knowledge-Guided Dermatological Diagnosis Using Vision-Language Models

Vision--language models (VLMs) have recently shown promise for assisting clinical reasoning in dermatological diagnosis. However, their trustworthiness and clinical utility remain limited by three key challenges: heterogeneous datasets with inconsistent diagnostic labels and concept annotations, the lack of grounded diagnostic rationales for reliable reasoning supervision, and limited scalability when transferring knowledge from small, densely annotated datasets to large collections with sparse labels. To address these challenges, we propose Skin-R1, a dermatology-oriented VLM that integrates textbook-grounded clinical reasoning supervision with reinforcement learning (RL) to improve the accuracy and robustness of diagnostic prediction. First, we construct a textbook-based reasoning generator that synthesizes hierarchy-aware and differential-diagnosis (DDx) diagnostic trajectories derived from authoritative dermatology knowledge. Second, these trajectories are used for supervised fine-tuning (SFT), establishing a clinically grounded reasoning foundation for the model. Finally, we introduce an RL training framework that incorporates the hierarchical structure of dermatological diseases into the reward design, enabling the model to generalize grounded diagnostic reasoning to large-scale datasets with sparse annotations. Extensive experiments across multiple dermatology benchmarks demonstrate that Skin-R1 consistently improves diagnostic accuracy and robustness compared to state-of-the-art Med-VLM baselines. Ablation studies further highlight the critical role of grounded reasoning supervision introduced during the SFT stage.

Zehao Liu, Weijieying Ren, Jipeng Zhang +4

Pennsylvania State University, University Park, PA 16802, USA · Stanford University, Stanford, CA 94305, USA · Hong Kong University of Science and Technology, Hong Kong, China

VLM Robustness Medical Diagnosis Medical VLMs VLM Reasoning RL for VLMs

Ask PDF