SkinGPT-X: A Self-Evolving Collaborative Multi-Agent System for Transparent and Trustworthy Dermatological Diagnosis

Zhangtianyi Chen; Yuhao Shen; Florensia Widjaja; Yan Xu; Liyuan Sun; Zijian Wang; Hongyi Chen; Wufei Dai

cs.CVMar 27, 2026

SkinGPT-X: A Self-Evolving Collaborative Multi-Agent System for Transparent and Trustworthy Dermatological Diagnosis

Authors: Zhangtianyi Chen, Yuhao Shen, Florensia Widjaja, Yan Xu, Liyuan Sun, Zijian Wang, Hongyi Chen, Wufei Dai, +1 more

Organizations: School of Data Science, The Chinese University of Hong Kong, Shenzhen · Department of Dermatology, Tianjin Institute of Integrative Dermatology, Tianjin Academy of Traditional Chinese Medicine Affiliated Hospital, Tianjin 300120, China · Department of Dermatology, Beijing AnZhen Hospital, Capital Medical University, Beijing 100029, China..

Abstract

While recent advancements in Large Language Models have significantly advanced dermatological diagnosis, monolithic LLMs frequently struggle with fine-grained, large-scale multi-class diagnostic tasks and rare skin disease diagnosis owing to training data sparsity, while also lacking the interpretability and traceability essential for clinical reasoning. Although multi-agent systems can offer more transparent and explainable diagnostics, existing frameworks are primarily concentrated on Visual Question Answering and conversational tasks, and their heavy reliance on static knowledge bases restricts adaptability in complex real-world clinical settings. Here, we present SkinGPT-X, a multimodal collaborative multi-agent system for dermatological diagnosis integrated with a self-evolving dermatological memory mechanism. By simulating the diagnostic workflow of dermatologists and enabling continuous memory evolution, SkinGPT-X delivers transparent and trustworthy diagnostics for the management of complex and rare dermatological cases. To validate the robustness of SkinGPT-X, we design a three-tier comparative experiment. First, we benchmark SkinGPT-X against four state-of-the-art LLMs across four public datasets, demonstrating its state-of-the-art performance with a +9.6% accuracy improvement on DDI31 and +13% weighted F1 gain on Dermnet over the state-of-the-art model. Second, we construct a large-scale multi-class dataset covering 498 distinct dermatological categories to evaluate its fine-grained classification capabilities. Finally, we curate the rare skin disease dataset, the first benchmark to address the scarcity of clinical rare skin diseases which contains 564 clinical samples with eight rare dermatological diseases. On this dataset, SkinGPT-X achieves a +9.8% accuracy improvement, a +7.1% weighted F1 improvement, a +10% Cohen's Kappa improvement.

Figures & tables

Fig. 1: a, The process of clinicians formulating a comprehensive Diagnosis Report : Clinicians formulate a Diagnosis Report by integrating three core pillars of medical intelligence: Personal Clinical Experience, Standard Medical Literature, and the Recall of Similar Historical Cases. This holistic synthesis, further augmented by Online Verification of the latest treatments, enables a validated Treatment Plan. b, The Architecture of the SkinGPT-X System : Upon receiving a disease image and a query, the Vision Agent extracts fine-grained Visual Findings, while the Diagnosis Diagnosis Agent performs an initial Pre-diagnosis Hypotheses. This process is anchored by the RAG, which retrieves Local Medical Knowledge to ensure evidence-based reasoning. To incorporate empirical experience, the system utilizes the self-evolving agent memory, where the Top-5 Similar Cases and Diagnostic Guidelines are retrieved from the Historical Case Graph Database. The visual features, retrieved guidelines, and historical precedents are synthesized by the Case-Review Agent, which conducts a rigorous cross-reference to produce a validated Report. The Summarize Agent ensures self-evolving agent memory by integrating new confirmed cases into the Dynamic Repository, iteratively distill the diagnostic guidelines without retraining.

Fig. 2: a, Performance comparison of SkinGPT-X versus four state-of-the-art models (MedGemma, Hulu-Med, Qwen3-VL, and PanDerm) across four benchmark skin disease datasets. $c$ indicates the class number; $n$ represents the data size. Metrics include ACC, Weighted F1, MCC, and Cohen’s Kappa. The error bars represent 95% CIs, and $P$ values were calculated using a two-sided $t$ -test to indicate statistical significance between SkinGPT-X and the second-best model. b, Case study of diagnostic case reviewing and visual findings generation. The panels display the text-based outputs from MedGemma, Hulu-Med, and SkinGPT-X for representative clinical cases. The results illustrate that SkinGPT-X provides more reliable and transparent diagnostics by reviewing current clinical findings with its self-evolving agent memory.

Fig. 3: a, Data distribution and hierarchical composition of the dataset series. This figure illustrates the incremental expansion of skin disease categories within the Dermnet dataset, scaling from Dermnet to the comprehensive Dermnet498. Each colored block represents a unique disease class. b, Comparative performance metrics on the Dermnet498 dataset. The bar chart presents ACC, MCC, Weighted F1, and Kappa scores for MedGemma, Hulu-Med, Qwen3-VL, PanDerm, and SkinGPT-X. Numerical values and error bars represent the mean and confidence intervals, with statistical significance between the top two models indicated by

P

-values. The radar chart displays the composition of Dermnet498. The legend specifies the top categories and their respective sample counts. c, Performance trends across varying category scales. The line graphs plot the changes in ACC, MCC, Weighted F1, and Kappa for the PanDerm and SkinGPT-X models as the number of disease categories increases.

Fig. 4: a, Construction pipeline of the Rare Skin Disease Dataset. The dataset integrates diverse data sources, including medical libraries, clinical centers, and specialized guidelines to ensure high-quality labels for rare dermatological conditions. The RSDD dataset comprises 8 rare conditions, including Cutaneous Neuroendocrine Carcinoma ( $n=115$ ), Generalized Pustular Psoriasis ( $n=110$ ), and Behcet’s Disease ( $n=81$ ), among others. b, Multi-dimensional performance comparison on the RSDD ( $n=564$ ). The radar chart illustrates the diagnostic efficacy of SkinGPT-X versus MedGemma, Hulu-Med, Qwen3-VL, and PanDerm across five metrics: ACC, MCC, Macro F1, Weighted F1, and Kappa. c, Representative case studies of rare disease diagnosis and differential reasoning. EvoDerma-Mem enhances the capability of the system differentiating the correct diagnosis form rare candidate skin diseases

Fig. 5: a, Performance comparison between SkinGPT-X and the baseline framework (without memory) on Fitzpatrick-17k and Dermnet498 datasets. Data are presented as mean metrics with 95% confidence intervals; statistical significance was determined via two-sided t-tests ( $p<0.001$ ). b, Spatiotemporal evolution trajectory of diagnostic guidelines. The bubble color intensity represent degree of knowledge refinement, where deeper shades indicate more substantial updates to the diagnostic guidelines. c, The standardized Physician Evaluation Form used for blinded clinical review, focusing on Rigorousness of Medical Logic, Validity of Diagnostic guidelines Completeness and Rationality of Clinical Manifestation Refinement. d, Results of physician validation across three critical dimensions.

Explore similar work

DermAgent: A Self-Reflective Agentic System for Dermatological Image Analysis with Multi-Tool Reasoning and Traceable Decision-Making
May 14, 2026Yize Liu, Siyuan Yan, Ming Hu +5Medical Image Analysis LLM Agent Orchestration
AIM for Health Lab, Faculty of Information Technology, Monash University, Melbourne, Australia · Faculty of Information Technology, Monash University, Melbourne, Australia · University College London, Institute of Ophthalmology, London, United Kingdom
Mind the Rarities: Can Rare Skin Diseases Be Reliably Diagnosed via Diagnostic Reasoning?
Mar 19, 2026Yang Liu, Jiyao Yang, Hongjin Zhao +10Rare Disease Diagnosis Medical VLMs
Carnegie Mellon University · University of Alabama at Birmingham · Australian National University +3
Skin-R1: Clinical Knowledge-Guided Dermatological Diagnosis Using Vision-Language Models
Nov 18, 2025Zehao Liu, Weijieying Ren, Jipeng Zhang +4VLM Robustness Medical Diagnosis
Pennsylvania State University, University Park, PA 16802, USA · Stanford University, Stanford, CA 94305, USA · Hong Kong University of Science and Technology, Hong Kong, China

May 14, 2026J/K move · Enter open · S save

DermAgent: A Self-Reflective Agentic System for Dermatological Image Analysis with Multi-Tool Reasoning and Traceable Decision-Making

Yize Liu, Siyuan Yan, Ming Hu, Lie Ju+4

AIM for Health Lab, Faculty of Information Technology, Monash University, Melbourne, Australia · Faculty of Information Technology, Monash University, Melbourne, Australia · University College London, Institute of Ophthalmology, London, United Kingdom

Medical Image Analysis LLM Agent Orchestration Tool-Using Vision-Language Agents Medical VLMs Explainable Medical Image Analysis

Open paper AskPDF arXiv

Dermatological diagnosis requires integrating fine-grained visual perception with expert clinical knowledge. Although Multimodal Large Language Models (MLLMs) facilitate interactive medical image analysis, their application in dermatology is hindered by insufficient domain-specific grounding and hallucinations. To address these issues, we propose DermAgent, a collaborative multi-tool agent that orchestrates seven specialized vision and language modules within a Plan-Execute-Reflect framework. DermAgent delivers stepwise, traceable diagnostic reasoning through three core components. First, it employs complementary visual perception tools for comprehensive morphological description, dermoscopic concept annotation, and disease diagnosis. Second, to overcome the lack of domain prior, a dual-modality retrieval module anchors every prediction in external evidence by cross-referencing 413,210 diagnosed image cases and 3,199 clinical guideline chunks. To further mitigate hallucinations, a deterministic critic module conducts strict post-hoc auditing via confidence, coverage, and conflict gates, automatically detecting inter-source disagreements to trigger targeted self-correction. Extensive experiments on five dermatology benchmarks demonstrate that DermAgent consistently outperforms state-of-the-art MLLMs and medical agent baselines across zero-shot fine-grained disease diagnosis, concept annotation, and clinical captioning tasks, exceeding GPT-4o by 17.6% in skin disease diagnostic accuracy and 3.15% in captioning ROUGE-L. Our code is available at https://github.com/YizeezLiu/DermAgent.

Abstract

Explore similar work

DermAgent: A Self-Reflective Agentic System for Dermatological Image Analysis with Multi-Tool Reasoning and Traceable Decision-Making

Mind the Rarities: Can Rare Skin Diseases Be Reliably Diagnosed via Diagnostic Reasoning?

Skin-R1: Clinical Knowledge-Guided Dermatological Diagnosis Using Vision-Language Models