DermaVQA-DAS: Dermatology Assessment Schema (DAS) & Datasets for Closed-Ended Question Answering & Segmentation in Patient-Generated Dermatology Images
Organizations: Microsoft Health AI · University of Washington · Stanford University · Hospital Clinic of Barcelona
Abstract
Recent advances in dermatological image analysis have been driven by large-scale annotated datasets; however, most existing benchmarks focus on dermatoscopic images and lack patient-authored queries and clinical context, limiting their applicability to patient-centered care. To address this gap, we introduce DermaVQA-DAS, an extension of the DermaVQA dataset that supports two complementary tasks: closed-ended question answering (QA) and dermatological lesion segmentation. Central to this work is the Dermatology Assessment Schema (DAS), a novel expert-developed framework that systematically captures clinically meaningful dermatological features in a structured and standardized form. DAS comprises 36 high-level and 27 fine-grained assessment questions, with multiple-choice options in English and Chinese. Leveraging DAS, we provide expert-annotated datasets for both closed QA and segmentation and benchmark state-of-the-art multimodal models. For segmentation, we evaluate multiple prompting strategies and show that prompt design impacts performance: the default prompt achieves the best results under Mean-of-Max and Mean-of-Mean evaluation aggregation schemes, while an augmented prompt incorporating both patient query title and content yields the highest performance under majority-vote-based microscore evaluation, achieving a Jaccard index of 0.395 and a Dice score of 0.566 with BiomedParse. For closed-ended QA, overall performance is strong across models, with average accuracies ranging from 0.729 to 0.798; o3 achieves the best overall accuracy (0.798), closely followed by GPT-4.1 (0.796), while Gemini-1.5-Pro shows competitive performance within the Gemini family (0.783). We publicly release DermaVQA-DAS, the DAS schema, and evaluation protocols to support and accelerate future research in patient-centered dermatological vision-language modeling (https://osf.io/72rp3).
Figures & tables
| Category | Type | Question | Percent |
|---|---|---|---|
| General | Site | How much of the body is effected? | 0.99 |
| General | Site | 1 Where is the affected area? | 0.98 |
| Skin Specific | Affected area description | 1 What label best describes the affected area? | 0.98 |
| Skin Specific | Lesion Color | Compared to the normal surrounding skin, what is the color of the skin lesion? | 0.97 |
| Skin Specific | Lesion Count | How many skin lesions are there? | 0.94 |
| General | Size | 1 How large are the affected areas? Please specify which affected area for each selection. | 0.94 |
| CQID | Question Type | Slots | # Options | Example Options |
| CQID010 | Site Extent | 1 | 4 | single spot; widespread |
| CQID011 | Site Location | 6 | 8 | head; upper extremities |
| CQID012 | Size | 6 | 4 | thumb nail; palm |
| CQID015 | Onset | 1 | 7 | within days; within months |
| CQID020 | Skin Description | 6 | 10 | raised; flat; crust |
| CQID025 | Itch | 1 | 3 | yes; no |
| Split | Encounters | Images | Masks |
|---|---|---|---|
| Train | 842 | 2,474 | 7,448 |
| Valid | 56 | 157 | 472 |
| Test | 100 | 314 | 944 |
| Total | 998 | 2,945 | 8,864 |
| Mean of Max | Mean of Mean | Majority Vote | ||||
|---|---|---|---|---|---|---|
| Model | Jaccard | Dice | Jaccard | Dice | Jaccard | Dice |
| MedSAM (no prompt) | 0.4097 | 0.5250 | 0.3299 | 0.4386 | 0.2597 | 0.4123 |
| BiomedParse (prompt I) | 0.5088 | 0.6133 | 0.4253 | 0.5315 | 0.3863 | 0.5573 |
| BiomedParse (prompt II) | 0.5034 | 0.6068 | 0.4215 | 0.5268 | 0.3893 | 0.5604 |
| BiomedParse (prompt III) | 0.5081 | 0.6120 | 0.4243 | 0.5298 | 0.3948 | 0.5661 |
| System | CQID010 | CQID011 | CQID012 | CQID015 | CQID020 | CQID025 | CQID034 | CQID035 | CQID036 | ALL |
|---|---|---|---|---|---|---|---|---|---|---|
| Gemini-1.5-Pro | 0.650 | 0.876 | 0.793 | 0.820 | 0.746 | 0.900 | 0.660 | 0.890 | 0.710 | 0.783 |
| Gemini 2.0 Flash | 0.640 | 0.873 | 0.769 | 0.790 | 0.757 | 0.930 | 0.630 | 0.840 | 0.680 | 0.768 |
| Claude 3.7 Sonnet (2025-02-19) | 0.620 | 0.853 | 0.780 | 0.840 | 0.720 | 0.960 | 0.560 | 0.880 | 0.640 | 0.761 |
| o4-mini (2025-04-16) | 0.660 | 0.847 | 0.590 | 0.810 | 0.735 | 0.970 | 0.620 | 0.900 | 0.430 | 0.729 |
| GPT-4.1 | 0.670 | 0.881 | 0.830 | 0.850 | 0.757 | 0.930 | 0.640 | 0.880 | 0.730 | 0.796 |
| o3 (2025-04-16) | 0.650 | 0.893 | 0.835 | 0.830 | 0.756 | 0.980 | 0.600 | 0.900 | 0.740 | 0.798 |
| QID | Spearman | Desc |
| CQID010 | -0.147 | Site Extent |
| CQID015 | 0.116 | Onset |
| CQID025 | 0.116 | Itch |
| CQID034 | 0.134 | Lesion Color |
| CQID035 | -0.001 | Lesion Count |
| CQID036 | 0.046 | Texture |
Medical Visual Question Answering
Medical visual question answering studies how AI systems answer clinical questions using medical images and related patient or report data.
See all 181 papers