cs.CVMar 19, 2026
SaveMind the Rarities: Can Rare Skin Diseases Be Reliably Diagnosed via Diagnostic Reasoning?
Organizations: Carnegie Mellon University · University of Alabama at Birmingham · Australian National University · University of Western Australia · POSTECH · Yale University
Abstract
Large vision-language models (LVLMs) demonstrate strong performance in dermatology; however, evaluating diagnostic reasoning for rare conditions remains largely unexplored. Existing benchmarks focus on common diseases and assess only final accuracy, overlooking the clinical reasoning process, which is critical for complex cases. We address this gap by constructing DermCase, a long-context benchmark derived from peer-reviewed case reports. Our dataset contains 26,030 multi-modal image-text pairs and 6,354 clinically challenging cases, each annotated with comprehensive clinical information and step-by-step reasoning chains. To enable reliable evaluation, we establish DermLIP-based similarity metrics that achieve stronger alignment with dermatologists for assessing differential diagnosis quality. Benchmarking 22 leading LVLMs exposes significant deficiencies across diagnosis accuracy, differential diagnosis, and clinical reasoning. Fine-tuning experiments demonstrate that instruction tuning substantially improves performance while Direct Preference Optimization (DPO) yields minimal gains. Systematic error analysis further reveals critical limitations in current models' reasoning capabilities.
Figures & tables
Figure 1: An example from DermCase. Each case comprises a clinical question with detailed context (patient history, laboratory results), multi-modal images, and step-wise diagnostic reasoning.
| Dataset | Num. of Samples | Avg. Tokens Per Data | Data Type | Evaluation Perspectives | Open Source | ||
| Diagnosis Accuracy | Differential Diagnosis | Rubrics - based Reasoning | |||||
| DermNet [ 5 ] | 19,500 | – | Image only | ✓ | ✗ | ✗ | ✓ |
| SCIN [ 34 ] | 10,408 | 10 | Image-caption pair | ✓ | ✗ | ✗ | ✓ |
| EVAL - GPT - Derm [ 3 ] | 93 | 10 | Text only | ✓ | ✗ | ✗ | ✗ |
| Pillai et al. [ 20 ] | 102 | 50 | Image-caption pair | ✓ | ✗ | ✗ | ✗ |
| SkinGPT4 - Clinical - Eval [ 43 ] | 150 | 20 | Image-caption pair | ✓ | ✗ | ✗ | ✗ |
Table 1: Comparison of existing dermatology benchmarks. Avg. Tokens indicates the average text length per data instance ( e.g. , case narrative, image caption, or question-answer pair). DermCase is the first benchmark providing full-length case narratives with multi-modal grounded images, enabling evaluation of diagnostic accuracy, differential diagnosis and multi-step clinical reasoning.
Figure 2: The dataset curation pipeline includes: data source selection and filtering, multi-modal image preprocessing, case-level information extraction, visual-grounded question generation, and expert-level diagnostic reasoning generation. Quality control (steps 3-5) combines LLM-based cross-verification and human sampling validation against source reports, triggering regeneration for inconsistencies.
Figure 3: Statistical overview of the DermCase dataset. (a) Age distribution peaking at 50–59 years (15.9%). (b) Female-dominant gender distribution (54.5%). (c) Top 10 diseases, led by Basal Cell Carcinoma and Squamous Cell Carcinoma.
| Methods | Final Diagnosis | Differential Diagnosis | ||||||||
| Accuracy | Coverage | Macro | Micro | |||||||
| D-Precision | D-Recall | D-F1 | D-Jaccard | D-Precision | D-Recall | D-F1 | D-Jaccard | |||
| Flamingo Series | ||||||||||
| Med-Flamingo [ 17 ] | 0.1098 | 1.0000 | 0.6544 | 0.1875 | 0.2676 | 0.1680 | 0.4742 | 0.1814 | 0.2624 | 0.1510 |
| InternVL Series | ||||||||||
| InternVL2.5-2B [ 2 ] | 0.0616 | 0.8952 | 0.3597 | 0.2734 | 0.2886 | 0.1885 | 0.3269 | 0.2692 | 0.2953 | 0.1732 |
Table 2: Benchmark on final diagnosis and differential diagnosis. Differential diagnosis is assessed by coverage and our proposed DermLIP-based metrics (D-Precision, Recall, F1, and Jaccard). Best and second-best are highlighted in red and blue, respectively.
| Model | Medical Knowledge | Image Interpretation | Laboratory Interpretation | Differential Diagnosis | Clinical Reasoning | Total |
| (0-5) | (0-25) | (0-15) | (0-25) | (0-30) | (0-100) | |
| LLaVA Series | ||||||
| Llava-1.5-7B [ 14 ] | 1.96 0.85 | 5.68 4.36 | 5.41 3.43 | 6.21 1.76 | 7.91 2.20 | 27.02 8.20 |
| Llava-1.6-Vicuna-7B [ 15 ] | 1.45 1.65 | 3.24 6.10 | 4.03 5.14 | 4.41 5.60 | 5.47 7.09 | 18.53 23.02 |
| Llava-Med-1.5-Mistral-7B [ 13 ] | 1.99 1.34 | 4.05 6.78 | 5.45 4.66 | 6.51 5.37 | 8.63 6.65 | 26.53 22.23 |
| Med-Flamingo | ||||||
Table 3: Rubrics-based reasoning evaluation. 22 LVMs across five dimensions: Medical Knowledge (0-5), Image Interpretation (0-25), Laboratory Interpretation (0-15), Differential Diagnosis (0-25), and Clinical Reasoning (0-30). Mean ± Std are reported.
Figure 4: Fine-tuning results on DermCase. SFT consistently improves final-diagnosis accuracy and differential-diagnosis metrics for InternVL and Med-Gemma; DPO offers modest additional gains; however, MPO slightly degrades performance on InternVL.
Figure 5: Validation of similarity metrics against dermatologist assessments. DermLIP Score exhibits substantially better alignment with dermatologist judgments than BERT Score in both correlation with similarity ratings and pairwise preference prediction.
Figure 6: Error taxonomy and representative failure cases. The distribution across 10 error categories is shown top-left, with examples illustrating the three most prevalent types: visual interpretation error, logical inconsistency, and factual medical error.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Token-length statistics and multimodal composition in DermCase. (a) Distribution of question lengths. (b) Distribution of reasoning-chain lengths. (c) Distribution of image modalities associated with each case.
Figure 8: Word cloud illustrating the spectrum of dermatologic conditions covered in DermCase. Text size reflects diagnosis frequency, with prominent conditions such as Basal Cell Carcinoma, Squamous Cell Carcinoma, Psoriasis, Mycosis Fungoides, and Atopic Dermatitis. Notably, even the most frequent diagnosis, Basal Cell Carcinoma, appears in only 86 cases, while 1277 conditions occur as singletons (one case each), highlighting a pronounced long-tailed distribution of rare dermatologic entities.
Figure 9: Information extraction prompt for parsing dermatology case reports. The prompt defines structured output fields with direct quotation requirements to preserve clinical terminology; it includes multi-case document handling rules to isolate specific cases and prevent data mixing, ensuring accurate extraction of clinical details into standardized markdown format.
Figure 10: The question generation prompt converts structured dermatology cases into natural clinical questions that provide sufficient systemic context while requiring image-based inspection for cutaneous findings. It offers multiple question templates and carefully restricts accessible information sources, encouraging genuine multimodal reasoning rather than reliance on textual heuristics.
Figure 11: The prompt defines a five-step clinical reasoning structure with explicit guidelines emphasizing narrative coherence and interpretive analysis while implementing strict constraints that prohibit inference or fabrication of clinical details not present in source case data.
Figure 12: Final diagnosis extraction prompt used to derive labels from free-text dermatology case reports. The prompt combines concise instructions and examples to identify the most likely or primary diagnosis, separate it from differential diagnosis, and normalize it to a concise disease name; it also defines a “None” fallback when no definitive diagnosis is present to reduce noisy labels.
Figure 13: Final diagnosis evaluation prompt in our LLM-as-a-judge setup. The prompt provides criteria for comparing predicted and reference diagnoses, encourages tolerance to harmless wording differences, and yields a categorical correctness label that supports the computation of final-diagnosis accuracy.
Figure 14: Differential diagnosis extraction prompt used to derive lists of differential diagnoses from both ground-truth clinical texts and model-generated responses. The prompt includes pattern-based rules and normalization heuristics to detect differential-diagnosis phrases in free text, merge synonymous disease terms, and output a canonical, de-duplicated set of alternative diagnoses for each case.
Figure 15: Distribution of normalized DermLIP-based similarity values. Blue bars show similarity scores between dermatology terms within models’ differential diagnosis responses, while orange bars show similarities between the degenerate output ’none’ and ground-truth differential diagnoses in cases where no differentials were predicted. The red dashed line indicates the maximum similarity among ’none’ -based scores (0.8388), motivating our choice of a 0.83 for threshold .
| Methods | BLEU | METEOR | ROUGE-1 | ROUGE-2 | ROUGE-L | BERTScore | LLM-As-a-Judge Accuracy |
| LLaVA Series | |||||||
| Llava-1_5-7B | 0.0207 | 0.0733 | 0.0767 | 0.0156 | 0.0763 | 0.8172 | 0.1138 |
| Llava-V1_6-Vicuna-7B | 0.0076 | 0.0300 | 0.0326 | 0.0082 | 0.0323 | 0.7921 | 0.0347 |
| Llava-Med-V1_5-Mistral-7B | 0.0181 | 0.0643 | 0.0690 | 0.0110 | 0.0686 | 0.8117 | 0.1914 |
| Med-Flamingo | |||||||
| Med-Flamingo | 0.0112 | 0.0380 | 0.0432 | 0.0087 | 0.0431 | 0.8023 | 0.1098 |
Table 4: Results for final diagnosis across different vision–language models. The table reports BLEU, METEOR, ROUGE-1/2/L, BERTScore, and LLM-as-a-judge Accuracy for 22 LVMs, including LLaVA, Med-Flamingo, InternVL, Qwen, MiniCPM, and MedGemma. The highest value in each column is highlighted in red, and the second highest in blue.
Figure 16: LLM-as-a-judge prompt used to score model responses across five weighted dimensions of diagnostic reasoning. The rubric enforces ground truth–anchored comparison against expert reference answers, specifies penalties for hallucinations and use of unprovided data, and constrains the output to structured scores with per-dimension justifications.
Figure 17: The prompt maintains the same clinical reasoning structure and factual data as correct diagnoses while introducing 3-4 cognitive biases across the diagnostic steps, producing wrong conclusions that are clinically defensible and useful for training diagnostic error detection.
Figure 18: The prompt preserves the clinical reasoning structure while embedding 3-4 logical errors (correlation-causation confusion, mechanistic oversimplification, evidence weighting errors, etc. ) that are subtle enough to require medical expertise to detect, yet still arrive at the correct final diagnosis.
Figure 19: Token length distribution for each case, including correct reasoning, flawed reasoning, and wrong diagnosis.
Figure 20: Questionnaire interface for dermatologist validation. The questionnaire comprises three evaluation tasks: (Task 1) rating image-text relevance between clinical images and diagnostic labels on a 1-7 Likert scale; (Task 2) rating diagnostic similarity between diagnosis pairs; (Task 3) selecting the more similar diagnosis pair in forced-choice comparisons. The interface provides clear task instructions, visual presentations, and progress tracking across 50 evaluation cases.
Figure 21: Representative failure cases across ten categories illustrating that LVM errors stem not from isolated mistakes but from systematic breakdowns in visual grounding, clinical reasoning, and medical knowledge integration.