When AI and Experts Agree on Error: Intrinsic Ambiguity in Dermatoscopic Images

Loris Cino; Pier Luigi Mazzeo; Alessandro Martella; Giulia Radi; Renato Rossi; Cosimo Distante

cs.CVApr 1, 2026

When AI and Experts Agree on Error: Intrinsic Ambiguity in Dermatoscopic Images

Authors: Loris Cino, Pier Luigi Mazzeo, Alessandro Martella, Giulia Radi, Renato Rossi, Cosimo Distante

Organizations: Sapienza - Dipartimento di Ingegneria informatica, automatica e gestionale (DIAG) , V. Ariosto, 25, Rome, 00185, Lazio, Italy · Istituto di Scienze Applicate e Sistemi Intelligenti (ISASI), Consiglio Nazionale delle Ricerche (CNR), Via Monteroni s.n, Lecce, 73100, Italy · Dermatologia Myskin, Poliambulatorio Specialistico Medico-Chirurgico, Via S. Marco, 21, Tiggiano, 73030, Puglia, Italy · AST Pesaro-Urbino, Via Borsellino 4, Fano, 60019 , Marche, Italy · La Rocca Skin Medical Center, Via Marchetti, 110, Senigallia, 61122 , Marche, Italy

Abstract

The integration of artificial intelligence (AI), particularly Convolutional Neural Networks (CNNs), into dermatological diagnosis demonstrates substantial clinical potential. While existing literature predominantly benchmarks algorithmic performance against human experts, our study adopts a novel perspective by investigating the intrinsic complexity of dermatoscopic images. Through rigorous experimentation with multiple CNN architectures, we isolated a subset of images systematically misclassified across all models-a phenomenon statistically proven to exceed random chance. To determine if these failures stem from algorithmic biases or inherent visual ambiguity, expert dermatologists independently evaluated these challenging cases alongside a control group. The results revealed a collapse in human diagnostic performance on the AI-misclassified images. First, agreement with ground-truth labels plummeted, with Cohen's kappa dropping to a mere 0.08 for the difficult images, compared to a 0.61 for the control group. Second, we observed a severe deterioration in expert consensus; inter-rater reliability among physicians fell from moderate concordance (Fleiss kappa = 0.456) on control images to only modest agreement (Fleiss kappa = 0.275) on difficult cases. We identified image quality as a primary driver of these dual systematic failures. To promote transparency and reproducibility, all data, code, and trained models have been made publicly available

Figures & tables

Figure 1: Representative dermatoscopic images for each diagnostic category included in the ISIC 2019 dataset. The first row shows examples correctly classified by all neural network models, while the second row presents images misclassified by the models. Each column corresponds to one diagnostic class (AK, BCC, BKL, DF, MEL, NV, SCC, VASC).

Figure 2: Distribution of the number of images jointly misclassified by all models, obtained through a stratified permutation test used to assess whether the observed level of simultaneous mistakes exceeds what would be expected under the null hypothesis.

	Easy		Difficult
Class	Sensitivity	Specificity	Sensitivity	Specificity
AK	0.3	0.9531	0.125	0.8529
BCC	0.625	0.9848	0.0000	0.9565
BKL	0.5	0.9687	0.0454	0.9846
DF	0.625	1.0000	0.0000	1.0000
MEL	0.9	0.875	0.5344	0.6489

Table 1: Per-class sensitivity and specificity of the dermatologists’ diagnoses for the two image groups: the easy group (images correctly classified by all models) and the difficult group (images consistently misclassified by the CNNs).

Table 2: Contingency tables reporting the distribution of true lesion labels (rows) versus the diagnoses assigned by the majority of dermatologists (columns).

Panel of Figure 3 — Figure 3: Examples of low-quality images excluded from the analysis. The first two images from the left ( 3(a) , 3(b) ) were considered to be of insufficient quality by expert dermatologists. The remaining two images ( 3(c) , 3(d) ) display blurred cases automatically identified by the blur score.

Figure 4: Effect of varying the blur-score threshold on the proportion of images classified as blurry. The curves compare the cumulative percentage of all images and dermatologist-annotated blurred images as the threshold increases.

Figure 5: Images exhibiting strong high-frequency components, identified via Fourier transform. These images are heavily occluded by hair artifacts, and some instances are exact duplicates.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Analysis of dataset distributions: (a) Actinic Keratosis frequency by position, and (b) normalized frequency of diseases by gender.

Panel of Figure 7 — Figure 7: Frequency of diseases normalized by age. The number of patients for each skin disease grouped by age was divided by the total number of patients for the corresponding age. The curve are smoothed to improve visualization.

Panel of Figure 8 — Figure 8: Screenshots of the web-based diagnostic platform developed for this study, designed to facilitate dermatologists’ assessments of challenging images while maintaining data security and user-friendliness.

Figure 9: The architecture utilized during the training phase is schematically represented in the diagram. Initially, the input image (i) is fed into various convolutional neural networks (ii). Each CNN (Convolutional Neural Network) processes the image and outputs probabilities/activations for each class (iii). Subsequently, in step (iv), an aggregation method is applied to these probabilities to generate an ensemble prediction. Finally, in step (v), a meta-classifier is trained using the aggregated probabilities from the ensemble. Additionally, (vi) patient data is incorporated.

Group	Model	BA (%)
Single	EfficientNetB4	$85.36\pm 1.98$
	EfficientNetB5	$84.91\pm 1.71$
	EfficientNetB6	$83.42\pm 0.97$
	ResNext50	$80.80\pm 0.88$
	ResNet152	$79.05\pm 1.99$
Ensemble	MP	$87.53$

Appendix

Table 3: Balanced accuracy (BA) values (mean ± standard deviation) for all evaluated models, including single CNN architectures, ensemble methods—where MP denotes mean prediction and MV denotes majority voting—and meta-classifiers (MC), and meta-classifiers with patient data (MC + PD). MLP refers to a multilayer perceptron. In bold the results with the higher mean.

Figure 10: Critical Difference (CD) diagram reporting the average ranks of the evaluated models across all experimental datasets. Lower ranks indicate better predictive performance. Horizontal connectors identify groups of models whose performance differences are not statistically significant according to the Nemenyi post-hoc test at the 5% significance level. ”PD” represent the model that involve patient data.

Method	F1-Score	Precision	Recall
ResAttentionNet* [ 59 ]	0.82	-	-
ARL-CNN* [ 67 ]	0.80	-	-
Soft* Attention CNN [ 3 ]	0.77	-	-
IARO* [ 1 ]	0.78	-	-
S 2 C-DeLeNet* [ 2 ]	0.81	-	-
X-R50* [ 48 ]	0.81	-	-

Appendix

Table 4: Performance comparison of various methods evaluated on the ISIC 2019 dataset in terms of F1-Score Macro, Precision Macro, and Recall Macro. Results marked with (*) are sourced from the benchmark study conducted by Ghazouani et al. [ 24 ] .

Figure 11: Confusion matrix of the Random Forest model.

Explore similar work

Benchmarking Off-the-Shelf Multimodal AI Models Against Dermatologists on Patient-Captured Skin Images
Sep 21, 2026Rian Dolphin, Laura KnowlesSkin Lesion Classification Medical VLMs
Independent Research, Dublin, Ireland · School of Medicine, University of Limerick, Ireland
Modeling Expert AI Diagnostic Alignment via Immutable Inference Snapshots
Feb 26, 2026Dimitrios P. Panagoulias, Evangelia-Aikaterini Tsichrintzi, Georgios Savvidis +1Clinical Decision Support Medical Diagnosis
Department of Informatics, University of Piraeus, Karaoli ke Dimitriou 80, Piraeus, 18534, Greece · Department of Research & Development, Noetiv PC, Valaoritou 18, Athens, 10671, Greece · Department of Dermatology, Dermacen SA, Valaoritou 18, Athens, 10671, Greece
Bridging Research and Practice: A Systematic Evaluation of Generalist and Dermatology-Specific Models in Clinical Skin Lesion Classification
Oct 2, 2026Emanoel dos Santos, Kelvin Cunha, Rodrigo Mota +8Skin Lesion Classification Cross-Dataset Generalization
Centro de Informática, Universidade Federal de Pernambuco (UFPE), Brazil · Hospital das Clinicas, Ebserh, UFPE, Pernambuco, Brazil

Sep 21, 2026J/K move · Enter open · S save

Benchmarking Off-the-Shelf Multimodal AI Models Against Dermatologists on Patient-Captured Skin Images

Rian Dolphin, Laura Knowles

Independent Research, Dublin, Ireland · School of Medicine, University of Limerick, Ireland

Skin Lesion Classification Medical VLMs Multimodal Model Evaluation

Open paper AskPDF arXiv

Artificial intelligence (AI) has advanced at a rapid pace in recent years. Initially, breakthroughs in large language models caught widespread attention. However, recent generations of frontier AI models have adopted multimodal capabilities as a first class citizen, with vision capabilities being central to that. In this paper, we evaluate three recently released models on the task of diagnosing dermatological conditions from patient-submitted images. The models chosen are at the low to mid tier in terms of pricing and thus represent a floor on current AI capabilities, not a ceiling. We evaluate AI performance relative to a panel of three certified dermatologists, who grade each image, and we present four interesting findings. Firstly, depending on the metric, the tested AI models are either on par or slightly trail humans in terms of inter-clinician agreement. Secondly, we find that asking AI models for a confidence rating produces poorly calibrated answers, meaning use of confidence thresholds should not be relied upon in a clinical setting. Thirdly, the effect of providing additional patient metadata is strongly model-specific, with one of the three models degrading on every metric considered. Finally, model cost is not predictive of performance. The best-performing model we tested costs on average $0.0045 per case.

Abstract

Explore similar work

Benchmarking Off-the-Shelf Multimodal AI Models Against Dermatologists on Patient-Captured Skin Images

Modeling Expert AI Diagnostic Alignment via Immutable Inference Snapshots

Bridging Research and Practice: A Systematic Evaluation of Generalist and Dermatology-Specific Models in Clinical Skin Lesion Classification