Organizations: Instituto de Ciências Matemáticas e de Computação (ICMC), Universidade de São Paulo (USP), São Carlos, São Paulo, Brazil · Instituto de Matemática Pura e Aplica, Rio de Janeiro, Brazil
Skin tone classification is a fundamental task in computer vision, with applications in quantifying medical features, assessing demographic representativeness, and mitigating algorithmic bias. However, existing datasets either rely on a restrictive 6-tone Fitzpatrick scale, prohibit training through restrictive licenses, or lack representation of extreme skin tones, while current methods often fail to disentangle true skin tone from in-the-wild imaging conditions, degrading to random-classifier performance. We introduce skin tone in the wild (STW), the first open-access dataset annotated with the 10-tone Monk Skin Tone (MST) scale, containing over 42K images and specifically curated to include underrepresented tones. Alongside a rigorous evaluation protocol, we propose SkinToneNet, a ViT-based model that leverages global image context for robust in-the-wild generalization. Experiments show that SkinToneNet achieves 90% weighted off-by-one accuracy, substantially outperforming prior baselines. We further use SkinToneNet to audit widely used face datasets, including CelebA and VGGFace2, revealing substantial representation imbalances across MST tones.
Figures & tables
Figure 1: We introduce skin tone in the wild (STW) , a large-scale dataset annotated on the 10-tone Monk Skin Tone (MST) scale. STW provides broader coverage of underrepresented MST tones, especially darker tones, compared to previous datasets such as FACET. We also present SkinToneNet , a ViT-based model that takes unconstrained face images as input and predicts MST labels. Its learned embedding organizes in-the-wild faces along the MST spectrum, revealing a continuous structure aligned with skin tone.
Figure 2: Dataset distribution along the MST scale. Compared to existing MST-annotated benchmarks, our skin tone in the wild (STW) dataset provides substantially more in-the-wild samples and vastly superior coverage across the full 10-tone scale, specifically expanding representation for tones at the absolute extremes.
Figure 3: Overview of the STW pipeline. We construct STW from multiple open-access face datasets and annotate identities on the 10-tone MST scale (top-left). Annotators compare each identity against MST reference examples and their in-the-wild image variants (top-right). We compare standard image-level splits, where the same identity may appear in both training and test sets, with identity-level splits, which prevent identity overlap (bottom-left). We evaluate both classic and deep learning models using different input representations (bottom-center). Finally, we benchmark prior and ours models on STW and external MST-annotated datasets, showing that our method achieves stronger in-domain and out-of-domain generalization (bottom-right).
Figure 4: Distribution of skin tone in the wild (STW). Standard image-level class distribution (left) and individual-level class distribution (right) across MST classes, with the y-axis denoting the number of images and individuals, respectively. The individual-level distribution shows how targeted aggregation of datasets such as CASIA Face Africa and FairFace improves coverage of underrepresented MST tones.
Figure 5: Inter-annotator confusion matrices. Correlation between Annotator 1 vs. 3, and Annotator 3 vs. 2. The tight clustering of predictions along the main and adjacent diagonals highlights the ordinal nature of human skin tone perception, resulting in an 88% off-by-one agreement rate, as shown in red.
Models\Dataset
N of Cls.
STW (Ours)
CCv2
MSTE
MSTE-G
FACET
Random
Tadesse et al. (2023)
2
0.89 / 1.00
0.53 / 1.00
0.60 / 1.00
0.56 / 1.00
0.56 / 1.00
0.5 / 1.00
Groh et al. (2021)
6
0.19 / 0.46
0.25 / 0.74
0.23 / 0.56
0.41 / 0.93
0.19 / 0.23
0.17 / 0.44
Kinyanjui et al. (2019)
8
0.12 / 0.20
0.17 / 0.68
0.15 / 0.29
0.15 / 0.29
0.08 / 0.22
0.13 / 0.34
stone Rejon Pina and Ma (2023)
10
0.12 / 0.28
0.14 / 0.07
0.14 / 0.45
0.20 / 0.43
0.06 / 0.22
0.10 / 0.28
VehicleNet Rachmadi and Purnama (2015)
10
0.38 / 0.75
0.16 / 0.46
0.19 / 0.63
0.24 / 0.82
0.22 / 0.60
0.10 / 0.28
LabNet Robin et al. (2020)
10
0.36 / 0.79
0.16 / 0.47
0.25 / 0.63
0.47 / 0.82
0.21 / 0.58
0.10 / 0.28
Table 1: In-domain and out-of-domain benchmark comparison. Each entry reports Acc/ OOAcc . SkinToneNet outperforms prior methods, classical baselines, and learning-based methods across STW and external MST-annotated datasets. Cross-dataset evaluation uses Acc and OOAcc , since balanced and weighted metrics depend on dataset-specific class distributions.
Figure 6: SkinToneNet confusion matrices. Evaluation on the in-domain STW test set (left) and the out-of-domain MSTE dataset (right). The strong diagonals and tight clustering of misclassifications along adjacent classes highlight the model’s robust ordinal predictions and high off-by-one accuracy ( OOAcc ), as shown in red.
Model
Split
Seg
STW - Test
STW - Custom Test
bAcc
wOOAcc
bAcc
wOOAcc
SkinToneCCV
Standard
Skin
0.889
0.941
0.168
0.442
SkinToneCCV
Cheeks-Nose
0.879
0.929
0.196
0.434
SkinToneNet
Skin
0.836
0.924
0.209
0.657
SkinToneNet
Full image
0.905
0.939
0.356
0.541
SkinToneCCV
Individual
Skin
0.331
0.682
0.322
0.656
Table 2: Effect of split protocol and input representation. Performance under the STD split is substantially inflated and drops on the identity-held-out Custom Test set, revealing identity leakage. The IND split provides a more reliable estimate of generalization. SkinToneCCV performs best with segmented facial inputs, whereas SkinToneNet performs best with full-image inputs.
Network architecture
STW
MSTE
CCv2
bAcc ↑
wOOAcc ↑
Acc ↑
OOAcc ↑
Acc ↑
OOAcc ↑
LabNet Robin et al. (2020)
0.328
0.777
0.227
0.588
0.195
0.507
VehicleNet Rachmadi and Purnama (2015)
0.355
0.769
0.151
0.638
0.163
0.470
ResNet18 He et al. (2016)
0.389
0.856
0.321
0.740
0.228
0.615
DenseNet121 Huang et al. (2017)
0.445
0.860
0.292
0.792
0.227
0.626
DINOv2 Oquab et al. (2024)
0.373
0.820
0.329
0.766
0.276
0.650
Table 3: Architecture comparison. SkinToneNet, a fine-tuned ViT-S/16, outperforms baseline architectures and shows strong generalization across STW, MSTE, and CCv2.
Figure 7: Grad-CAM visualizations on the out-of-domain MSTE-G dataset. The top row shows unconstrained input faces spanning MST tones 1–10. The middle and bottom rows show activation maps for SkinToneNet and DenseNet121, respectively. SkinToneNet produces more localized activations on facial and nearby skin regions.
Figure 8: Confusion matrix analysis of the SkinToneCCV model. The model collapses towards frequent labels.
Figure 9: Qualitative failure analysis on the MSTE set. Representative correct predictions are indicated by green borders. Failure cases, defined as instances misclassified by two or more classes, are highlighted with red borders and mapped to their corresponding positions on the confusion matrix.
Figure 10: Zero-shot skin tone distribution audit across standard datasets Gustafson et al. (2023) ; Wang et al. (2018) ; Liu et al. (2018) ; Cao et al. (2018) ; Banerjee et al. (2020) ; Karkkainen and Joo (2021) ; Phillips et al. (1998) ; Huang et al. (2008) . Proportions of predicted MST classes across eight standard vision benchmarks. The dashed horizontal line indicates an ideal uniform distribution of 10% per class.
Skin cancer is one of the most common cancers worldwide and early detection is critical for effective treatment. However, current AI diagnostic tools are often trained on datasets dominated by lighter skin tones, leading to reduced accuracy and fairness for people with darker skin. The International Skin Imaging Collaboration (ISIC) dataset, one of the most widely used benchmarks, contains over 70% light skin images while dark skins fewer than 8%. This imbalance poses a significant barrier to equitable healthcare delivery and highlights the urgent need for methods that address demographic diversity in medical imaging. This paper addresses this challenge of skin tone imbalance in automated skin cancer detection using dermoscopic images. To overcome this, we present a generative augmentation pipeline that fine-tunes a pre-trained Stable Diffusion model using Low-Rank Adaptation (LoRA) on the image dark-skin subset of the ISIC dataset and generates synthetic dermoscopic images conditioned on lesion type and skin tone. In this study, we investigated the utility of these images on two downstream tasks: lesion segmentation and binary classification. For segmentation, models trained on the augmented dataset and evaluated on held-out real images show consistent improvements in IoU, Dice coefficient, and boundary accuracy. These evalutions provides the verification of Generated dataset. For classification, an EfficientNet-B0 model trained on the augmented dataset achieved 92.14% accuracy. This paper demonstrates that synthetic data augmentation with Generative AI integration can substantially reduce bias with increase fairness in conventional dermatological diagnostics and open challenges for future directions.
Areez Muhammed Shabu, Mohammad Samar Ansari, Asra Aslam
University of Sheffield, UK · University of Chester, UK
Skin diseases represent a major global public health burden, yet machine learning tools developed to assist in their diagnosis suffer from two critical limitations: reliance on only one modality for diagnosis and systematic performance disparities across skin tones. While existing approaches address each challenge separately, this work proposes a modality-invariant framework with fair representation (MIFR) for skin disease classification. The architecture pairs clinical photographs with dermoscopic images using ViT-based encoders, projecting each input into a high-dimensional embedding space via modality-specific projection heads. The resulting model is trained with a five-component multi-objective loss including weighted cross-entropy for classification, confusion and skin-type classification losses for fairness, per-modality supervised contrastive loss for class alignment, and a modality-invariance loss for clinical and dermoscopic modality alignment. Experiments on the HIBA+Derm7pt paired dataset and the external PAD-UFES-20 and ISIC 2019 datasets showed that modality-invariant representation learning provides competitive predictive performance compare to relevant baseline models and competitive fairness on the internal dataset. t-SNE visualizations confirmed that clinical and dermoscopic embeddings of the same disease are geometrically aligned, validating the joint objectives.
Asonyu Senge Njih, Yvan Guifo Fodjo, Vianney Kengne Tchendji +2
Department of Mathematics and Computer Science, University of Dschang, Cameroon · EFREI Research Lab, Université Paris-Panthéon-Assas, France · MIS, Université de Picardie Jules Verne, France +1
Accurate dermatological diagnosis naturally necessitates equitable performance across diverse populations, yet a systematic lack of expertly annotated images, especially for underrepresented skin tones and rare diseases, impedes progress toward measurably fair methods. We introduce cgDDI (Controllable Generation of Diverse Dermatological Imagery), a hybrid framework that (1) synthesizes realistic healthy skin samples without disturbing other input properties, (2) maps single-sample rare lesions onto novel skin-tones and locations non-parametrically, and (3) allows for efficient parametric generation with as few as 10 training samples. The framework supports both human and automated segmentation masking, enabling scalability to datasets without pre-made lesion masks. We grow a 656-image dataset by more than 400x and validate across two datasets: biopsy-confirmed Diverse Dermatology Images (DDI) and expert-verified Fitzpatrick17k (F17k). On the DDI benchmark, we achieve malignancy classification accuracy of 86.4% under synthetic-only training and 90.9% state-of-the-art performance with real data fine-tuning, alongside leading fairness metrics. Cross-dataset experiments show +13.9% accuracy improvements on unseen F17k data despite minimal disease overlap. We openly release 266k+ synthetic images, code, and generative models to further support fairness research at https://github.com/hectorcarrion/ControllableGenDDI.
Héctor Carrión, Narges Norouzi
University of California, Santa Cruz, USA · University of California, Berkeley, USA