cs.CVMar 2, 2026
SaveSkin Tone in the Wild: A Dataset and Benchmark for 10-tone MST Classification
Organizations: Instituto de Ciências Matemáticas e de Computação (ICMC), Universidade de São Paulo (USP), São Carlos, São Paulo, Brazil · Instituto de Matemática Pura e Aplica, Rio de Janeiro, Brazil
Abstract
Skin tone classification is a fundamental task in computer vision, with applications in quantifying medical features, assessing demographic representativeness, and mitigating algorithmic bias. However, existing datasets either rely on a restrictive 6-tone Fitzpatrick scale, prohibit training through restrictive licenses, or lack representation of extreme skin tones, while current methods often fail to disentangle true skin tone from in-the-wild imaging conditions, degrading to random-classifier performance. We introduce skin tone in the wild (STW), the first open-access dataset annotated with the 10-tone Monk Skin Tone (MST) scale, containing over 42K images and specifically curated to include underrepresented tones. Alongside a rigorous evaluation protocol, we propose SkinToneNet, a ViT-based model that leverages global image context for robust in-the-wild generalization. Experiments show that SkinToneNet achieves 90% weighted off-by-one accuracy, substantially outperforming prior baselines. We further use SkinToneNet to audit widely used face datasets, including CelebA and VGGFace2, revealing substantial representation imbalances across MST tones.
Figures & tables
Figure 1: We introduce skin tone in the wild (STW) , a large-scale dataset annotated on the 10-tone Monk Skin Tone (MST) scale. STW provides broader coverage of underrepresented MST tones, especially darker tones, compared to previous datasets such as FACET. We also present SkinToneNet , a ViT-based model that takes unconstrained face images as input and predicts MST labels. Its learned embedding organizes in-the-wild faces along the MST spectrum, revealing a continuous structure aligned with skin tone.
Figure 2: Dataset distribution along the MST scale. Compared to existing MST-annotated benchmarks, our skin tone in the wild (STW) dataset provides substantially more in-the-wild samples and vastly superior coverage across the full 10-tone scale, specifically expanding representation for tones at the absolute extremes.
Figure 3: Overview of the STW pipeline. We construct STW from multiple open-access face datasets and annotate identities on the 10-tone MST scale (top-left). Annotators compare each identity against MST reference examples and their in-the-wild image variants (top-right). We compare standard image-level splits, where the same identity may appear in both training and test sets, with identity-level splits, which prevent identity overlap (bottom-left). We evaluate both classic and deep learning models using different input representations (bottom-center). Finally, we benchmark prior and ours models on STW and external MST-annotated datasets, showing that our method achieves stronger in-domain and out-of-domain generalization (bottom-right).
Figure 4: Distribution of skin tone in the wild (STW). Standard image-level class distribution (left) and individual-level class distribution (right) across MST classes, with the y-axis denoting the number of images and individuals, respectively. The individual-level distribution shows how targeted aggregation of datasets such as CASIA Face Africa and FairFace improves coverage of underrepresented MST tones.
Figure 5: Inter-annotator confusion matrices. Correlation between Annotator 1 vs. 3, and Annotator 3 vs. 2. The tight clustering of predictions along the main and adjacent diagonals highlights the ordinal nature of human skin tone perception, resulting in an 88% off-by-one agreement rate, as shown in red.
| Models\Dataset | N of Cls. | STW (Ours) | CCv2 | MSTE | MSTE-G | FACET | Random |
|---|---|---|---|---|---|---|---|
| Tadesse et al. (2023) | 2 | 0.89 / 1.00 | 0.53 / 1.00 | 0.60 / 1.00 | 0.56 / 1.00 | 0.56 / 1.00 | 0.5 / 1.00 |
| Groh et al. (2021) | 6 | 0.19 / 0.46 | 0.25 / 0.74 | 0.23 / 0.56 | 0.41 / 0.93 | 0.19 / 0.23 | 0.17 / 0.44 |
| Kinyanjui et al. (2019) | 8 | 0.12 / 0.20 | 0.17 / 0.68 | 0.15 / 0.29 | 0.15 / 0.29 | 0.08 / 0.22 | 0.13 / 0.34 |
| stone Rejon Pina and Ma (2023) | 10 | 0.12 / 0.28 | 0.14 / 0.07 | 0.14 / 0.45 | 0.20 / 0.43 | 0.06 / 0.22 | 0.10 / 0.28 |
| VehicleNet Rachmadi and Purnama (2015) | 10 | 0.38 / 0.75 | 0.16 / 0.46 | 0.19 / 0.63 | 0.24 / 0.82 | 0.22 / 0.60 | 0.10 / 0.28 |
| LabNet Robin et al. (2020) | 10 | 0.36 / 0.79 | 0.16 / 0.47 | 0.25 / 0.63 | 0.47 / 0.82 | 0.21 / 0.58 | 0.10 / 0.28 |
Table 1: In-domain and out-of-domain benchmark comparison. Each entry reports Acc/ OOAcc . SkinToneNet outperforms prior methods, classical baselines, and learning-based methods across STW and external MST-annotated datasets. Cross-dataset evaluation uses Acc and OOAcc , since balanced and weighted metrics depend on dataset-specific class distributions.
Figure 6: SkinToneNet confusion matrices. Evaluation on the in-domain STW test set (left) and the out-of-domain MSTE dataset (right). The strong diagonals and tight clustering of misclassifications along adjacent classes highlight the model’s robust ordinal predictions and high off-by-one accuracy ( OOAcc ), as shown in red.
| Model | Split | Seg | STW - Test | STW - Custom Test | ||
|---|---|---|---|---|---|---|
| bAcc | wOOAcc | bAcc | wOOAcc | |||
| SkinToneCCV | Standard | Skin | 0.889 | 0.941 | 0.168 | 0.442 |
| SkinToneCCV | Cheeks-Nose | 0.879 | 0.929 | 0.196 | 0.434 | |
| SkinToneNet | Skin | 0.836 | 0.924 | 0.209 | 0.657 | |
| SkinToneNet | Full image | 0.905 | 0.939 | 0.356 | 0.541 | |
| SkinToneCCV | Individual | Skin | 0.331 | 0.682 | 0.322 | 0.656 |
Table 2: Effect of split protocol and input representation. Performance under the STD split is substantially inflated and drops on the identity-held-out Custom Test set, revealing identity leakage. The IND split provides a more reliable estimate of generalization. SkinToneCCV performs best with segmented facial inputs, whereas SkinToneNet performs best with full-image inputs.
| Network architecture | STW | MSTE | CCv2 | |||
|---|---|---|---|---|---|---|
| bAcc | wOOAcc | Acc | OOAcc | Acc | OOAcc | |
| LabNet Robin et al. (2020) | 0.328 | 0.777 | 0.227 | 0.588 | 0.195 | 0.507 |
| VehicleNet Rachmadi and Purnama (2015) | 0.355 | 0.769 | 0.151 | 0.638 | 0.163 | 0.470 |
| ResNet18 He et al. (2016) | 0.389 | 0.856 | 0.321 | 0.740 | 0.228 | 0.615 |
| DenseNet121 Huang et al. (2017) | 0.445 | 0.860 | 0.292 | 0.792 | 0.227 | 0.626 |
| DINOv2 Oquab et al. (2024) | 0.373 | 0.820 | 0.329 | 0.766 | 0.276 | 0.650 |
Table 3: Architecture comparison. SkinToneNet, a fine-tuned ViT-S/16, outperforms baseline architectures and shows strong generalization across STW, MSTE, and CCv2.
Figure 7: Grad-CAM visualizations on the out-of-domain MSTE-G dataset. The top row shows unconstrained input faces spanning MST tones 1–10. The middle and bottom rows show activation maps for SkinToneNet and DenseNet121, respectively. SkinToneNet produces more localized activations on facial and nearby skin regions.
Figure 8: Confusion matrix analysis of the SkinToneCCV model. The model collapses towards frequent labels.
Figure 9: Qualitative failure analysis on the MSTE set. Representative correct predictions are indicated by green borders. Failure cases, defined as instances misclassified by two or more classes, are highlighted with red borders and mapped to their corresponding positions on the confusion matrix.
Figure 10: Zero-shot skin tone distribution audit across standard datasets Gustafson et al. (2023) ; Wang et al. (2018) ; Liu et al. (2018) ; Cao et al. (2018) ; Banerjee et al. (2020) ; Karkkainen and Joo (2021) ; Phillips et al. (1998) ; Huang et al. (2008) . Proportions of predicted MST classes across eight standard vision benchmarks. The dashed horizontal line indicates an ideal uniform distribution of 10% per class.