Dataset Diversity Metrics and Impact on Classification Models
Authors: Théo Sourget, Niclas Claßen, Jack Junchi Xu, Rob van der Goot, Veronika Cheplygina
Organizations: IT University of Copenhagen, Denmark · Copenhagen University Hospital, Herlev and Gentofte, Denmark · Radiological AI Testcenter, Denmark
The diversity of training datasets is usually perceived as an important aspect to obtain a robust model. However, the definition of diversity is often not defined or differs across papers, and while some metrics exist, the quantification of this diversity is often overlooked when developing new algorithms. In this work, we study the behaviour of multiple dataset diversity metrics for image, text and metadata using MorphoMNIST, a toy dataset with controlled perturbations, and PadChest, a publicly available chest X-ray dataset. We evaluate whether these metrics correlate with each other but also with the intuition of a clinical expert. We also assess whether they correlate with downstream-task performance and how they impact the training dynamic of the models. We find limited correlations between the AUC and image or metadata reference-free diversity metrics, but higher correlations with the FID and the semantic diversity metrics. Finally, the clinical expert indicates that scanners are the main source of diversity in practice. However, we find that the addition of another scanner to the training set leads to shortcut learning. The code used in this study is available at https://github.com/TheoSourget/dataset_diversity_evaluation
Figures & tables
Figure 1: Overview of our study. We assessed dataset diversity measures across multiple modalities and compare their correlations as well as their alignment with domain experts knowledge through interviews. We evaluate their impact on downstream task performances and the effect on subgroups training dynamics.
Figure 2: Example of data from MorphoMNIST
Figure 3: Example of data from both scanners in the PadChest dataset.
Plain
Thin
Thick
Fracture
Swelling
Plain ∪ Thin
Plain ∪ Thick
Plain ∪ Fracture
Plain ∪ Swelling
(N=60k)
(N=60k)
(N=60k)
(N=60k)
(N=60k)
(N=120k)
(N=120k)
(N=120k)
(N=120k)
IS ↑
2.28
2.24
2.35
2.39
2.36
2.36
2.35
2.41
2.40
FID ↓
12.26
31.70
13.66
19.82
25.22
14.72
9.33
5.49
6.24
Vendi (pixel values) ↑
23.64
38.23
13.89
15.89
30.77
30.45
18.45
20.21
27.25
Vendi (hog) ↑
9.70
9.66
10.20
9.10
8.44
9.84
10.09
9.48
9.25
Vendi (Inception) ↑
5.19
5.45
5.55
5.52
5.55
5.62
5.55
5.81
5.94
Table 1: Metrics on the MorphoMNIST dataset. ↑ indicates that a higher value is best, ↓ indicates that a lower value is best. +/- for AUC scores indicate the standard deviation of the five models trained using 5-fold cross-validation on the test set. N indicates the number of samples in the set.
All
Female
Male
Philips scanner
ImagingDynamics scanner
(N=86,460; P=306)
(N=42,823; P=110)
(N=43,625; P=196)
(N=43,047; P=270)
(N=43,413; P=36)
IS ↑
2.30
2.21
2.35
2.62
1.91
[2.30, 2.31]
[2.20, 2.22]
[2.34, 2.36]
[2.61, 2.64]
[1.91, 1.92]
FID ↓
0.57
2.81
2.86
9.82
13.17
[0.67, 0.68]
[2.99, 3.05]
[3.00, 3.09]
[9.85, 10.15]
[13.24, 13.40]
Vendi Score (pixel values) ↑
2.96
3.09
2.75
2.87
2.56
Table 2: Metrics on the chest X-ray dataset. ↑ indicates that a higher value is best, ↓ indicates that a lower value is best. Values in [] indicate the 95% confidence interval computed with the bootstrap method. ± for AUC scores indicate the standard deviation of the five models trained using 5-fold cross-validation on the test set. We show two decimal points for each confidence interval which is why some confidence intervals appear to have an equal lower and upper bound. N indicates the number of samples in the set and P the number of positive samples.
Figure 4: Correlations between the metrics’ ranking of (a) MorphoMNIST and (b) chest X-rays scenarios using the Spearman’s rank correlation.
Male
Female
Philips scanner
ImagingDynamics scanner
(N=10.798, P=50)
(N=10.812, P=27)
(N=10.684, P=74)
(N=10.931, P=3)
All model
0.794
0.802
0.636
0.525
± 0.025
± 0.009
± 0.031
± 0.10
Male model
0.817
0.812
0.667
0.693
± 0.009
± 0.018
± 0.025
± 0.093
Female model
0.759
0.771
0.581
0.484
Table 3: Subgroups’ AUC of the models trained on each subset. We report the mean AUC over the 5-fold models. ± indicates the standard deviation. Bold values indicate the highest mean AUC per subgroup.
Figure 5: Density plots of data maps for models trained with (a) plain images only, (b) plain and thin images, and (c) plain and swelling images for the MorphoMNIST dataset. A type of image is learnt faster if the density of the top left part is higher meaning that the probability in the correct class of this type of image is higher and more constant during the training.
Figure 6: Data maps of positive cases of pneumothorax in the test set of PadChest. Each subplot shows for all samples in the test set the mean and standard deviation of the predicted probability when trained with (a) different scanners, and (b) different patient sex.
Department of Computer Science, Cornell University · Statistics and Data Science Center, Massachusetts Institute of Technology · Department of Statistics, Cornell University