Deep learning from the crowd Fundamentals of morphological galaxy classification
Authors: Luis Enrique Sucar, Carlos del Burgo, Jonathan Serrano-Pérez
Organizations: Instituto de Astrofísica de Canarias, Vía Láctea S/N, La Laguna 38200, Tenerife, Spain · Instituto Nacional de Astrofísica, Óptica y Electrónica, Luis Enrique Erro #1, CP 72840, Tonantzintla, Puebla, Mexico · Departamento de Astrofísica, Universidad de la Laguna, La Laguna 38200, Tenerife, Spain · Facultad de Ciencias Básicas, Ing. y Tec., Universidad Autónoma de Tlaxcala, Carretera Apizaquito S/N, CP 90401, San Luis Apizaquito, Tlaxcala, Mexico
Aims. The objective of this work is to adapt a deep neural network model to perform galaxy morphological classification trained from crowd annotations, considering the training scheme, the agreement between the annotators, and the hierarchy. Methods. We use Galaxy Zoo 1 as our experimental testbed and trained a convolutional neural network (CNN) for the automatic classification of galaxies' morphologies. We analyze the impact of the following aspects on the classification accuracy and training efficiency: (i) Training only the last layer vs. training all the network; (ii) Classification with only the CNN vs. considering the hierarchy; (iii) Comparing the models trained with different amounts of data and levels of agreement between the annotators; (iv) Training by stages, transferring knowledge from one model to another; and (v) Combining several models as an ensemble. Results From the experiments, we derive the following results: (i) Training all the layers in the network significantly improves the accuracy (10% increase in exact match), compared to training only the last layer; (ii) There is a tradeoff between the amount of data and the level of agreement between the annotators used for training; (iii) Using the hierarchy can improve accuracy when the amount of training data is reduced; (iv) Training by stages through transfer learning (curriculum learning) produces higher accuracy for limited data; (v) Ensembles can improve accuracy; (vi) Models achieve a low accuracy for the most difficult cases, but, if we consider hierarchical measures, we can derive useful results for upper levels in the hierarchy. An accuracy above 99% is achieved when training all layers of the network and considering a high agreement between the annotators. Conclusions. Training deep learning models from crowd annotations involves additional challenges than learning from hard annotations.
Figures & tables
Figure 1: Hubble’s tuning fork scheme for morphological classification of galaxies.
Figure 2: Agreement vs. probability for two classes.
Figure 3: Distribution of agreements computed with WHA and NMDWA for the annotated instances in Galaxy Zoo 1.
Figure 4: Galaxy hierarchy considered in the experiments for GZ1.
Figure 5: Number of training images for each cumulative range of agreement.
Figure 6: Results in terms of Exact Match for each of the different levels of cumulative agreement and quantity of training data for the CNN and BCNN classifiers, retraining the last layer ( Last ) or all the layers ( All ).
Figure 7: Number of training images for each exclusive range of agreement.
Figure 8: Results in terms of Exact Match for ERA considering the different ranges of agreement and quantity of training data, for the CNN and BCNN classifiers retraining all the layers.
Figure 9: A comparison of training in an cumulative scheme vs. training by ranges with transfer learning, when the evaluation is done with all the test images.
Figure 10: Results in terms of Exact Match for a reduced number of training images with agr=1 for the CNN and BCNN classifiers.
Figure 11: Results in terms of Exact Match for each of the different levels of agreement and quantity of training data when training is done in a cumulative scheme and the evaluation is done with all the test images, CNN retrained in the last or all layers.
Figure 12: Results in terms of Exact Match for the different ranges of agreement and quantity of training data when training is done by exclusive ranges with transfer learning and the evaluation is done with all the test images; the weights of all layers in the CNN are retrained.
Table 1: Confusion matrices for agr=1 , retraining all the layers
Table 2: Confusion matrices for agr≥0.7 with CRA, retraining all the layers
Model
Training
Exact
Training
Type
Range
Match
Scheme
CNN
≥ 0.3
90.91
cumulative
CNN
(0.5,0.4]
90.72
ranges
BCNN
≥ 0.3
90.51
cumulative
CNN
(0.4,0.3]
90.51
ranges
CNN
≥ 0.4
90.21
cumulative
Table 3: Best models in terms of exact match evaluated in the complete test set
Ensemble
EM
Accuracy
hR
hP
hF
3 best
90.86
93.45
94.59
94.31
94.45
5 best
91.04
93.64
94.84
94.46
94.65
Table 4: Results of the ensemble classifier form by the 3 and 5 best models
Figure 13: Training time (in hours), for each range of agreement and quantity of training images for cumulative ranges.
Figure 14: Training time (in hours), for each range of agreement and quantity of training images for exclusive ranges.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Agr.
Set
EL
CW
ACW
EDGE
DK
MG
SPIRAL
GX
1
Tr
2532
1220
1519
600
289
6
3339
5877
Val
633
305
380
150
73
1
835
1469
Te
791
381
475
188
91
2
1044
1837
≥ 0.9
Tr
11027
3401
3933
2414
613
22
9748
20797
Val
2757
850
984
604
154
5
2438
5200
Te
3446
1063
1229
755
193
7
3047
6500
Appendix
Table 5: Number of images associated to each class for each cumulative range. 2 2 2 Notes. MG: Merger, GX: Galaxy, Tr: training, Te: test and Val: validation.
Agr.
Model
EM
Accuracy
hR
hP
hF
1
CNN
79.1
89.16
91.41
91.22
91.32
BCNN
78.99
89.09
91.25
91.26
91.26
≥ 0.9
CNN
83.73
91.55
93.06
93.03
93.04
BCNN
83.36
91.31
92.71
92.99
92.85
≥ 0.8
CNN
89.05
94.24
95.11
95.02
95.06
BCNN
88.93
94.14
94.95
95.02
94.98
Appendix
Table 6: Results of the CNN and BCNN on several evaluation measures for each cumulative range. 3 3 3 Notes. The training is carried out only on the last layer of the CNN.
Agr.
Model
EM
Accuracy
hR
hP
hF
1
CNN
99.01
99.25
99.48
99.31
99.4
BCNN
98.96
99.17
99.46
99.21
99.34
≥ 0.9
CNN
99.67
99.76
99.85
99.77
99.81
BCNN
99.51
99.64
99.71
99.72
99.72
≥ 0.8
CNN
99.55
99.68
99.78
99.7
99.74
BCNN
99.53
99.65
99.72
99.72
99.72
Appendix
Table 7: Results of the CNN and BCNN on several evaluation measures for each cumulative range. 4 4 4 Notes. The training updates the weights of all the layers of the CNN.
Agr.
Training
Test
Validation
Total
1
6166
1928
1542
9636
(1,0.9]
15244
4765
3812
23821
(0.9,0.8]
36503
11408
9126
57037
(0.8,0.7]
50118
15662
12530
78310
(0.7,0.6]
57598
18000
14400
89998
(0.6,0.5]
64964
20302
16241
101507
Appendix
Table 8: Number of images associated to each exclusive range.
Agr.
Model
EM
Accuracy
hR
hP
hF
1
CNN
99.12
99.33
99.52
99.4
99.46
BCNN
99.22
99.43
99.6
99.48
99.54
(1,0.9]
CNN
99.66
99.75
99.84
99.76
99.8
BCNN
99.41
99.56
99.63
99.68
99.65
(0.9,0.8]
CNN
99.7
99.77
99.84
99.78
99.81
BCNN
99.66
99.75
99.76
99.82
99.79
Appendix
Table 9: Results of the CNN and BCNN on several evaluation measures for each exclusive range with transfer learning.
Agr.
Model
EM
Accuracy
hR
hP
hF
1
CNN
60.12
72.73
81.85
73.38
77.38
BCNN
59.33
72.11
81.45
72.85
76.91
≥ 0.9
CNN
64.36
75.77
83.6
76.24
79.75
BCNN
61.75
74.02
82.37
74.99
78.51
≥ 0.8
CNN
68.06
78.28
85.17
78.62
81.77
BCNN
64.84
76.08
83.78
76.77
80.13
Appendix
Table 10: Results of the CNN and BCNN trained on cumulative ranges on several evaluation measures assessed in the full test set. 5 5 5 Notes. The training is carried out only on the last layer of the CNN.
Agr.
Model
EM
Accuracy
hR
hP
hF
1
CNN
68.9
78.24
85.76
78.24
81.82
BCNN
61.28
72.88
82.53
73.3
77.64
≥ 0.9
CNN
74.02
81.57
88.16
81.22
84.55
BCNN
72.02
80.45
86.77
81.25
83.92
≥ 0.8
CNN
79.9
85.84
90.61
85.66
88.07
BCNN
78.42
84.77
89.71
85.04
87.32
Appendix
Table 11: Results of the CNN and BCNN trained on the cumulative ranges on several evaluation measures assessed in the full test set. 6 6 6 Notes. The training updates the weights of all the layers of the CNN.
Agr.
Model
EM
Accuracy
hR
hP
hF
1
CNN
69.14
78.38
85.87
78.37
81.95
BCNN
61.15
72.95
82.62
73.29
77.68
(1,0.9]
CNN
74.86
82.38
88.39
82.18
85.17
BCNN
73.06
81.23
86.91
81.99
84.38
(0.9,0.8]
CNN
82.55
87.74
91.65
87.7
89.63
BCNN
80.81
86.43
90.59
86.84
88.68
Appendix
Table 12: Results of the CNN and BCNN trained by exclusive ranges on several evaluation measures assessed in the full test set. 7 7 7 Notes. The training updates the weights of all the layers of the CNN, and transfer learning is carried out.
Figure 15: Example images in which the ML differs from the crowd for agr=1 . On top the annotations by the crowd and below the ML classification.
Figure 16: Example images in which the ML differs from the crowd for difficult cases, agr≤0.4 . On top the annotations by the crowd and below the ML classification.
Department of Computer Science Stony Brook University Stony Brook, New York, USA · School of Computer Science University of Technology Sydney Sydney, New South Wales, Australia · Futurewei Technologies San Jose, California, USA
Data Science Program, College of Science, University of the Philippines - Diliman, Philippines · Department of Computer Science, College of Engineering, University of the Philippines - Diliman, Philippines · National Institute of Physics, College of Science, University of the Philippines - Diliman, Philippines
The Hong Kong University of Science and Technology, Hong Kong, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China