Deep learning from the crowd Fundamentals of morphological galaxy classification
Authors: Luis Enrique Sucar, Carlos del Burgo, Jonathan Serrano-Pérez
Organizations: Instituto de Astrofísica de Canarias, Vía Láctea S/N, La Laguna 38200, Tenerife, Spain · Instituto Nacional de Astrofísica, Óptica y Electrónica, Luis Enrique Erro #1, CP 72840, Tonantzintla, Puebla, Mexico · Departamento de Astrofísica, Universidad de la Laguna, La Laguna 38200, Tenerife, Spain · Facultad de Ciencias Básicas, Ing. y Tec., Universidad Autónoma de Tlaxcala, Carretera Apizaquito S/N, CP 90401, San Luis Apizaquito, Tlaxcala, Mexico
Aims. The objective of this work is to adapt a deep neural network model to perform galaxy morphological classification trained from crowd annotations, considering the training scheme, the agreement between the annotators, and the hierarchy. Methods. We use Galaxy Zoo 1 as our experimental testbed and trained a convolutional neural network (CNN) for the automatic classification of galaxies' morphologies. We analyze the impact of the following aspects on the classification accuracy and training efficiency: (i) Training only the last layer vs. training all the network; (ii) Classification with only the CNN vs. considering the hierarchy; (iii) Comparing the models trained with different amounts of data and levels of agreement between the annotators; (iv) Training by stages, transferring knowledge from one model to another; and (v) Combining several models as an ensemble. Results From the experiments, we derive the following results: (i) Training all the layers in the network significantly improves the accuracy (10% increase in exact match), compared to training only the last layer; (ii) There is a tradeoff between the amount of data and the level of agreement between the annotators used for training; (iii) Using the hierarchy can improve accuracy when the amount of training data is reduced; (iv) Training by stages through transfer learning (curriculum learning) produces higher accuracy for limited data; (v) Ensembles can improve accuracy; (vi) Models achieve a low accuracy for the most difficult cases, but, if we consider hierarchical measures, we can derive useful results for upper levels in the hierarchy. An accuracy above 99% is achieved when training all layers of the network and considering a high agreement between the annotators. Conclusions. Training deep learning models from crowd annotations involves additional challenges than learning from hard annotations.
Figures & tables
Figure 1: Hubble’s tuning fork scheme for morphological classification of galaxies.
Figure 2: Agreement vs. probability for two classes.
Figure 3: Distribution of agreements computed with WHA and NMDWA for the annotated instances in Galaxy Zoo 1.
Figure 4: Galaxy hierarchy considered in the experiments for GZ1.
Figure 5: Number of training images for each cumulative range of agreement.
Figure 6: Results in terms of Exact Match for each of the different levels of cumulative agreement and quantity of training data for the CNN and BCNN classifiers, retraining the last layer ( Last ) or all the layers ( All ).
Figure 7: Number of training images for each exclusive range of agreement.
Figure 8: Results in terms of Exact Match for ERA considering the different ranges of agreement and quantity of training data, for the CNN and BCNN classifiers retraining all the layers.
Figure 9: A comparison of training in an cumulative scheme vs. training by ranges with transfer learning, when the evaluation is done with all the test images.
Figure 10: Results in terms of Exact Match for a reduced number of training images with agr=1 for the CNN and BCNN classifiers.
Figure 11: Results in terms of Exact Match for each of the different levels of agreement and quantity of training data when training is done in a cumulative scheme and the evaluation is done with all the test images, CNN retrained in the last or all layers.
Figure 12: Results in terms of Exact Match for the different ranges of agreement and quantity of training data when training is done by exclusive ranges with transfer learning and the evaluation is done with all the test images; the weights of all layers in the CNN are retrained.
Table 1: Confusion matrices for agr=1 , retraining all the layers
Table 2: Confusion matrices for agr≥0.7 with CRA, retraining all the layers
Model
Training
Exact
Training
Type
Range
Match
Scheme
CNN
≥ 0.3
90.91
cumulative
CNN
(0.5,0.4]
90.72
ranges
BCNN
≥ 0.3
90.51
cumulative
CNN
(0.4,0.3]
90.51
ranges
CNN
≥ 0.4
90.21
cumulative
Table 3: Best models in terms of exact match evaluated in the complete test set
Ensemble
EM
Accuracy
hR
hP
hF
3 best
90.86
93.45
94.59
94.31
94.45
5 best
91.04
93.64
94.84
94.46
94.65
Table 4: Results of the ensemble classifier form by the 3 and 5 best models
Figure 13: Training time (in hours), for each range of agreement and quantity of training images for cumulative ranges.
Figure 14: Training time (in hours), for each range of agreement and quantity of training images for exclusive ranges.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Agr.
Set
EL
CW
ACW
EDGE
DK
MG
SPIRAL
GX
1
Tr
2532
1220
1519
600
289
6
3339
5877
Val
633
305
380
150
73
1
835
1469
Te
791
381
475
188
91
2
1044
1837
≥ 0.9
Tr
11027
3401
3933
2414
613
22
9748
20797
Val
2757
850
984
604
154
5
2438
5200
Te
3446
1063
1229
755
193
7
3047
6500
Appendix
Table 5: Number of images associated to each class for each cumulative range. 2 2 2 Notes. MG: Merger, GX: Galaxy, Tr: training, Te: test and Val: validation.
Agr.
Model
EM
Accuracy
hR
hP
hF
1
CNN
79.1
89.16
91.41
91.22
91.32
BCNN
78.99
89.09
91.25
91.26
91.26
≥ 0.9
CNN
83.73
91.55
93.06
93.03
93.04
BCNN
83.36
91.31
92.71
92.99
92.85
≥ 0.8
CNN
89.05
94.24
95.11
95.02
95.06
BCNN
88.93
94.14
94.95
95.02
94.98
Appendix
Table 6: Results of the CNN and BCNN on several evaluation measures for each cumulative range. 3 3 3 Notes. The training is carried out only on the last layer of the CNN.
Agr.
Model
EM
Accuracy
hR
hP
hF
1
CNN
99.01
99.25
99.48
99.31
99.4
BCNN
98.96
99.17
99.46
99.21
99.34
≥ 0.9
CNN
99.67
99.76
99.85
99.77
99.81
BCNN
99.51
99.64
99.71
99.72
99.72
≥ 0.8
CNN
99.55
99.68
99.78
99.7
99.74
BCNN
99.53
99.65
99.72
99.72
99.72
Appendix
Table 7: Results of the CNN and BCNN on several evaluation measures for each cumulative range. 4 4 4 Notes. The training updates the weights of all the layers of the CNN.
Agr.
Training
Test
Validation
Total
1
6166
1928
1542
9636
(1,0.9]
15244
4765
3812
23821
(0.9,0.8]
36503
11408
9126
57037
(0.8,0.7]
50118
15662
12530
78310
(0.7,0.6]
57598
18000
14400
89998
(0.6,0.5]
64964
20302
16241
101507
Appendix
Table 8: Number of images associated to each exclusive range.
Agr.
Model
EM
Accuracy
hR
hP
hF
1
CNN
99.12
99.33
99.52
99.4
99.46
BCNN
99.22
99.43
99.6
99.48
99.54
(1,0.9]
CNN
99.66
99.75
99.84
99.76
99.8
BCNN
99.41
99.56
99.63
99.68
99.65
(0.9,0.8]
CNN
99.7
99.77
99.84
99.78
99.81
BCNN
99.66
99.75
99.76
99.82
99.79
Appendix
Table 9: Results of the CNN and BCNN on several evaluation measures for each exclusive range with transfer learning.
Agr.
Model
EM
Accuracy
hR
hP
hF
1
CNN
60.12
72.73
81.85
73.38
77.38
BCNN
59.33
72.11
81.45
72.85
76.91
≥ 0.9
CNN
64.36
75.77
83.6
76.24
79.75
BCNN
61.75
74.02
82.37
74.99
78.51
≥ 0.8
CNN
68.06
78.28
85.17
78.62
81.77
BCNN
64.84
76.08
83.78
76.77
80.13
Appendix
Table 10: Results of the CNN and BCNN trained on cumulative ranges on several evaluation measures assessed in the full test set. 5 5 5 Notes. The training is carried out only on the last layer of the CNN.
Agr.
Model
EM
Accuracy
hR
hP
hF
1
CNN
68.9
78.24
85.76
78.24
81.82
BCNN
61.28
72.88
82.53
73.3
77.64
≥ 0.9
CNN
74.02
81.57
88.16
81.22
84.55
BCNN
72.02
80.45
86.77
81.25
83.92
≥ 0.8
CNN
79.9
85.84
90.61
85.66
88.07
BCNN
78.42
84.77
89.71
85.04
87.32
Appendix
Table 11: Results of the CNN and BCNN trained on the cumulative ranges on several evaluation measures assessed in the full test set. 6 6 6 Notes. The training updates the weights of all the layers of the CNN.
Agr.
Model
EM
Accuracy
hR
hP
hF
1
CNN
69.14
78.38
85.87
78.37
81.95
BCNN
61.15
72.95
82.62
73.29
77.68
(1,0.9]
CNN
74.86
82.38
88.39
82.18
85.17
BCNN
73.06
81.23
86.91
81.99
84.38
(0.9,0.8]
CNN
82.55
87.74
91.65
87.7
89.63
BCNN
80.81
86.43
90.59
86.84
88.68
Appendix
Table 12: Results of the CNN and BCNN trained by exclusive ranges on several evaluation measures assessed in the full test set. 7 7 7 Notes. The training updates the weights of all the layers of the CNN, and transfer learning is carried out.
Figure 15: Example images in which the ML differs from the crowd for agr=1 . On top the annotations by the crowd and below the ML classification.
Figure 16: Example images in which the ML differs from the crowd for difficult cases, agr≤0.4 . On top the annotations by the crowd and below the ML classification.
Existing astronomy foundation models provide strong galaxy representations, but adapting them to new survey conditions and survey-specific morphology recognition tasks still requires substantial human supervision. We show that VLM-based VQA systems contain meaningful visual-semantic priors that can serve as weak supervision for downstream morphology classifiers and improve morphology classification under limited human-label budgets. We first introduce a survey-oriented VQA benchmark spanning two representative imaging regimes and evaluate state-of-the-art VLMs on galaxy morphology questions. The results show that these models capture useful morphology signals and informative uncertainty, but are not sufficiently reliable to replace human annotators. Motivated by this finding, we use a general-purpose VLM as a morphology teacher for Zoobot, an astronomy foundation model pretrained on large-scale Galaxy Zoo annotations. Across two survey domains and multiple annotation budgets, the VLM teacher consistently improves Zoobot's downstream morphology classification. These results demonstrate that a general-purpose VLM provides knowledge complementary to an astronomy foundation model and can teach it to better recognize galaxy morphology under limited human supervision. The resulting pipeline is designed for label-efficient adaptation to forthcoming large-scale surveys, including the Vera C. Rubin Observatory's Legacy Survey of Space and Time (LSST) and the Nancy Grace Roman Space Telescope. The benchmark and code are publicly available at https://github.com/fw-ic/VLM-morphology-teacher.
Dichang Zhang, Jiaqi Deng, Yixuan Shao +7
Department of Computer Science Stony Brook University Stony Brook, New York, USA · School of Computer Science University of Technology Sydney Sydney, New South Wales, Australia · Futurewei Technologies San Jose, California, USA
Image data regarding galactic morphology is expected to increase both in quantity and quality for the next foreseeable years; thus it is important to explore which deep learning architectures adapted for image classification tasks are cost-effective. Residual and Inception networks are ideal for exploring classification convolutional neural networks (CNNs) due to their computational efficiency, achieved through techniques such as residual connections and parallelized inception modules, enabling deeper networks without excessively increasing computational complexity. In this work, we analyze the performance of ResNet101 and InceptionV4 on a spatially-augmented Galaxy10 DECals dataset. Retaining the ten-class classification of galaxies, we modify the image count of each class. We find that ResNet101 and InceptionV4 models achieved accuracies of ∼ 90%, comparable with reported performance in the literature. In terms of performance metrics, ResNet101 is superior to InceptionV4. Our results indicate that either of these CNN architectures could serve as a robust foundation for specialized pipelines for classification of galaxy images from upcoming surveys.
Lanz Anthonee A. Lagman, Prospero C. Naval, Reinabelle C. Reyes
Data Science Program, College of Science, University of the Philippines - Diliman, Philippines · Department of Computer Science, College of Engineering, University of the Philippines - Diliman, Philippines · National Institute of Physics, College of Science, University of the Philippines - Diliman, Philippines
Astronomers classify galaxy morphology to investigate cosmic evolution. While deep foundation models are increasingly utilized in Galaxy Morphology Classification (GMC), little work has been done on evaluating the uncertainty of GMC results. Uncertainty evaluation is important because astronomical data are inherently noisy due to instrumental and environmental limitations. Also, the continuous evolution of galaxies creates intrinsic morphological ambiguity. However, current foundation models operate as deterministic point estimators, failing to quantify the uncertainty. To overcome this limitation, we propose UEGMC, a post-hoc framework of Uncertainty Estimation for Galaxy Morphology Classification. It categorizes uncertainty in GMC into distinct types by model parameters, astronomical data, reference standards, or intrinsic physical ambiguities, thereby facilitating better classification. Our framework can directly predict uncertainties from representations extracted from the frozen backbones of foundation models, without computationally expensive sampling, therefore enabling fine-grained uncertainty evaluations. Our experimental results demonstrate that UEGMC provides competitive uncertainty quantification performance compared with previous methods.
Kai Cheng, Ruoqi Wang, Qiong Luo
The Hong Kong University of Science and Technology, Hong Kong, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China