Organizations: School of Computer Science, Jiangsu University of Science and Technology, Zhenjiang 212100, People′s Republic of China · State Key Laboratory of Space Weather, Chinese Academy of Sciences, Beijing 100190, People′s Republic of China · School of software, Southeast University, Nanjing, People′s Republic of China · Yunnan Astronomical Observatory, Chinese Academy of Sciences, Kunming 650216, People′s Republic of China · Radio Cosmology Lab, Centre for Astronomy and Astrophysics Research, Department of Physics, Faculty of Science, Universiti Malaya, 50603 Kuala Lumpur, Malaysia · National Centre for Particle Physics, Universiti Malaya, 50603 Kuala Lumpur, Malaysia · Department of Electrical Engineering, Faculty of Engineering, Universiti Malaya, 50603 Kuala Lumpur, Malaysia
In this work, we develop, for the first time, a supervised classification framework with class-dependent rewards (CDR) to predict ≥MM flares within 24 hr. We construct multiple datasets, covering knowledge-informed features and line-of sight (LOS) magnetograms. We also apply three deep learning models (CNN, CNN-BiLSTM, and Transformer) and three CDR counterparts (CDR-CNN, CDR-CNN-BiLSTM, and CDR-Transformer). First, we analyze the importance of LOS magnetic field parameters with the Transformer, then compare its performance using LOS-only, vector-only, and combined magnetic field parameters. Second, we compare flare prediction performance based on CDR models versus deep learning counterparts. Third, we perform sensitivity analysis on reward engineering for CDR models. Fourth, we use the SHAP method for model interpretability. Finally, we conduct performance comparison between our models and NASA/CCMC. The main findings are: (1)Among LOS feature combinations, R_VALUE and AREA_ACR consistently yield the best results. (2)Transformer achieves better performance with combined LOS and vector magnetic field data than with either alone. (3)Models using knowledge-informed features outperform those using magnetograms. (4)While CNN and CNN-BiLSTM outperform their CDR counterparts on magnetograms, CDR-Transformer is slightly superior to its deep learning counterpart when using knowledge-informed features. Among all models, CDR-Transformer achieves the best performance. (5)The predictive performance of the CDR models is not overly sensitive to the reward choices.(6)Through SHAP analysis, the CDR model tends to regard TOTUSJH as more important, while the Transformer tends to prioritize R_VALUE more.(7)Under identical prediction time and active region (AR) number, the CDR-Transformer shows superior predictive capabilities compared to NASA/CCMC.
Figures & tables
Group name of knowledge-informed features
Keyword
Description
Reference
R_VALUE & AREA_ACR
R_VALUE
Sum of flux near polarity inversion line
Schrijver (2007)
AREA_ACR
Area of strong field pixels in the AR
Bobra & Couvidat (2015)
Gradient features
Gradient mean
To condense the gradient information into single descriptors (features) for each image, the gradient features are computed as the (1) mean, (2) standard deviation, (3) maximum, (4) minimum, (5) median, (6) skewness, and (7) kurtosis of the gradients in each image.
Gradient std
Gradient median
Gradient min
Table 1: 31 knowledge-informed features calculated from SHARP LOS magnetogram images.
Dataset
Training(No-flare/C/M/X)
Validation(No-flare/C/M/X)
Testing(No-flare/C/M/X)
No.1
223/168/46/8
70/37/11/1
58/33/16/4
No.2
223/168/46/8
64/38/12/1
63/37/16/3
No.3
223/168/46/8
67/37/10/1
59/30/14/4
No.4
223/168/46/8
66/37/8/2
53/29/17/5
No.5
223/168/46/8
62/37/10/2
56/30/13/4
No.6
223/168/46/8
64/33/9/1
50/30/17/3
Table 3: The distribution of the number of ARs from the 10 CV datasets used in this study, including magnetogram images and knowledge-informed features. The “No-flare /C/M/X” in the table represents the number of ARs corresponding to their respective categories among the training, validation, and testing sets.
Parameter name
CNN
CNN-BiLSTM
Transformer
Batch size
16
16
10
Epochs
120
120
150
Loss function
cross-entropy
cross-entropy
cross-entropy
Optimizer
SGD
SGD
Adam
Learning rate
0.01
0.01
0.0001
Table 4: The parameter configuration of deep learning models.
Feature
Model
Metric
TSS
BSS
The combination of LOS and vector magnetic field parameters
Transformer
0.814±0.029
0.452±0.095
8 vector magnetic field parameters
0.719±0.090
0.403±0.098
2 LOS magnetic field parameters
0.697±0.137
0.448±0.079
Table 7: The TSS and BSS scores for ≥ M-class flare prediction using the Transformer on the testing set with three distinct combinations of magnetic field parameters.
Models
p-value
TSS
BSS
CDR-Transformer-10 and Transformer-10
0.262
0.094
CDR-Transformer-2 and Transformer-2
0.366
0.431
Table 9: The paired t-test results between the CDR-Transformer and Transformer models.
TP reward
TN reward
FP reward
FN reward
TSS
BSS
5
4
-20
-15
0.803±0.048
0.461±0.172
6
4
-20
-15
0.811±0.062
0.467±0.160
7
4
-20
-15
0.800±0.064
0.475±0.148
8
4
-20
-15
0.785±0.076
0.481±0.136
9
4
-20
-15
0.801±0.082
0.482±0.136
10
4
-20
-15
0.829±0.059
0.489±0.120
Table 10: The effects of applying a perturbation (step size=1) to the TP reward on the TSS and BSS scores of the CDR-Transformer-10 model.
TP reward
TN reward
FP reward
FN reward
TSS
BSS
10
1
-20
-15
0.797±0.067
0.492±0.119
10
2
-20
-15
0.826±0.061
0.492±0.117
10
3
-20
-15
0.815±0.062
0.489±0.119
10
4
-20
-15
0.829±0.059
0.489±0.120
10
5
-20
-15
0.826±0.063
0.488±0.121
10
6
-20
-15
0.821±0.070
0.486±0.123
Table 11: The effects of applying a perturbation (step size=1) to the TN reward on the TSS and BSS scores of the CDR-Transformer-10 model.
TP reward
TN reward
FP reward
FN reward
TSS
BSS
10
4
-15
-15
0.807±0.060
0.497±0.106
10
4
-16
-15
0.813±0.061
0.492±0.101
10
4
-17
-15
0.820±0.065
0.495±0.112
10
4
-18
-15
0.825±0.066
0.494±0.110
10
4
-19
-15
0.819±0.072
0.491±0.114
10
4
-20
-15
0.829±0.059
0.489±0.120
Table 12: The effects of applying a perturbation (step size=1) to the FP reward on the TSS and BSS scores of the CDR-Transformer-10 model.
TP reward
TN reward
FP reward
FN reward
TSS
BSS
10
4
-20
-10
0.801±0.072
0.498±0.096
10
4
-20
-11
0.819±0.051
0.483±0.122
10
4
-20
-12
0.807±0.056
0.493±0.110
10
4
-20
-13
0.813±0.061
0.485±0.107
10
4
-20
-14
0.813±0.070
0.486±0.113
10
4
-20
-15
0.829±0.059
0.489±0.120
Table 13: The effects of applying a perturbation (step size=1) to the FN reward on the TSS and BSS scores of the CDR-Transformer-10 model.
Data
Model
TSS
Recall
FPR
Filtered testing dataset
NASA/CCMC
0.853
1
0.147
iTransformer
0.906 ± 0.043
1.000 ± 0.000
0.094 ± 0.043
Transformer
0.933 ± 0.057
0.980 ± 0.060
0.047 ± 0.020
CDR-Transformer
0.968 ± 0.016
1.000 ± 0.000
0.032 ± 0.016
Original testing dataset
NASA/CCMC
0.633
0.706
0.072
iTransformer
0.668 ± 0.060
1.000 ± 0.000
0.332 ± 0.060
Table 14: The performance results for CDR-Transformer-10, Transformer-10, iTransformer, and NASA/CCMC at the probability thresholds corresponding to their respective optimal TSS.
Figure 4: A bar chart depicting the global significance of the 10 features for Transformer-10 on one testing set from 10 CV datasets.
Figure 5: Beeswarm plot showing the effect of the 10 features on Transformer-10 for each AR.
Figure 6: The waterfall plot for the correct prediction of a positive class on Transformer-10 for AR12257. The AR12257 produced an M-class flare at 04:13 UTC on January 13, 2015.
Figure 7: The waterfall plot for the correct prediction of a negative class on Transformer-10 for AR12367. The AR12367 produced a C-class flare at 19:07 UTC on June 20, 2015.
Apr 11, 2026·Jialiang Li, Vasyl Yurchyshyn, Jason T. L. Wang +6Solar Physics
Institute for Space Weather Sciences, New Jersey Institute of Technology, University Heights, Newark, NJ 07102, USA · Department of Computer Science, New Jersey Institute of Technology, University Heights, Newark, NJ 07102, USA · Big Bear Solar Observatory, New Jersey Institute of Technology, 40386 North Shore Lane, Big Bear City, CA 92314, USA +4