Current solar flare predictions often lack precise quantification of their reliability, resulting in frequent false alarms, particularly when dealing with datasets skewed towards extreme events. To improve the trustworthiness of space weather forecasting, it is crucial to establish confidence intervals for model predictions. Conformal prediction, a machine learning framework, presents a promising avenue for this purpose by constructing prediction intervals that ensure valid coverage in finite samples without making assumptions about the underlying data distribution. In this study, we explore the application of conformal prediction to regression tasks in space weather forecasting. Specifically, we implement full-disk solar flare prediction using images created from magnetic field maps and adapt four pre-trained deep learning models to incorporate three distinct methods for constructing confidence intervals: conformal prediction, quantile regression, and conformalized quantile regression. Our experiments demonstrate that conformalized quantile regression achieves higher coverage rates and more favorable average interval lengths compared to alternative methods, underscoring its effectiveness in enhancing the reliability of solar weather forecasting models.
Figures & tables
Fig. 1 : Distribution of normalized peak X-ray flux values (target values) across different partition.
Fig. 2 : Overview of the process for conformal prediction.
Optimal Parameters
Search Space
Confidence Level
AlexNet
MobileNet
InceptionV3
ResNet50
Learning Rate
10−7 to 10−5
95%
2.5×10−7
10−6
5×10−7
5×10−5
90%
5×10−6
5×10−6
10−6
10−5
85%
10−6
6×10−8
5×10−7
10−5
80%
5×10−6
10−6
10−6
10−5
Weight Decay
0, 10−5 , 10−4
95%
10−5
10−4
0
10−4
TABLE I: Optimal parameters for each model with each confidence level
Model
Average
Standard Deviation
AlexNet
0.473
0.040
MobileNet
0.454
0.037
InceptionV3
0.458
0.028
ResNet50
0.496
0.017
TABLE II : The R-squared results of the models, with each learning algorithm trained using four different significance levels ( α ). Among the models, ResNet50 demonstrates the highest average R-squared score.
Avg. Length
Avg. Coverage
ICC
CP
AlexNet
2.39
87.3
0.530
MobileNet
2.35
86.9
0.516
InceptionV3
2.49
84.6
0.380
ResNet50
2.33
86.1
0.478
QR
AlexNet
2.38
85.8
0.454
MobileNet
2.08
82.6
0.336
TABLE III : The results of average prediction lengths, marginal coverages, and ICC across different models and uncertainty quantification methods are presented. In this table, we report only the results for models with a significance level of 0.1.
Fig. 3 : This figure presents the evaluation of prediction intervals from MobileNet. In this result, a 90% confidence level and four different target bins are used to analyze three uncertainty quantification methods: CP, QR, and CQR. While QR and CQR produce narrower prediction intervals, they exhibit lower marginal coverage compared to CP. However, the prediction interval lengths from CP remain constant across all bins.
Fig. 4 : Comparison of prediction intervals generated by CP and CQR over an eight-year period.
Confidence
80%
85%
90%
95%
CP
1.78
1.92
2.35
2.98
Length
QR
1.36
1.83
2.08
2.34
CQR
1.60
1.87
2.19
2.70
CP
76.5
78.6
86.9
90.9
Coverage
QR
67.1
75.9
82.6
85.4
CQR
74.3
77.2
84.6
89.9
TABLE IV : The result of MobileNet with four different confidence levels.
B. R. Ambedkar National Institute of Technology, Jalandhar, India · Department of Computer Science and Engineering, Dr. B.R. Ambedkar National Institute of Technology Jalandhar, India