Operational evaluation of data-driven forest fire forecasting models
Authors: Shahbaz Alvi, Italo Epicoco, Jose Maria Costa Saura
Organizations: CMCC Foundation - Euro-Mediterranean Center on Climate Change, Italy · Department of Engineering for Innovation, University of Salento, Italy
A growing body of literature has focused on predicting wildfire occurrence using machine learning methods, capitalizing on high-resolution data and fire predictors that canonical process-based frameworks largely ignore. Standard evaluation metrics for an ML classifier, while important, provide a potentially limited measure of the model's operational performance for the Fire Danger Index (FDI) forecast. Furthermore, model evaluation is frequently conducted without adequately accounting for false positive rates, despite their critical relevance in operational contexts. In this paper, we revisit the daily FDI model evaluation paradigm and propose a novel method for evaluating a forest fire forecasting model that is aligned with real-world decision-making. Furthermore, we systematically assess performance in accurately predicting fire activity and the false positives (false alarms). We further demonstrate that an ensemble of ML models improves both fire identification and reduces false positives.
Figures & tables
Figure 1: Architecture of the CNN considered in this manuscript. The deeper variations of the vanilla CNN differ in having more Convolutional blocks in the network.
Figure 2: Architecture of the ConvLSTM considered in this manuscript. The architecture of the network is identical to that used and provided by ( Kondylatos et al., 2022 ) .
No.
Variable
Units
1
Normalized Difference Vegetation Index (NDVI)
-
2-3
Day & night Land Surface Temperature (LST)
K
4
Daily maximum dew point temperature at 2-meters
K
5
Daily maximum air temperature at 2-meters
K
6
Daily maximum surface pressure at sea level
Pa
7
Daily maximum total precipitation
meters
Table 1: List of fire predictors used to forecast forest fire probability. More details on the remapping of original CLC classes can be found in Kondylatos et al., 2022 .
Figure 3: (Left) Figure shows the format of a single sample for the CNN network, which is a 3D tensor. (Right) A single sample of the ConvLSTM network also includes a time dimension of length 10, and it is a 4D tensor.
Figure 4: Plot showing the distribution of the fires in the entire dataset by their CLC class. On the x-axis are the CLC labels, and on the y-axis is the frequency of fire activity with that CLC class. The orange colored bars represent the distribution of the fire samples used for model training.
Metric/ Model
Basic CNN
Deeper CNN 1
Deeper CNN 2
ConvLSTM
Recall
0.89
0.90
0.90
0.87
Precision
0.77
0.77
0.78
0.73
F1-score
0.82
0.83
0.83
0.80
Table 2: The table reports the statistics computed on the samples from year 2020 and 2021, which were reserved for testing using the model configuration corresponding to the lowest validation loss.
Q-Quantiles /Model
Basic CNN
Deeper CNN 1
Deeper CNN 2
ConvLSTM
40%
0.74
0.74
0.75
0.77
50%
0.82
0.83
0.84
0.92
60%
0.92
0.92
0.92
0.99
70%
0.97
0.98
0.96
1.00
80%
1.00
1.00
1.00
1.00
90%
1.00
1.00
1.00
1.00
Table 3: Q-quantiles of daily recall (the fraction of fire pixels on a given day assigned fire probability > 0.5) for days with fire computed from full-map inference in fire seasons in 2020 and 2021 - a total of 103 days.
Figure 5: The distribution of the fire probability occurrence on no-fire days randomly selected. A positively skewed distribution indicates fewer false positives on that day compared to a negatively skewed distribution. ConvLSTM generally has a more positively skewed distribution than the CNN architectures. The right-most column shows the distribution of Canadian FWI on the same day, normalized by 70, which is classified as an extreme danger class.
Figure 6: Region-wide inference for the no-fire days randomly selected (the same days as in Figure 5 ). Similarities between the maps indicate that each model is a noisy estimator of the underlying true fire probability; however, the ConvLSTM identifies fire-safe conditions far more confidently than the CNN variants (fire probabilities are far from the decision threshold of 0.5.
Figure 7: Comparison between fire probability maps from ConvLSTM (left column) and Canadian Fire Weather Index (right column), provided in the dataset ( Prapas et al., 2022 ) dataset, on the four random days. In both plots, recorded fire events are represented by a danger icon.
Figure 8: Daily recall computed on each day by each ensemble member is shown as a heatmap. The daily recall computed from the ensemble-averaged map is shown in the 8th column with the label “Average”.
Figure 9: The probability distribution for six days. Since August 2021, there have been very few days with no fires; we also include some days in August 2020 in the analysis. The top plot (in orange) shows the distribution of the ensemble-averaged map.
Max Planck Institute for Biogeochemistry, Jena, Germany · Orion Lab, National Technical University of Athens · University of Utah, Salt Lake City, USA +4