WildfireSpreadBench: The Metric Decides the Model in Wildfire Spread Prediction

Arin Gopakumar; Marco Pannozzo

cs.LGAug 29, 2026

WildfireSpreadBench: The Metric Decides the Model in Wildfire Spread Prediction

Authors: Arin Gopakumar, Marco Pannozzo

Organizations: UC Berkeley · Purdue University

Abstract

Machine learning is being increasingly used to predict where active wildfires will burn the following day, helping inform evacuation boundaries and containment lines. Most models are evaluated using Average Precision (AP), which summarizes performance across all decision thresholds, although acting on a forecast requires choosing one. We benchmarked five discriminative architectures and one generative model on WildfireSpreadTS using a shared evaluation pipeline and two input configurations. We found that model rankings varied depending on whether performance was measured by AP or by threshold-dependent metrics like F1 and IoU. At the default 0.5 threshold, the highest-AP model flagged 4 to 5 times the observed active-fire area and ranked fifth of six on F1 and IoU, and the most recall-heavy model flagged 16 to 23 times. The three models whose flagged area stayed close to the observed area had AP scores 24 to 37% lower. Across models, we identified three distinct prediction profiles: over-predicting, balanced, and under-predicting, which AP alone could not distinguish. Expanding the input from 7 to 23 channels changed AP by 0.03 on average, against a 0.21 to 0.24 spread across models. Ultimately, these results show that AP alone can favor models whose predictions are poorly suited for operational wildfire forecasting.

Figures & tables

	AP		F1		IoU		P		R
Model	Veg	All	Veg	All	Veg	All	Veg	All	Veg	All
BCE U-Net	0.562	0.529	0.308	0.329	0.182	0.197	0.186	0.203	0.896	0.861
UTAE	0.383	0.322	0.115	0.080	0.061	0.042	0.061	0.042	0.969	0.964
ResNet18 U-Net	0.375	0.401	0.568	0.585	0.396	0.413	0.558	0.599	0.578	0.572
ConvLSTM	0.403	0.390	0.583	0.570	0.411	0.398	0.565	0.543	0.602	0.599
Logistic Regression	0.352	0.371	0.548	0.566	0.378	0.394	0.501	0.551	0.606	0.581

Table 1: The models predict the wildfire spread 24h ahead based on two input feature sets: Vegetation and All . The performance displayed is the test set AP, F1, IoU, Precision, and Recall for the 2021 test year, with threshold metrics computed at 0.5. Rows are grouped by operating-point profile, and bold marks the best.

Figure 1: (a) Model rank under AP against rank under F1, for both feature sets; bolded lines mark ranks that move three places or more. (b) The same twelve runs as precision–recall operating points, with iso-F1 contours.

Figure 2: Active fire on day $t$ , the observed fire on day $t{+}1$ , and the ConvLSTM and BCE U-Net predictions at a 0.9 threshold, with the observed footprint outlined on both prediction panels. BCE U-Net covers 2.1 $\times$ the observed area and merges the observed patches into roughly a dozen blobs; ConvLSTM covers 0.9 $\times$ .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Group	Source product	Channels	In Vegetation
Active fire	VIIRS VNP14IMG	1	Yes
Surface reflectance	VIIRS VNP09GA	3	Yes
Vegetation indices	VIIRS VNP13A1	2	Yes
Weather	GRIDMET	6	No
ERC and drought	GRIDMET	2	No
Weather forecast	GFS	5	No

Appendix

Table 2: The 23 released channels by group, following the feature groupings used in the ablation studies of Gerard et al. [2023b] , and which groups the Vegetation configuration draws on.

Model	Family	Days	Params	Loss	LR
Logistic Regression	Discriminative	1	361	Dice	0.1
ResNet18 U-Net	Discriminative	1	14.4M	Dice	0.001
BCE U-Net	Discriminative	1	32.5M	Weighted BCE	$2\times 10^{-4}$
ConvLSTM	Discriminative	5	240K	Jaccard	0.01
UTAE	Discriminative	5	1.1M	Weighted BCE	0.01
Flow Matching	Generative	1	94.5M	MSE on velocity	$5\times 10^{-5}$

Appendix

Table 3: Model and training configuration. Day

t

is the most recent observed day of a fire, and every model predicts the active fire mask on day

t{+}1

. Parameter counts are for the All configuration. The four dataset baselines use the configurations released with the benchmark.

Model	Vegetation	All
BCE U-Net	4.82	4.24
UTAE	15.88	22.95
ResNet18 U-Net	1.04	0.96
ConvLSTM	1.07	1.10
Logistic Regression	1.21	1.05
Flow Matching	0.82	0.84

Appendix

Table 4: Predicted active-fire area divided by observed active-fire area, equal to

R/P

, at threshold 0.5 over the 2021 test year. A value of 1 matches the observed extent, above 1 over-predicts and below 1 under-predicts.

	Vegetation		All
Model	Published	Ours	Published	Ours
Logistic Regression	0.279 $\pm$ 0.092	0.352	0.286 $\pm$ 0.092	0.371
ResNet18 U-Net	0.328 $\pm$ 0.090	0.375	0.341 $\pm$ 0.086	0.401
ConvLSTM	0.306 $\pm$ 0.082	0.403	0.292 $\pm$ 0.094	0.390
UTAE	0.372 $\pm$ 0.088	0.383	0.321 $\pm$ 0.135	0.322

Appendix

Table 5: AP for the four architectures shared with Gerard et al. [2023b] . Published values are means over the full twelve-fold cross-validation with the standard deviation across folds. Ours are fold 0, with 2021 held out.

Explore similar work

Conformal Risk Control for Safety-Critical Wildfire Evacuation Mapping: A Comparative Study of Tabular, Spatial, and Graph-Based Models
Mar 20, 2026Baljinnyam DayanWildfire Forecasting Conformal Risk Control
Imperial College London, London, United Kingdom.
Operational evaluation of data-driven forest fire forecasting models
Mar 26, 2026Shahbaz Alvi, Italo Epicoco, Jose Maria Costa SauraWildfire Forecasting
CMCC Foundation - Euro-Mediterranean Center on Climate Change, Italy · Department of Engineering for Innovation, University of Salento, Italy
Set Prediction for Next-Day Active Fire Forecasting
May 11, 2026Yuchen Bai, Georgios Athanasiou, Xin Yu +4Wildfire Forecasting Set Prediction
Max Planck Institute for Biogeochemistry, Jena, Germany · Orion Lab, National Technical University of Athens · University of Utah, Salt Lake City, USA +4

Mar 20, 2026J/K move · Enter open · S save

Conformal Risk Control for Safety-Critical Wildfire Evacuation Mapping: A Comparative Study of Tabular, Spatial, and Graph-Based Models

Baljinnyam Dayan

Imperial College London, London, United Kingdom.

Wildfire Forecasting Conformal Risk Control Disaster Response

Open paper AskPDF arXiv

Every wildfire prediction model deployed today shares a dangerous property: none of these methods provides formal guarantees on how much fire spread is missed. Despite extensive work on wildfire spread prediction using deep learning, no prior study has applied distribution-free safety guarantees to this domain, leaving evacuation planners reliant on probability thresholds with no formal assurance. We address this gap by presenting, to our knowledge, the first application of conformal risk control (CRC) to wildfire spread prediction, providing finite-sample guarantees on false negative rate (FNR <= 0.05). We expose a stark failure: across three model families of increasing complexity (tabular: LightGBM, AUROC 0.854; convolutional: Tiny U-Net, AUROC 0.969; and graph-based: Hybrid ResGNN-UNet, AUROC 0.964), standard thresholds capture only 7-72% of true fire spread. CRC eliminates this failure uniformly. Our central finding is that model architecture determines evacuation efficiency, while CRC determines safety: both spatial models with CRC achieve approximately 95% fire coverage while flagging only approximately 15% of total pixels, making them 4.2x more efficient than LightGBM, while the graph model's additional complexity over a simple U-Net yields no meaningful efficiency gain. We propose a shift-aware three-way CRC framework that assigns SAFE/MONITOR/EVACUATE zones for operational triage, and characterize a fundamental limitation of prevalence-weighted bounds under extreme class imbalance (approximately 5% fire prevalence). All models, calibration code, and evaluation pipelines are released for reproducibility.

Abstract

Explore similar work

Conformal Risk Control for Safety-Critical Wildfire Evacuation Mapping: A Comparative Study of Tabular, Spatial, and Graph-Based Models

Operational evaluation of data-driven forest fire forecasting models

Set Prediction for Next-Day Active Fire Forecasting