Towards Explainable Benchmarking for Data-driven Post-Wildfire Debris Flow Prediction

Zhisheng Qi; Li Zhu; Utkarsh Sahu; Douglas Tommey; Josh Roering; Yu Wang

cs.LGOct 5, 2026

Towards Explainable Benchmarking for Data-driven Post-Wildfire Debris Flow Prediction

Authors: Zhisheng Qi, Li Zhu, Utkarsh Sahu, Douglas Tommey, Josh Roering, Yu Wang

Organizations: University of Georgia Athens, Georgia, USA · University of Oregon Eugene, Oregon, USA

Abstract

Post-wildfire debris flows (PFDFs) are destructive sediment-laden hazards triggered when intense rainfall strikes recently burned terrain, destabilizing hillslopes and threatening infrastructure, local economies, and community safety. Data-driven methods have been proposed to learn predictive patterns directly from historical PFDF observations. However, the current research landscape of data-driven PFDF prediction remains highly fragmented across feature spaces, model architectures, and evaluation protocols, making rigorous comparison and the derivation of scientific insights difficult. Moreover, existing studies lack a systematic investigation into the relative importance of heterogeneous factors (e.g., meteorological conditions, terrain characteristics, soil properties, and burn severity) in triggering PFDF. To address these limitations, we present a unified benchmark for data-driven PFDF prediction, enabling fair and comprehensive evaluation across diverse models and feature configurations. Furthermore, to better understand the underlying drivers of PFDF formation, we propose a reinforcement learning-based feature selection framework that identifies factors whose perturbations render positive and negative events indistinguishable, thereby discovering the regional underlying mechanisms of PFDF occurrence across regions. Our code and benchmark are publicly available at https://github.com/KINDLab-Fly/PFDF-Benchmark.

Figures & tables

Figure 1. Benchmark Framework of Data-driven PFDF Prediction, including data curation, evaluation metrics, baseline, and feature selection. Heterogeneous environmental inputs support standardized model comparisons, while reinforcement learning identifies influential features. Benchmark Framework of Data-driven PFDF Prediction, including data curation, evaluation metrics, baseline, and feature selection. Heterogeneous environmental inputs support standardized model comparisons, while reinforcement learning identifies influential features.

Panel of Table 1 — Table 1. The LLM is constrained to fill a predefined JSON schema with 18 numeric PFDF-relevant weather indicators and one concise textual summary. These fields decompose broad forecast signals, including rainfall intensity, storm duration or repetition, flash-flood language, convection, terrain enhancement, antecedent wetness, burn-scar sensitivity, and overall PFDF weather risk.

Method	50/25/25	60/20/20	70/15/15	80/10/10
USGS ( Staley et al., 2016b )	$0.388\pm 0.031$	$0.371\pm 0.030$	$0.376\pm 0.030$	$0.402\pm 0.044$
LR ( Nikolopoulos et al., 2018 ; Kern et al., 2017 )	$0.449\pm 0.022$	$0.409\pm 0.052$	$0.441\pm 0.035$	$0.484\pm 0.035$
CART ( Kern et al., 2017 )	$0.523\pm 0.032$	$0.509\pm 0.043$	$0.556\pm 0.078$	$0.529\pm 0.042$
SVM ( Kern et al., 2017 )	$0.504\pm 0.046$	$0.464\pm 0.034$	$0.524\pm 0.058$	$0.552\pm 0.066$
RF ( Nikolopoulos et al., 2018 )	$0.585\pm 0.027$	$\bm{0.617\pm 0.037}$	$\bm{0.658\pm 0.046}$	$\bm{0.674\pm 0.041}$
HGB	$\bm{0.604\pm 0.038}$	$0.611\pm 0.032$	$0.640\pm 0.040$	$0.648\pm 0.052$

Table 2. Baseline benchmark under original and selected augmented feature sets. Entries report the mean test set Threat Score across five zone-stratified split seeds.

Figure 2. Top-10 Important Features for PFDF prediction across regions with color denoting feature category. Top-10 Important Features for PFDF prediction across regions with color denoting feature category.

Area	Zone	Records	50/25/25	60/20/20	70/15/15	80/10/10
Southern California	OSD	103 (25/78)	$+0.19$	$+0.23$	$+0.22$	$+0.19$
	SGSBSJ	646 (176/470)	$+0.16$	$+0.19$	$+0.19$	$+0.17$
	VEN	24 (0/24)	$+0.00$	$+0.00$	N/A	$+0.00$
Arizona	NAZ	71 (5/66)	$+0.05$	$+0.12$	$+0.00$	$+0.00$
Arizona	SAZ	13 (2/11)	$+0.00$	$+0.00$	$+0.00$	$+0.00$
N-Mexico	CNM	52 (35/17)	$-0.01$	$+0.00$	$+0.05$	$+0.06$

Table 3. Region-wise absolute Threat Score improvement of the original-feature neural network over the fixed USGS logistic-regression reference. Entries report

\Delta\mathrm{TS}=\mathrm{TS}_{\mathrm{NN}}-\mathrm{TS}_{\mathrm{USGS}}

averaged across split seeds. Region counts are reported as total records (positive/negative).

Figure 3. Regional diagnostic for OSD, SWCO, and MT. Left: PFDF/non-PFDF mean ratios for USGS inputs and high-ranked learned features. Right: fixed USGS predicted probability by observed class, with the decision threshold and missed PFDF events. Regional diagnostic for OSD, SWCO, and MT. Left: PFDF/non-PFDF mean ratios for USGS inputs and high-ranked learned features. Right: fixed USGS predicted probability by observed class, with the decision threshold and missed PFDF events.

Explore similar work

Evaluating Machine Learning Models for Post-Wildfire Debris-Flow Prediction
Aug 5, 2026Quinn Ledingham, Zhengsen Xu, Yimin Zhu +6Wildfire Prediction Rainfall
Department of Geomatics Engineering, University of Calgary, Calgary, AB, Canada
Does Your Wildfire Prediction Model Actually Work, or Just Score Well?
May 14, 2026Yangshuang Xu, Yuyang Dai, Liling Chang +2Wildfire Prediction Wireless Foundation Models
Florida State University · Northeastern University
DELUGE: Towards Continental-Scale Daily Pluvial Flood Damage Prediction via Interpretable Conditioning on Foundation Model Embeddings
Jul 17, 2026Yuya Kawakami, Daniel Cayan, Dongyu Liu +2Accurate Flood Prediction Sen1Floods11
University of California, Davis Davis, California, USA · Scripps Institution of Oceanography University of California, San Diego La Jolla, California, USA

Aug 5, 2026, cs.LGJ/K move · Enter open · S save

Evaluating Machine Learning Models for Post-Wildfire Debris-Flow Prediction

Quinn Ledingham, Zhengsen Xu, Yimin Zhu, Zack Dewis+5

Department of Geomatics Engineering, University of Calgary, Calgary, AB, Canada

Wildfire Prediction Rainfall Model Evaluation Tabular Prior-Data Fitted Network Shapley Additive Explanations Deep Learning

Open paper AskPDF arXiv

Prediction of post-wildfire debris flows is critical for mitigating hazards to communities, infrastructure, and resources during intense rainfall in recently burned areas. However, identifying reliable machine learning models is complicated by overlapping debris-flow and non-debris-flow events in feature space, the need for model interpretability, and limited training data. This paper addresses these challenges through a systematic evaluation of machine learning models in terms of predictive performance, feature importance, and synthetic data augmentation. Using basin-scale observations of post-wildfire debris-flow events across the western United States, we compare 15 models, including the Tabular Prior-Data Fitted Network (TabPFN). Repeated stratified cross-validation shows that TabPFN achieves the highest unaugmented performance with a threat score of 0.637, closely followed by the best tree-based models. SHapley Additive exPlanations (SHAP) are used to identify the features driving predictions, revealing that short-duration rainfall intensity and storm accumulation consistently rank highest, while burn severity and terrain features contribute less. We further evaluate synthetic data augmentation using TabPFN-generated samples to address the scarcity of debris-flow observations. Synthetic augmentation improves the performance of all models except CNN, with the largest mean threat score increase of +0.041 among the deep learning models. By combining rigorous model benchmarking, interpretable feature analysis, and synthetic data augmentation, this work provides a comprehensive framework for improving post-wildfire debris-flow prediction.

Abstract

Explore similar work

Evaluating Machine Learning Models for Post-Wildfire Debris-Flow Prediction

Does Your Wildfire Prediction Model Actually Work, or Just Score Well?

DELUGE: Towards Continental-Scale Daily Pluvial Flood Damage Prediction via Interpretable Conditioning on Foundation Model Embeddings