Quantifying Protocol-Induced Uncertainty in Comparative Predictive-Model Evaluation: Evidence from Large-Scale Daily PM10 Forecasting

Rafael da Silva; Kiersten Monahan

cs.LGAug 26, 2026

Quantifying Protocol-Induced Uncertainty in Comparative Predictive-Model Evaluation: Evidence from Large-Scale Daily PM10 Forecasting

Authors: Rafael da Silva, Kiersten Monahan

Organizations: Applied Data Science program, Eastern University 1300 Eagle Road, St. Davids, PA 19087, USA

Abstract

Comparative studies of predictive models often end by ranking candidate models, yet these rankings depend on evaluation protocols whose influence is rarely treated as a source of uncertainty. We formalize this problem as protocol-induced ranking uncertainty and introduce a framework that compares ranking displacement caused by switching protocols with displacement produced by conventional choices within a fixed protocol. We quantify these effects using the Protocol Sensitivity Score (PSS) and a full-refit intraprotocol reference. We validate the framework in a large-scale sequential prediction study of daily PM10. Static-split and rolling-origin evaluation are compared across 425 European background stations and 365 US EPA monitors. Switching protocols produces mean PSS values of 0.801 and 0.772 and changes the selected model at 35.3% and 31.5% of stations, respectively. In Europe, intraprotocol perturbations with identical scored targets produce PSS values of 0.072 and 0.230, with winner-swap rates of 0.8% and 4.9%. Between-protocol displacement is therefore substantially larger than the selected within-protocol references. Expanding the candidate set from three to nine models increases the between-protocol winner-swap rate to 60.2% in Europe. The pattern also persists under a frozen protocol applied to held-out background and non-background stations. These results show that model-selection conclusions can depend materially on legitimate evaluation choices. We recommend reporting ranking stability under a small set of defensible intraprotocol perturbations alongside claims of model superiority.

Figures & tables

Panel	Network	Universe	Eligible	$N$	Blocks	Role
EEA Heisig background	Heisig	2807	1216	425	15	confirmatory core
EPA AirData	EPA	–	365	365	38	cross-continental
EEA airbase (excluded)	airbase	2615	1172	–	–	outside confirmatory $N$
Single-site (Elche)	EEA local	1	1	1	–	not aggregated

Table 1: Panel summary for H1 and H4 claims.

Figure 1: Station selection funnel by network—universe, eligibility, European background stratum, and US census—with branches kept separate (no pooling).

Figure 2: Study design at a glance: ranking variance under two evaluation protocols. Attribution contrasts between-protocol discrepancy with intraprotocol noise.

Figure 3: Static-split versus rolling-origin protocols on a shared timeline: both use the full series; evaluation origins differ.

Figure 4: PSS distributions on the European and US panels, with equivalence margin

\delta

and the discrete measurement grid.

Figure 5: Spatial maps of station-level PSS on the European and US panels.

Panel	Measure	Between protocols	Rolling convention	Static convention
EEA	PSS	0.801 [0.660; 0.908]	0.230 [0.212; 0.257]	0.072 [0.064; 0.083]
EEA	Winner swap	35.3% [22.1%; 47.6%]	4.9% [3.0%; 8.1%]	0.8% [0.4%; 1.3%]
EPA	PSS	0.772 [0.711; 0.823]	0.237 [0.197; 0.277]	0.065 [0.044; 0.085]
EPA	Winner swap	31.5% [20.6%; 38.9%]	6.3% [3.4%; 10.0%]	1.2% [0.4%; 2.1%]

Table 2: Ranking-variance decomposition (H2) on both panels.

Figure 6: Ranking-variance decomposition: between-protocol contrast versus intraprotocol nulls (PSS and winner-swap distributions).

Panel	Measure	Between protocols	Static placebo	Rolling placebo
EEA	PSS	0.540 [0.499; 0.572]	0.157 [0.148; 0.165]	0.251 [0.233; 0.276]
EEA	Winner swap	60.2% [56.0; 64.3]	27.2% [18.8; 35.0]	22.8% [17.5; 30.3]
EEA	MCS exclusion	1.4% [0.4%; 3.4%]	1.2% [0.5%; 2.5%]	4.7% [3.0%; 7.5%]
EPA	PSS	0.550 [0.505; 0.583]	0.169 [0.153; 0.181]	0.273 [0.251; 0.294]
EPA	Winner swap	67.4% [61.3; 73.3]	16.2% [12.3; 21.6]	27.4% [21.6; 32.7]
EPA	MCS exclusion	1.4% [0.4%; 2.7%]	0.4% [0.1%; 0.9%]	4.4% [2.1%; 7.9%]

Table 3: Nine-model arm: between-protocol contrast vs intraprotocol placebos.

Figure 7: Three-model versus nine-model decision spaces: between-protocol contrast and intraprotocol nulls under PSS and winner swap.

Figure 8: Winner-swap rate versus number of Borda-aggregated intraprotocol runs (

K

). Rows: EEA and EPA panels. Columns: Curve A (aggregate static, compare to rolling reference) and Curve B (aggregate rolling, compare to static reference). Solid line: median over 100 random subsets; band: interquartile range.

Panel	Curve	$K$	Median [%]	IQR [%]
EEA	Static agg.	1	35.5	[34.4; 38.8]
EEA	Static agg.	3	34.1	[33.4; 35.1]
EEA	Static agg.	5	33.9	[33.4; 34.1]
EEA	Static agg.	7	33.6	[33.4; 33.9]
EEA	Static agg.	10	33.9	[33.8; 34.1]
EEA	Static agg.	13	33.2	[33.2; 33.2]

Table 4: Winner-swap rate (%) under Borda aggregation: median [IQR] over 100 subsets.

Stratum	$N$	Blocks	Mean PSS [95% CI]	Winner swap [95% CI]
Background confirmatory	425	15	0.801 [0.660; 0.908]	35.3% [22.1%; 47.6%]
Background holdout	271	4	0.676 [0.606; 0.833]	20.3% [17.9%; 37.5%]
Traffic / industrial	520	19	0.701 [0.617; 0.847]	23.1% [16.5%; 32.8%]

Table 5: Frozen-protocol confirmation sets.

Panel	$N$	Blocks	Mean PSS	Winner swap	PSS $>$ 0	TOST ( $\delta=0.10$ )	BH-FDR
EEA	425	15	0.801 [0.660; 0.908]	35.3% [22.1; 47.6]	420/425	not equiv.	83
EPA	365	38	0.772 [0.711; 0.823]	31.5% [20.6; 38.9]	362/365	not equiv.	–

Table 6: Side-by-side confirmatory battery.

Alternative	How tested	Outcome
Single-station artefact	Panels of 425 European and 365 US monitors, independent networks, identical protocol, no pooling	PSS positive at 420/425 and 362/365 (Sections 5.1 , 5.9 ; Supplementary Section SM-08)
Evaluation-procedure noise	Full-refit intraprotocol references with strict target pairing	PSS 0.801 between protocols versus 0.072 and 0.230 within protocols; winner swaps 35.3% versus 0.8% and 4.9% (Section 5.2 )
Unequal test sets	Count of date–horizon pairs per station under each protocol	Counts are identical on both panels (Section 4.4 )
Static-split training lag	European re-run with training contiguous to evaluation	PSS 0.773 versus 0.801; mean absolute difference 0.073, below the grid step (Section 6.4 )
Background-stratum restriction	Frozen protocol on 520 traffic and industrial stations	Mean PSS 0.701, with an interval that overlaps the background interval (Section 5.6 )
Adaptation to examined data	Dated freeze before selecting 271 previously unexamined stations; one run	Mean PSS 0.676, inside the confirmatory interval (Section 5.6 )

Table 7: Attempted alternative explanations and outcomes.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Checklist item	Status	In SM/main
Models	Clear description of setting / algorithm / model	Yes	Methods
Models	Clear explanation of assumptions	Yes	Methods, limits
Models	Complexity analysis (time / space / sample)	Partial	Main + runtime note
Theory	Clear statement of claim	Yes	Intro / RQs
Theory	Complete proof	NA	Empirical paper
Data	Relevant statistics (sample sizes)	Yes	Data, tables

Appendix

Table 1: Machine Learning Reproducibility Checklist v2.0 ( Pineau et al., 2021 ) : compliance for this study. Full comments and file pointers: notes/phase5/S3_NeurIPS_checklist_draft.md .

Criterion	Threshold	Obtained	Decision
EEA sample floor	250	1216	pass
External sample floor	50	365	pass
Instrument check ( $\mathrm{PSS}\geq 1.0$ )	1.000	1.143	pass
Equivalence margin $\delta(M=3)$	0.100	0.100	locked
Equivalence margin $\delta(M=9)=4/[M(M-1)H]$	0.008	0.008	locked
Index viability (adj. $R^{2}$ )	0.150	0.105	supplementary

Appendix

Table 2: Pre-specified criteria and outcomes.

Figure 1: Macro execution sequence with hypothesis labels [Hn] , explicit branch terminals, and parallel European and US arms.

Station	Pre-specified criterion	PSS	Reversal
ES/ES1968A	hard reversal or $\mathrm{PSS}\geq 1.0$	1.143	yes

Appendix

Table 3: Instrument check before the panel.

Figure 2: Diagnostic-index fit diagnostics (adj.

R^{2}=0.105

below the pre-specified viability gate of 0.15; reported in supplementary material by rule).

Threshold ( $\mu$ g m -3 )	Min events	Stations $\geq$ min	Exceedance days	Tier	POD/FAR/CSI
50	10	399	62017	SUPPLEMENTARY	not computed

Appendix

Table 4: Exceedance-layer tier (Supplementary Section SM-07).

Panel	$K$	Median [%]	IQR [%]
EEA	1	37.6	[35.5; 40.0]
EEA	3	35.5	[33.9; 37.2]
EEA	5	34.8	[33.4; 35.5]
EEA	7	34.6	[33.4; 35.3]
EEA	10	35.5	[34.6; 36.2]
EEA	13	34.4	[33.9; 34.6]

Appendix

Table 5: Curve C winner-swap rate (%) under joint Borda aggregation: median [IQR] over 100 subsets.

Model Selection

Model selection chooses models, features, or configurations that balance predictive fit, generalization, complexity, and deployment constraints.

See all 90 papers

Rolling-Origin Validation Reverses Model Rankings in Multi-Step PM10 Forecasting: XGBoost, SARIMA, and Persistence
Mar 19, 2026Federico Garcia Crespi, Eduardo Yubero Funes, Marina Alfosea SimonTime Series Forecasting Air Quality Prediction
Departamento de Ingeniería de Computadores, Universidad Miguel Hernández, Elche, Spain · Departamento de Física Aplicada, Universidad Miguel Hernández, Elche, Spain
Evaluation Choices Decide the Forecasting Leaderboard: Evidence from a Production Marketplace Panel
Aug 20, 2026Md Rezwanul Islam, Wael MohammedForecasting Benchmarks Benchmark Design
Field Nation LLC, Minneapolis, MN, USA
Model selection with proper scoring rules on data sets of time series: prefer the mean scaled score
Jun 23, 2026Giorgio Corani, Stefano Damato, Dario Azzimonti +1Model Selection Time Series Forecasting
SUPSI, Istituto Dalle Molle di Studi sull’Intelligenza Artificiale (IDSIA), Lugano, Switzerland

Mar 19, 2026J/K move · Enter open · S save

Rolling-Origin Validation Reverses Model Rankings in Multi-Step PM10 Forecasting: XGBoost, SARIMA, and Persistence

Federico Garcia Crespi, Eduardo Yubero Funes, Marina Alfosea Simon

Departamento de Ingeniería de Computadores, Universidad Miguel Hernández, Elche, Spain · Departamento de Física Aplicada, Universidad Miguel Hernández, Elche, Spain

Time Series Forecasting Air Quality Prediction Forecasting Benchmarks Gradient-Boosted Decision Trees

Open paper AskPDF arXiv

(a) Many air quality forecasting studies report gains from machine learning, but evaluations often use static chronological splits and omit persistence baselines, so the operational added value under routine updating is unclear. (b) Using 2,350 daily PM10 observations from 2017 to 2024 at an urban background monitoring station in southern Europe, we compare XGBoost and SARIMA against persistence under a static split and a rolling-origin protocol with monthly updates. We report horizon-specific skill and the predictability horizon, defined as the maximum horizon with positive persistence-relative skill. Static evaluation suggests XGBoost performs well from one to seven days ahead, but rolling-origin evaluation reverses rankings: XGBoost is not consistently better than persistence at short and intermediate horizons, whereas SARIMA remains positively skilled across the full range. (c) For researchers, static splits can overstate operational usefulness and change rankings. For practitioners, rolling-origin, persistence-referenced skill profiles show which methods stay reliable at each lead time.

Abstract

Model Selection

Explore similar work

Rolling-Origin Validation Reverses Model Rankings in Multi-Step PM10 Forecasting: XGBoost, SARIMA, and Persistence

Evaluation Choices Decide the Forecasting Leaderboard: Evidence from a Production Marketplace Panel

Model selection with proper scoring rules on data sets of time series: prefer the mean scaled score