Reliable estimation of predictive performance is essential for spatial environmental modeling, where machine-learning models are used to generate maps from unevenly distributed observations. Standard cross-validation (CV) assumes that validation data are representative of prediction conditions across the target domain. In practice, this assumption is often violated due to preferential or clustered sampling, leading to biased performance and uncertainty estimates. We introduce a deployment-oriented validation framework based on weighted CV that aligns validation tasks with the distribution of prediction tasks across a specified domain. The framework includes importance-weighted cross-validation (IWCV) and a calibration-based approach, Target-Weighted Cross-Validation (TWCV), which uses spatially meaningful task descriptors such as environmental covariates and prediction distance. Simulation experiments show that conventional non-spatial and spatial CV strategies can exhibit substantial bias under realistic sampling designs, whereas weighted CV approaches substantially reduce this bias when validation tasks adequately cover the deployment-task space. A case study on mapping nitrogen dioxide (NO2) concentrations across Germany demonstrates that standard CV can overestimate prediction error due to sampling bias, while weighted CV yields estimates more consistent with deployment conditions. The framework separates validation task generation from risk estimation and provides a practical approach for improving performance assessment in spatial prediction settings where sample distributions differ from prediction domains.
Figures & tables
Figure 1: Illustration of the three sampling designs and their effects on spatial prediction. Columns correspond to random, clustered, and preferentially biased sampling for one representative simulation setting (strong trend, ρ=0.1 ). The top row shows the simulated response field on D=[0,1]2 together with sampled locations. The second and third rows show the corresponding prediction maps obtained from random forest and heteroskedastic regression–kriging, respectively.
Figure 2: Predicted annual mean NO 2 concentrations across Germany obtained with random forest (left) and regression–kriging models (right), along with the locations of 503 monitoring stations. Both models use the same set of topographic and demographic covariates.
Figure 3: Joint distribution of prediction tasks in (x1,d) space in the simulation study. Panels compare the deployment task distribution with validation tasks generated by LOOCV, random 10 -fold CV, LOBOCV, and buffered LOOCV, separately for the three sampling designs. Here, x1 denotes the large-scale environmental gradient and d the nearest-neighbour prediction distance to the corresponding training set.
Figure 4: Mean error of RMSE estimators in the simulation study for different validation schemes and model-based uncertainty estimators for one representative simulation scenario (strong trend, ρ=0.1 ). Points show the mean difference between the estimated RMSE and the true deployment RMSE, with approximate 95% confidence intervals. The dashed line indicates unbiased estimation of deployment RMSE. Positive values indicate overestimation of prediction error. Refer to the Supplement for results of other simulation scenarios and estimators.
Figure 5: Empirical distribution of prediction tasks in the space spanned by population density and prediction distance d for the 2018 NO 2 case study in Germany. Panels compare the deployment tasks on the 2 km target grid with validation tasks induced by LOOCV, random CV, LOBOCV, kNNDM, and buffered LOOCV. Local population density is shown on the log(1+x) scale.
Figure 6: Estimated root mean squared prediction error (RMSE) for annual mean NO 2 concentrations in Germany obtained from different uncertainty estimators and prediction models. Shown are the model-based estimate, LOOCV, random CV, LOBOCV, and buffered CV in unweighted, importance-weighted (IWCV), distance-weighted (DWCV), and target-weighted (TWCV) form.
Split conformal prediction uses prediction errors on held-out (calibration) data to determine how wide the prediction intervals should be. It guarantees distribution-free finite-sample coverage when these errors and the error at the target site are exchangeable. This assumption may fail under spatial dependence and nonrandom sampling geometry. Existing spatial methods use fitting residuals to remove the predictable part of spatial variation from calibration and target errors. However, the spatial variation that only the calibration residuals can predict remains in both the target and calibration errors, reducing the efficiency and stability of the interval. We address this by additionally conditioning on the calibration residuals sequentially, which scales to large networks through nearest-neighbour approximations. Under a correct working covariance and an elliptical residual law, the resulting interval has exact finite-sample coverage under any spatial design, and under further conditions it is asymptotically oracle efficient. We also bound coverage loss under covariance misspecification and develop a diagnostic that identifies regions at risk of undercoverage. In simulated data, our method produces narrower and more stable intervals than global and localized state-of-the-art alternatives. In a national PM2.5 application, it produces narrower intervals within the network and identifies regions at risk of coverage failure.
Ayush Baran Sen, Arkajyoti Saha
Indian Statistical Institute, Kolkata, India · Department of Statistics, University of California, Irvine
Understanding how a prediction model will perform in a new environment before deployment is essential to preventing harm when algorithms inform decision-making. Two common sources of model performance degradation are (i) covariate shift, where the target covariate distribution differs from the source, and (ii) selective labels, where the observability of outcomes depends on historical decisions. We study pre-deployment model evaluation under the joint presence of covariate shift and labeling of outcomes selectively based on observed features. In particular, we present a double machine learning procedure for estimating the target risk of an arbitrary black-box prediction model under a general loss function. We show identification of this estimand under standard assumptions and derive a bias-corrected estimator based on the influence function of the target risk. Finally, we evaluate our estimator through experiments using the eICU electronic health records database, showing that it tracks the true target risk more accurately than methods that address either selective labels or covariate shift alone, as well as baselines that combine standard plug-in approaches.
Annie Ulichney, Amanda Coston
Department of Statistics, University of California, Berkeley
When outcome data are expensive or onerous to collect, scientists increasingly substitute predictions from machine learning and AI models for unlabeled cases, a process which has consequences for downstream statistical inference. While recent methods provide valid uncertainty quantification under independent sampling, real-world applications involve missing at random (MAR) labeling and spatial dependence. For inference in this setting, we propose a doubly robust estimator with cross-fit nuisances. We show that cross-fitting induces fold-level correlation that distorts spatial variance estimators, producing unstable or overly conservative confidence intervals. To address this, we propose a jackknife spatial heteroscedasticity and autocorrelation consistent (HAC) variance correction that separates spatial dependence from fold-induced noise. Under standard identification and dependence conditions, the resulting intervals are asymptotically valid. Simulations and benchmark datasets show substantial improvement in finite-sample calibration, particularly under MAR labeling and clustered sampling.
Stephen Salerno, Zhenke Wu, Tyler McCormick
Fred Hutch Cancer Center · University of Michigan · University of Washington