Reliable estimation of predictive performance is essential for spatial environmental modeling, where machine-learning models are used to generate maps from unevenly distributed observations. Standard cross-validation (CV) assumes that validation data are representative of prediction conditions across the target domain. In practice, this assumption is often violated due to preferential or clustered sampling, leading to biased performance and uncertainty estimates. We introduce a deployment-oriented validation framework based on weighted CV that aligns validation tasks with the distribution of prediction tasks across a specified domain. The framework includes importance-weighted cross-validation (IWCV) and a calibration-based approach, Target-Weighted Cross-Validation (TWCV), which uses spatially meaningful task descriptors such as environmental covariates and prediction distance. Simulation experiments show that conventional non-spatial and spatial CV strategies can exhibit substantial bias under realistic sampling designs, whereas weighted CV approaches substantially reduce this bias when validation tasks adequately cover the deployment-task space. A case study on mapping nitrogen dioxide (NO2) concentrations across Germany demonstrates that standard CV can overestimate prediction error due to sampling bias, while weighted CV yields estimates more consistent with deployment conditions. The framework separates validation task generation from risk estimation and provides a practical approach for improving performance assessment in spatial prediction settings where sample distributions differ from prediction domains.
Figures & tables
Figure 1: Illustration of the three sampling designs and their effects on spatial prediction. Columns correspond to random, clustered, and preferentially biased sampling for one representative simulation setting (strong trend, ρ=0.1 ). The top row shows the simulated response field on D=[0,1]2 together with sampled locations. The second and third rows show the corresponding prediction maps obtained from random forest and heteroskedastic regression–kriging, respectively.
Figure 2: Predicted annual mean NO 2 concentrations across Germany obtained with random forest (left) and regression–kriging models (right), along with the locations of 503 monitoring stations. Both models use the same set of topographic and demographic covariates.
Figure 3: Joint distribution of prediction tasks in (x1,d) space in the simulation study. Panels compare the deployment task distribution with validation tasks generated by LOOCV, random 10 -fold CV, LOBOCV, and buffered LOOCV, separately for the three sampling designs. Here, x1 denotes the large-scale environmental gradient and d the nearest-neighbour prediction distance to the corresponding training set.
Figure 4: Mean error of RMSE estimators in the simulation study for different validation schemes and model-based uncertainty estimators for one representative simulation scenario (strong trend, ρ=0.1 ). Points show the mean difference between the estimated RMSE and the true deployment RMSE, with approximate 95% confidence intervals. The dashed line indicates unbiased estimation of deployment RMSE. Positive values indicate overestimation of prediction error. Refer to the Supplement for results of other simulation scenarios and estimators.
Figure 5: Empirical distribution of prediction tasks in the space spanned by population density and prediction distance d for the 2018 NO 2 case study in Germany. Panels compare the deployment tasks on the 2 km target grid with validation tasks induced by LOOCV, random CV, LOBOCV, kNNDM, and buffered LOOCV. Local population density is shown on the log(1+x) scale.
Figure 6: Estimated root mean squared prediction error (RMSE) for annual mean NO 2 concentrations in Germany obtained from different uncertainty estimators and prediction models. Shown are the model-based estimate, LOOCV, random CV, LOBOCV, and buffered CV in unweighted, importance-weighted (IWCV), distance-weighted (DWCV), and target-weighted (TWCV) form.