Crop Mapping Evaluation under Sampling Bias

Akram Zaytar; Kemunto Ochwang'i; Caleb Robinson; Girmaw A. Tadesse; Gilles Q. Hacheme; Rahul Dodhia; Inbal Becker-Reshef; Juan M. Lavista Ferres

cs.LGOct 31, 2026

Crop Mapping Evaluation under Sampling Bias

Authors: Akram Zaytar, Kemunto Ochwang'i, Caleb Robinson, Girmaw A. Tadesse, Gilles Q. Hacheme, Rahul Dodhia, Inbal Becker-Reshef, Juan M. Lavista Ferres

Organizations: Microsoft AI for Good Research Lab · Microsoft Research

Abstract

Government agencies use crop maps to estimate production, monitor food security, and set agricultural policy. These crop maps are usually validated against field labels, but these labels are rarely a random sample of the area the map covers. Field teams tend to label parcels near roads and skip ones that are small or hard to reach, so performance measured on held-out labels mainly describes the places they visited. We call the difference between this reported performance and performance over the whole mapped region the deployment performance gap. To estimate performance over the whole region, we use a weighted form of prediction-powered inference (PPI). The classifier's predictions on unlabeled points across the region give a first estimate, and its errors on the labeled points correct that estimate, with more weight on labels from under-sampled conditions. We test this in five European regions where crop registers label every declared parcel, so the true performance is known. Simulating six documented collection practices there produces gaps of up to 0.20 macro-F1. On the collection rule that creates the largest gap in each region, 5 km block cross-validation still overestimates macro-F1 by as much as 0.18, while PPI with weights estimated from satellite embeddings reduces the largest error across regions to 0.07. Importance-weighted cross-validation with the same weights does almost as well, so most of the improvement comes from the weights themselves. We also apply the method to three Kenyan datasets that have no region-wide reference labels. For a roadside dataset collected with helmet-mounted cameras, the estimated macro-F1 falls from 0.93 under random cross-validation to 0.61, although the correction relies on a small number of heavily weighted labels. Our code is available at https://github.com/microsoft/pappi.

Figures & tables

Panel of Figure 1 — Figure 1: Random CV, block CV, and PPI on one simulated campaign in Vaucluse. Light grey areas are crop parcels. (a) An elevation cutoff draws 1,000 labels, mostly from low-lying parcels. The models make fewer errors there, so random CV (0.903) and 5 km block CV (0.887), which test the models only on these labels, overestimate their performance. (b) A propensity model estimates how likely each parcel was to be labeled. PPI predicts the 5,000 parcels of a deployment sample, then corrects that estimate with the labels, and rarely sampled labels get larger weights. The heavily weighted misclassified labels lower the estimate to 0.775. (c) The crop register labels every parcel, so we know which parcels most of the five fold models misclassify and what their true pooled macro-F1 is (0.769). This is one run of the evaluation stage (Section 2.4 ).

Collection rule	Collection practice	Sources
Road	Fields labeled from a vehicle; inclusion falls with road distance.	Defourny et al.,2014 ; Jolivot et al.,2021 ; Waldner et al.,2019 ; Nakalembe et al.,2025
Settlement	Enumerators skip parcels far from homes or villages.	Carletto et al.,2016 ; Kilic et al.,2017
Elevation	Survey frames remove high points from field visits.	Eurostat,2025
Slope	A fixed walking-time budget reaches less land on steep terrain.	Tobler,1993 ; Nelson et al.,2019 ; Eurostat,2019
Pilot region	Labels come from one pilot region; use extends beyond it.	Jolivot et al.,2021 ; Orynbaikyzy et al.,2022
Parcel size	Rules or cost exclude small parcels and favor large ones.	Defourny et al.,2014 ; Azzari et al.,2021

Table 1: Collection rules. Six documented collection practices and the studies that report them. Eq. ( 7 ) defines each rule.

Collection rule	Vaucluse	Goriška	Bouches-du-Rhône	Hérault	Osnabrück
Uniform control	$+0.005$ $\pm.006$	$-0.005$ $\pm.007$	$-0.003$ $\pm.005$	$-0.008$ $\pm.012$	$+0.009$ $\pm.010$
Road	$-0.001$ $\pm.007$	$-0.006$ $\pm.007$	$-0.004$ $\pm.006$	$-0.020$ $\pm.012$	$+0.007$ $\pm.007$
Settlement	$-0.004$ $\pm.004$	$-0.003$ $\pm.008$	$-0.004$ $\pm.006$	$-0.009$ $\pm.009$	$+0.012$ $\pm.010$
Elevation	$\mathbf{+0.115}$ $\pm.011$	$+0.043$ $\pm.010$	$\mathbf{+0.109}$ $\pm.033$	$+0.060$ $\pm.009$	$+0.008$ $\pm.010$
Slope	$+0.093$ $\pm.006$	$\mathbf{+0.112}$ $\pm.019$	$+0.025$ $\pm.011$	$+0.071$ $\pm.010$	$+0.022$ $\pm.009$
Pilot region	$\mathbf{+0.171}$ $\pm.011$	$+0.047$ $\pm.016$	$\mathbf{+0.124}$ $\pm.037$	$+0.043$ $\pm.007$	$+0.011$ $\pm.010$

Table 2: Four collection rules make random CV overstate deployment macro-F1 by 0.10 or more in at least one region. The gap is random CV minus the fold models’ true pooled macro-F1; bold marks gaps of 0.10 or more. Each cell gives the mean gap

\pm

the half-width of its 95% interval over 16 runs, at the rule strength with the largest mean gap; the uniform control has no strength.

Method	Variant	Vaucluse	Goriška	Bouches-du-Rhône	Hérault	Osnabrück
Random CV	—	$+0.170$ $\pm.008$	$+0.192$ $\pm.005$	$+0.111$ $\pm.025$	$+0.119$ $\pm.003$	$+0.026$ $\pm.005$
Block CV	Random blocks	$+0.166$ $\pm.007$	$+0.181$ $\pm.006$	$+0.031$ $\pm.027$	$+0.115$ $\pm.003$	$+0.010$ $\pm.006$
	Checkerboard blocks	$+0.168$ $\pm.008$	$+0.183$ $\pm.005$	$+0.046$ $\pm.029$	$+0.116$ $\pm.003$	$+0.012$ $\pm.005$
kNNDM CV	Geographic	$+0.158$ $\pm.007$	$+0.186$ $\pm.005$	$-0.035$ $\pm.030$	$+0.118$ $\pm.002$	$+0.024$ $\pm.005$
	Feature space	$+0.140$ $\pm.010$	$+0.144$ $\pm.013$	$-0.080$ $\pm.034$	$+0.110$ $\pm.003$	$-0.023$ $\pm.007$
IWCV	Embeddings	$+0.067$ $\pm.022$	$+0.058$ $\pm.014$	$-0.032$ $\pm.040$	$+0.010$ $\pm.017$	$+0.004$ $\pm.016$

Table 3: A weighted estimator (PPI or IWCV) leaves the mean error closest to zero in every region; block and kNNDM CV remove at most a quarter of the three largest gaps. Mean error by method on each region’s worst-case collection rule, chosen on the sweep runs: pilot region in Vaucluse and Bouches-du-Rhône, and parcel size elsewhere. Bold marks the error closest to zero among methods usable in practice; true weights, the simulated inclusion probabilities, are unknown in practice. Cells give the mean error over 32 evaluation runs

\pm

the half-width of its 95% percentile bootstrap interval; for random CV, this is the deployment performance gap.

Figure 2: PPI places the roadside Helmets estimate below every CV score. Bars show random CV, 5 km block CV, geographic kNNDM CV, IWCV, and PPI with its 95% block bootstrap interval. The text beside each PPI bar gives the change from random CV (

\Delta

) with its 95% interval and the effective sample size (

n_{\mathrm{eff}}

). Each dataset label gives its number of labels,

n_{L}

. Datasets are ordered by

|\Delta|

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Source / collection	Year	$n$	Positive class (share)
Goriška, SI	EuroCrops	2021	21,388	Vineyards (26%)
Hérault, FR	EuroCrops	2018	27,116	Vineyards (83%)
Osnabrück, DE	EuroCrops	2021	18,104	Silage maize (27%)
Vaucluse, FR	RPG	2024	59,241	Vineyards (56%)
Bouches-du-Rhône, FR	RPG	2024	23,462	Vineyards (27%)

Appendix

Table 4: European regions.

n

counts crop parcels, and share is the positive class’s share of

n

. Hérault and Osnabrück are restricted to 113 and 26 municipalities.

Dataset	Source / collection	Year	$n$	Positive class (share)
Crop vs. non-crop over the whole region
Helmets	Roadside camera	2021–22	2,478	Crop (50%)
WorldCereal	Stratified survey	2021	1,720	Crop (58%)
Most common crop vs. other crops over cropland
PlantVillage	Walk-around GPS app	2019	296	Maize (67%)

Appendix

Table 5: Kenyan datasets.

n

counts labels, and share is the positive class’s share of

n

. PlantVillage deploys over WorldCover cropland ( Zanaga et al., 2022 ) . Sources: Helmets ( Nakalembe et al., 2025 ) , WorldCereal ( European Commission, Joint Research Centre, 2021 ; Van et al., 2023 ) , PlantVillage ( PlantVillage, 2019 ) .

Figure 4: Weighting raises the land-cover overlap of the Helmets labels from 43% to 74%, but half the weight falls on 8 labels. Both maps show the $1^{\circ}\times 1^{\circ}$ deployment region. Brightness compares each WorldCover class’s share of label weight with its share of deployment area, capped at one; a halo aids visibility. Left: equal label weights, with labels shown as red dots. Right: the inverse-probability weights of the PPI estimate in Section 4 . Red rings identify the labels holding half of the total weight. Bars compare each class’s area share (light gray) with its weight share before (gray) and after (blue) correction. Map percentages give land-cover distribution overlap, $\sum_{c}\min(a_{c},u_{c})$ .

Crop Type Classification

See all 17 papers

May 20, 2026

Mapping Tomato Cropping Systems in California Using AlphaEarth Geospatial Embeddings and Deep Learning Analysis

Field-scale crop maps support supply-chain forecasting and policy, yet statewide crop identification still often depends on retrospective surveys or remote-sensing workflows built around hand-engineered spectral features. Those pipelines can be accurate, but they require repeated preprocessing and often lose robustness across years. This study evaluated whether Google DeepMind's AlphaEarth geospatial embeddings can serve as an analysis-ready alternative for mapping processing tomato systems in California. LandIQ 2018 crop polygons were used to assemble a balanced reference dataset of 4,742 tomato and 4,742 non-tomato fields. For each polygon, 64-band AlphaEarth embedding chips were extracted and aligned with binary masks, then divided into spatially independent training (n = 6,638), validation (n = 1,422), and test (n = 1,424) sets. A U-Net segmentation model was trained on AWS SageMaker using a composite masked binary cross-entropy and soft Dice loss. To complement hard predictions, Monte Carlo dropout was retained at inference and repeated 100 times per chip to estimate predictive mean and variance. On the independent test set, the model achieved 99.19% pixel accuracy, 98.69% precision, 99.40% recall, 99.04% F1 score, 98.11% intersection over union, and 99.02% chip accuracy. Uncertainty maps were consistently highest near field edges and low within field interiors. The results show that AlphaEarth embeddings retain crop-relevant spatial and temporal structure and can support accurate, field-scale tomato mapping without manual feature engineering.

Mohammadreza Narimani, Alireza Pourreza, Parastoo Farajpoor

Department of Biological and Agricultural Engineering, University of California, Davis, Davis, CA, 95616, USA

Remote Sensing Image Segmentation Agricultural Remote Sensing Geospatial Foundation Models Crop Type Classification

Ask PDF

Oct 6, 2026

Prediction-powered inference for time series across space

The following motif is common in spatiotemporal settings: we have a sequence of covariate and label pairs observed for a relatively short, recent time period. We have access to unlabeled covariates over a longer time period. Data is observed over many spatial locations. For instance, crop yield might be observed over a large geographical area for recent years, but weather data (which is informative about crop yield) is available for a much longer period. The goal is to estimate, at each spatial location, the expected label (e.g., crop yield) in the future and provide a valid confidence interval for this value. The observed time period alone is too short for reliable estimates. Imputing missing labels with machine learning can cause substantial bias. Prediction-powered inference (PPI) can correct for this bias, but it relies on an i.i.d. assumption that breaks under our expected temporal dependencies. Heteroskedasticity and autocorrelation consistent (HAC) procedures account for temporal correlation, but have not been adapted to cases where some labels are imputed. We provide reliable point estimates and confidence intervals given: short labeled time series (across spatial locations), a longer unlabeled time series, and an imperfect predictor of labels given covariates. We show our method outperforms natural alternatives.

Shahzar Rizvi, David Burt, Vishwak Srinivasan +3

MIT LIDS · UMass Amherst · Stanford University +1

Spatiotemporal Forecasting Prediction-Powered Inference Robust Estimation

Ask PDF

Mar 31, 2026

Aligning Validation with Deployment in Spatial Prediction: Target-Weighted Cross-Validation

Reliable estimation of predictive performance is essential for spatial environmental modeling, where machine-learning models are used to generate maps from unevenly distributed observations. Standard cross-validation (CV) assumes that validation data are representative of prediction conditions across the target domain. In practice, this assumption is often violated due to preferential or clustered sampling, leading to biased performance and uncertainty estimates. We introduce a deployment-oriented validation framework based on weighted CV that aligns validation tasks with the distribution of prediction tasks across a specified domain. The framework includes importance-weighted cross-validation (IWCV) and a calibration-based approach, Target-Weighted Cross-Validation (TWCV), which uses spatially meaningful task descriptors such as environmental covariates and prediction distance. Simulation experiments show that conventional non-spatial and spatial CV strategies can exhibit substantial bias under realistic sampling designs, whereas weighted CV approaches substantially reduce this bias when validation tasks adequately cover the deployment-task space. A case study on mapping nitrogen dioxide (NO

_2

) concentrations across Germany demonstrates that standard CV can overestimate prediction error due to sampling bias, while weighted CV yields estimates more consistent with deployment conditions. The framework separates validation task generation from risk estimation and provides a practical approach for improving performance assessment in spatial prediction settings where sample distributions differ from prediction domains.

Alexander Brenning, Thomas Suesse

Friedrich Schiller University Jena, Department of Geography, Jena, Germany · ELLIS Unit Jena, Jena, Germany

Topic pending

Ask PDF

Abstract

Crop Type Classification

Explore similar work

Mapping Tomato Cropping Systems in California Using AlphaEarth Geospatial Embeddings and Deep Learning Analysis

Prediction-powered inference for time series across space

Aligning Validation with Deployment in Spatial Prediction: Target-Weighted Cross-Validation