Inferential Evaluation of Surrogate-Derived Models under Covariate Shift

Longtian Shi; Molei Liu; Doudou Zhou

stat.MLAug 16, 2026

Inferential Evaluation of Surrogate-Derived Models under Covariate Shift

Authors: Longtian Shi, Molei Liu, Doudou Zhou

Organizations: Department of Statistics and Data Science, National University of Singapore · Department of Biostatistics, Peking University Health Science Center, Peking University, Beijing, China · Beijing International Center for Mathematical Research, Peking University, Beijing, China

Abstract

In transfer-learning settings, a model derived from abundant surrogate labels may be deployed in a target population where gold-standard outcomes are unobserved. Evaluating its target performance is essential for determining whether decisions based on the model remain reliable, yet it is difficult when gold labels are scarce, and covariate distributions differ across data sources. We study a three-sample setting with a small gold-labeled source, a larger surrogate-labeled source, and an unlabeled target. Under conditional transportability, we evaluate the surrogate-derived model against the latent gold-standard outcome in the target population. We propose cross-fitted estimators that transport information from the two labeled sources through source-specific density ratios. We also combine outcome-regression augmentation with a kernel correction for estimating the model near a threshold, accounting for uncertainty from all three samples. We establish asymptotically linear inference for TPR and FPR, consistency and pointwise inference for the ROC curve, and asymptotically normal inference for AUC. Simulations assess bias, coverage, and sensitivity to bandwidth and relative sample sizes. A retrospective temporal validation on Chatbot Arena and a semi-synthetic ACS-Income study provide validation in real-world AI applications.

Figures & tables

Panel of Figure 1 — Figure 1: Three-population setting. Gold and surrogate labels are observed in separate sources; only covariates are observed in the target population. Each source may have its own covariate shift relative to the target (dashed arrows).

Figure 2: Study 1 Standardized TPR errors at $c=0.5$ . Dashed lines mark $\pm 1.96$ , and outliers are shown.

Figure 3: Study 3 empirical 95% coverage over 500 replications. Coverage, rather than the magnitude of the performance metric, is the target of this comparison.

Figure 4: Natural temporal shift in Chatbot Arena. Each stacked bar is the distribution of candidate-model family appearances within a fixed cohort; quantitative overlap diagnostics appear in Appendix Figure F.1 .

Figure 5: ACS gold-label validation for the spline-3 score. Only Full carries the proposed three-source interval; the red line and band evaluate a different pooled cross-fitted finite-score object after the Target-outcome reveal. The paired Full-minus-human contrast is 0.028 [0.004, 0.051].

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

DGP	Source	Domain AUC	Mean wt.	Mean max.	ESS
DGP-A	HL	0.689	1.002	7.800	2462
DGP-A	AL	0.688	1.001	7.759	9913
DGP-B	HL	0.713	1.000	8.814	2096
DGP-B	AL	0.659	1.000	6.563	11447

Appendix

Table E.1: Study 3 cross-fitted density-ratio diagnostics, averaged over 500 replications. ESS is

(\sum_{i}w_{i})^{2}/\sum_{i}w_{i}^{2}

Design	DGP-A	DGP-B
P1	0.0295/0.0281	0.0323/0.0292
P2	0.0208/0.0205	0.0233/0.0216
P3	0.0155/0.0151	0.0161/0.0161
HL 0.25k	0.0354/0.0343	0.0333/0.0328
HL 2k	0.0188/0.0175	0.0214/0.0194
AL 1k	0.0304/0.0271	0.0385/0.0301

Appendix

Table E.2: Study 1 TPR empirical SD/mean analytical SE at

c=0.5

for the proposed Full estimator. Non-proportional designs change one P2 sample size at a time.

Figure E.1: Study 1 standardized FPR errors at $c=0.5$ . Full is the proposed estimator; IW-HL is importance-weighted gold-label validation; TSC is transported score calibration. Colors distinguish DGPs. HL, AL, and T denote the gold-labeled, surrogate-labeled, and target samples; dashed lines mark $\pm 1.96$ , and outliers are shown.

Panel of Figure 9 — Figure E.2: Study 2 standardized errors at $c=0.5$ . The five Full groups vary $C$ ; the $h$ -free IW-HL and TSC references are shown once. Colors distinguish DGPs, hatches distinguish methods, dashed lines mark $\pm 1.96$ , and outliers are shown.

Figure E.3: Normal Q–Q diagnostics for the proposed Full estimator under DGP-A at P2. TPR and FPR use $c=0.5$ , ROC uses $u=0.2$ , and AUC is the scalar one-step estimator.

Study	Target	Coordinate	Coverage range	SE/SD range	Worst
Study 1	TPR	$c=0.2$	.896–.944	.826–1.029	B/AL 1k (.896)
Study 1	FPR	$c=0.2$	.662–.930	.496–.916	B/AL 1k (.662)
Study 1	TPR	$c=0.5$	.872–.952	.782–1.014	B/AL 1k (.872)
Study 1	FPR	$c=0.5$	.914–.960	.883–1.022	A/AL 1k (.914)
Study 2	TPR	$c=0.2$	.904–.946	.876–1.007	B/ $C=2$ (.904)
Study 2	FPR	$c=0.2$	.580–.928	.434–.952	B/ $C=2$ (.580)

Appendix

Table E.3: Empirical coverage and SE/SD of the proposed Full construction over 500 replications. Study 3 rows select the Full ablation member; HGB rows use the same estimator with histogram-gradient-boosting nuisances. “Worst” gives the configuration attaining minimum coverage.

Target/design	Method	Cells	Mean $\|$ bias $\|$	Mean RMSE	Mean SE/SD	Coverage
S1 TPR(.5), all	Full	20	.0020	.0250	.950	.934 [.872,.952]
	IW-HL	20	.0041	.0284	.821	.879 [.668,.930]
	TSC	20	.0056	.0230	.701	.805 [.520,.928]
S1 FPR(.5), all	Full	20	.0020	.0217	.975	.941 [.914,.960]
	IW-HL	20	.0030	.0238	.842	.887 [.686,.948]
	TSC	20	.0031	.0192	.770	.850 [.600,.920]

Appendix

Table E.4: Comparison with fixed-score transported validation. Entries average over the stated design-by-DGP cells. Coverage is mean [minimum, maximum]. Study 2 compares the default Full choice

C=1

with the

h

-free comparators.

DGP	Estimator	TPR(.5)	FPR(.5)	ROC(.1)	AUC
DGP-A	Plug-in	$-.15/.010/.002/.260$	$-.02/.010/.002/.258$	$-.09/.012/.001/.112$	$-.08/.008/{<}.001/.100$
DGP-A	HL-only	$-.18/.012/.010/.904$	$+.02/.012/.010/.892$	$+.02/.018/.017/.940$	$-.05/.010/.010/.942$
DGP-A	AL-only	$-.09/.016/.014/.936$	$-.14/.016/.015/.926$	$+.13/.014/.006/.616$	$+.03/.008/.001/.298$
DGP-A	Full	$-.12/.017/.017/.956$	$-.10/.017/.018/.950$	$+.25/.019/.018/.938$	$+.06/.010/.010/.926$
DGP-A	Oracle nuis.	$-.15/.017/.017/.960$	$-.08/.017/.018/.948$	$+.16/.018/.018/.934$	$-.01/.010/.010/.946$
DGP-B	Plug-in	$+.28/.012/.002/.296$	$+.17/.007/.001/.234$	$-.03/.013/.001/.138$	$-.13/.009/.001/.104$

Appendix

Table E.5: Unified Study 3 ablation. Entries are bias (percentage points)/empirical SD/mean SE/95% coverage.

Figure E.4: Confirmed P4 Study 3 diagnostics at $c=0.5$ , $u=0.1$ , and $C=0.5$ . The top row reports bias in percentage points; the bottom row reports mean analytical SE divided by empirical SD.

Figure E.5: Study 4 standardized AUC and fixed- $u$ ROC errors.

Figure E.6: Regularized full quadratic-dictionary diagnostic under DGP-D over 500 replications. The $4\times$ P2 design is $(n_{\mathrm{HL}},n_{\mathrm{AL}},n_{\mathrm{T}})=(4000,16000,80000)$ . Gray coverage bands show the Monte Carlo 95% neighborhood of nominal 0.95 coverage; dashed lines mark the ideal value in each panel. All coordinates, including the weaker lower-threshold results, are displayed.

Cohort	Calendar window	$n$	Visible labels
HL	Apr. 24–May 7, 2023	2,816	$X,Y$
AL	May 7–26, 2023	3,590	$X,S$
Target	May 26–Jun. 26, 2023	1,499	$X$ before reveal

Appendix

Table F.1: Fixed temporal cohorts. Intervals are UTC and follow the left-closed, right-open convention except at the observed endpoint.

Figure F.1: Outcome-blind temporal-shift diagnostics. (a) Candidate-family shares across the fixed calendar cohorts. (b) OOF domain discrimination and uncapped effective-sample-size fractions for each transport. Domain AUC measures separability and is not a positivity proof. These diagnostics show that the calendar cohorts differ in observed covariates without imposed tilting; they do not assess estimator accuracy.

Diagnostic	HL $\to$ T	AL $\to$ T
Domain AUC	0.802	0.628
Uncapped ESS	427.9	2,356.7
ESS/source $n$	0.152	0.656
Weight $p_{99}$	9.43	3.69
Maximum weight	55.12	7.40
Guard fraction	0.000	0.000

Appendix

Table F.2: Outcome-blind overlap diagnostics for the semantic-plus-metadata density design. Primary ratios are uncapped; caps are sensitivities only.

A. Population-score estimation
Object	AUC [95% interval]
Outcome regression (g-computation)	0.796 [0.786,0.807]
Importance-weighted human-label score	0.841 [0.803,0.878]
Human-label correction only	0.825 [0.793,0.857]
LLM-label correction only	0.819 [0.805,0.833]
Full three-source (ours)	0.848 [0.814,0.881]

Appendix

Table F.3: Chatbot Arena scalar-AUC results. Panel A concerns the population surrogate-score estimand. Panel B evaluates frozen finite-sample scores against human Target outcomes revealed after estimation. The panels have different estimands and should not be read as a method ranking.

Component	Diagnostic	Value/status
$\hat{m}_{Y}$ (HL)	OOF AUC / Brier	0.814 / 0.178
$\hat{m}_{S}$ (AL)	OOF AUC / Brier	0.869 / 0.149
$\hat{m}_{S}$	Swap-complement error	$1.5\times 10^{-16}$
Target score	Unique / $n$ ; max. atom	1,499/1,499 / 0.00067
ROC(.10)	Local count / kernel ESS	734 / 1,029.0
ROC(.15)	Local count / kernel ESS	632 / 905.9

Appendix

Table F.4: Nuisance and local-information diagnostics. Pointwise inference is reported only when every stated criterion is satisfied.

Comparison	Estimate	95% interval
Full $-$ outcome regression	0.051	[0.020,0.083]
Full $-$ human-label correction	0.023	[0.014,0.031]
All-LLM-label deployment $-$ cross-fitted score	0.002	[-0.002,0.007]
Cross-fitted score $-$ Full	$-0.038$	[-0.061,-0.016]
All-LLM-label deployment $-$ Full	$-0.036$	[-0.058,-0.014]

Appendix

Table F.5: Paired comparisons and cross-estimand discrepancies. Analytic population-model comparisons and conditional target-only bootstrap comparisons have different uncertainty scopes.

Figure F.2: Low-FPR target ROC results. Solid curves are the two frozen finite-score empirical validations; square markers and error bars are the Full population-model estimates, for which all stated pointwise diagnostics were satisfied. The figure checks operating-point scale and behavior, but the curves and markers target different estimands and therefore do not rank methods or measure bias.

Figure F.3: Prespecified scalar-AUC sensitivities for the Full estimator. The uncapped analysis is primary; the capped-weight and pilot-bandwidth results are sensitivity analyses specified before Target reveal. Their stability concerns numerical sensitivity only and does not establish identification or transportability.

Cohort	$n$	Mean age	Age $>45$	Label prevalence
HL	447	50.79	75.6%	$227/447=0.508$
AL	552	35.78	15.4%	$244/552=0.442$
Target	186,193	42.93	43.8%	$81,674/186,193=0.4387$

Appendix

Table F.6: ACS analytical cohorts after opposing age-biased sampling.

Ratio	Median	Maximum	ESS	ESS fraction
$\hat{w}_{\mathrm{HL}}$	0.816	4.730	332.8	0.745
$\hat{w}_{\mathrm{AL}}$	0.864	3.452	438.2	0.794

Appendix

Table F.7: ACS held-out density-ratio weight diagnostics.

Metric	Plug-in	Human correction	Full [95% interval]	CF validation
AUC	0.707	0.838	0.866 [0.822, 0.911]	0.855
TPR at FPR .10	0.293	0.536	0.585 [0.458, 0.713]	0.577
TPR at FPR .15	0.391	0.631	0.652 [0.532, 0.772]	0.682
TPR at FPR .20	0.472	0.692	0.725 [0.610, 0.841]	0.754

Appendix

Table F.8: ACS-Income estimates using the cross-fitted spline-3 logistic score. Full uses the proposed three-source analytic interval under the stated regularity conditions. CF values evaluate a different finite-sample score against revealed Target outcomes.

Figure F.4: Complete ACS comparison for the spline-3 score. Gray, orange, and blue denote the plug-in, human-correction-only, and Full estimates. Only Full carries the proposed three-source intervals; the ablations are shown as point estimates. The red line and band evaluate pooled cross-fitted finite-sample scores after Target-outcome reveal and do not represent population-score truth. Full is descriptively closer than the human-correction-only ablation at all four reported metrics.

Learner	AL log loss	Full	Human corr.	CF validation	Full $-$ Human [95% interval]
Spline-3 logistic	0.2966	0.8663	0.8383	0.8549	0.0280 [0.0044, 0.0515]
Elastic-net logistic	0.2992	0.8632	0.8405	0.8543	0.0226 [0.0034, 0.0419]
Spline-5 logistic	0.3059	0.8638	0.8302	0.8517	0.0336 [0.0098, 0.0573]
Calibrated RBF-SVC	0.3164	0.8618	0.8307	0.8486	0.0311 [0.0112, 0.0511]
Ridge logistic	0.3178	0.8616	0.8354	0.8493	0.0261 [0.0058, 0.0464]
Calibrated quadratic-SVC	0.3331	0.8745	0.8394	0.8468	0.0351 [0.0125, 0.0577]

Appendix

Table F.9: Post-reveal exploratory ACS surrogate-learner sensitivity. Learners are ordered by AL out-of-fold log loss. The paired intervals concern Full minus the human-correction-only ablation for the same learner; they are not simultaneous confidence intervals or a direct ranking of estimator accuracy.

Figure F.5: Paired Full-minus-human-correction AUC contrasts across the six ACS surrogate learners, ordered by AL out-of-fold log loss. Bars are learner-specific 95% intervals; all remain above zero. The intervals are learner-specific and do not provide an inferential ranking of learner accuracy.

Distribution Shift Robustness

Distribution shift robustness studies how to keep machine-learning predictions reliable when deployment data differ from training data.

See all 274 papers

Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias
Jun 12, 2026Annie Ulichney, Amanda CostonDouble ML Distribution Shift Robustness
Department of Statistics, University of California, Berkeley
Conformal Calibration Transfer
Sep 9, 2026Achref DoulaWeighted Conformal Prediction Conformal Prediction
Technical University of Darmstadt, Darmstadt, Germany · Tongji University, Shanghai, China.
Transfer Learning for Meta-analysis Under Covariate Shift
Apr 3, 2026Zilong Wang, Ali Abdeen, Turgay AyerCausal Inference Individualized Treatment Effect Estimation
Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia, USA

Jun 12, 2026J/K move · Enter open · S save

Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias

Annie Ulichney, Amanda Coston

Department of Statistics, University of California, Berkeley

Double ML Distribution Shift Robustness Distribution Shift Covariate Shift Selection Bias Doubly Robust Estimation+1

Open paper AskPDF arXiv

Understanding how a prediction model will perform in a new environment before deployment is essential to preventing harm when algorithms inform decision-making. Two common sources of model performance degradation are (i) covariate shift, where the target covariate distribution differs from the source, and (ii) selective labels, where the observability of outcomes depends on historical decisions. We study pre-deployment model evaluation under the joint presence of covariate shift and labeling of outcomes selectively based on observed features. In particular, we present a double machine learning procedure for estimating the target risk of an arbitrary black-box prediction model under a general loss function. We show identification of this estimand under standard assumptions and derive a bias-corrected estimator based on the influence function of the target risk. Finally, we evaluate our estimator through experiments using the eICU electronic health records database, showing that it tracks the true target risk more accurately than methods that address either selective labels or covariate shift alone, as well as baselines that combine standard plug-in approaches.

Abstract

Distribution Shift Robustness

Explore similar work

Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias

Conformal Calibration Transfer

Transfer Learning for Meta-analysis Under Covariate Shift