The First-Order Value of Free Covariates in Budgeted Risk Monitoring

Yanze Cao

cs.LGOct 31, 2026

The First-Order Value of Free Covariates in Budgeted Risk Monitoring

Authors: Yanze Cao

Organizations: Shaanxi University of Technology

Abstract

A deployed model sees its inputs for free, but checking whether its risk has risen requires costly labels. We study how to spend a label budget so that a risk increase is detected quickly while false alarms stay controlled, including under input shifts that keep the risk below a safety threshold $τ$ . Let $A$ be the false-alarm control scale and $h(r)=\mathrm{kl}(r,τ)$ the Bernoulli KL divergence between the changed risk $r$ and $τ$ . At label rate $b=c/\log A$ , we prove that every rule that chooses requests from its own label history has worst-case (Lorden) delay at least $(1/h(r)+1/c)\log A$ to first order: besides collecting evidence, it must wait for its next label. The bound needs only a label budget on one finite stretch of safe source data and false-alarm control over an initial window; a bound on the average time to false alarm does not suffice. When a harmful change also shifts the inputs, a rule that lets the inputs trigger labeling attains delay $\log A/h(r)$ to first order on a fixed Gaussian model, so free covariates save at least $1/b$ rounds. A hybrid rule adds a label-history fallback for risk changes that leave the inputs unchanged. Synthetic experiments and two ACS tasks, with strengthened label-only baselines, show the predicted pattern. The covariate trigger has a cost: because its budget is calibrated on the source, a benign input shift can make it request nearly every label.

Figures & tables

Figure 1: One path, two successive costs. Before the first changed label, the observations agree with a safe reference. The first label costs at most

\kappa_{\tau}

in stopped KL, and each later calendar round at most

h(r)

. Lorden’s criterion allows conditioning on the history just after a request.

Policy	Covariate shift	Pure label change
L	$\log A/h+\Theta(1/b)$	$\log A/h+\Theta(1/b)$
C	$\log A/h+O(\mathcal{E})$	No label fallback
Hybrid	$\log A/h+O(\mathcal{E})$	$\log A/h+O(1/b_{L})$

Table 1: Delay scales as

b\downarrow 0

with

A\geq b^{-3}

\log\log A=o(1/b)

and fixed positive channel fractions (Corollary E.2 ). The unit waiting coefficient in Theorem 2 (a) uses

A\geq b^{-5}

. The covariate column uses the fixed Gaussian harmful shift.

Figure 2: Excess mean delay over full audit in the synthetic SR study at

\nu=1

. The

1/b

line from Theorem 2 is shown for scale;

b

times the excess is 33–39.

		Natural harmful input shifts			Input-preserving loss flips
Task	$b$	C delay	Best DE/C	H task /C	L	C	H task
Income	1%	1,063–2,030	3.44–3.57	0.995–0.999	4,906 (0)	136,060 (153)	9,614 (0)
Income	0.5%	1,131–2,118	5.65–5.88	1.013–1.024	8,417 (0)	171,413 (316)	17,597 (0)
Income	0.2%	1,245–2,230	12.15–12.92	1.006–1.012	18,957 (0)	193,622 (458)	41,934 (4)
TravelTime	1%	838–1,029	4.43–4.61	1.002–1.003	11,052 (0)	191,466 (411)	18,634 (0)
TravelTime	0.5%	847–1,042	7.49–7.61	1.001–1.002	18,211 (0)	199,508 (493)	36,372 (0)

Table 2: Cross-task results at

A=10^{6}

. Natural-shift ranges cover the three harmful targets of each task. Best DE/C uses the post hoc best of three prespecified DE designs, so no interval is given for the selected design. Flip cells give restricted mean rounds with censored paths out of 500 in parentheses. H task is the capped H D supplement for Income and the uncapped Hybrid for TravelTime; L is each task's primary label controller. At Income 1%, H D can gather label evidence before C activates, giving ratios slightly below one; only SD’s pointwise paired 95% interval excludes 1 (Appendix G.9 ).

Appendix figures & tables38 assets

Supplementary material from the paper’s appendix.

Appendix

$A$	$J_{A}$	G1 bound	Minimum $g/h$
1e+04	41	0.902060	0.997381154
1e+06	60	0.932503	0.998794830
1e+08	79	0.948508	0.999310467

Appendix

Table 3: Growing-grid guarantees and numerical minima for the fixed design interval. Numerical minima are floating-point evaluations, not certified lower bounds.

Budget	SR L	SR C	CUSUM L	CUSUM C
1.0%	500/500	500/500	3/500	3/500
0.5%	500/500	500/500	1/500	0/500
0.2%	500/500	500/500	0/500	0/500

Appendix

Table 4: No-change alarms among 500 streams by round 205,000 at

A=10^{4}

. Censoring prevents these counts from establishing actual ARL.

Budget	C whole	Reset whole	Reset final	Reset alarms
1.0%	99.871%	2.174%	0.702%	0/500
0.5%	99.850%	1.887%	0.400%	0/500
0.2%	99.786%	1.765%	0.151%	0/500

Appendix

Table 5: Benign-shift acquisition in DC at

A=10^{6}

. The final window and whole post-change window show the initial cost of resetting.

Figure 3: CUSUM benign-shift cost recovery and the unchanged-input label-flip stress test at

A=10^{6}

. C and C_reset have identical plotted delay values in the right panel, so that panel shows C once. Tables report whole-horizon acquisition costs and censoring.

Scenario	Comparison	Ratio [95% CI], $A=10^{4}$	Pairs
SD	Hybrid/C	1.026 [1.013, 1.041]	500
WV	Hybrid/C	1.020 [1.006, 1.036]	500
MT	Hybrid/C	1.023 [1.014, 1.034]	500
Label flip	Hybrid/L	1.919 [1.720, 2.134]	500

Appendix

Table 6: Hybrid supplement at the second threshold

A=10^{4}

, budget 0.5%. All 500 pairs survive the pre-change period.

Scenario	Pre-change rate	MC SE (pp)	Above 0.55%
SD	0.420%	0.056	False
WV	0.520%	0.064	False
MT	0.434%	0.056	False
Label flip	0.531%	0.071	False

Appendix

Table 7: Measured Hybrid source-prefix acquisition rates and Monte Carlo standard errors. The diagnostic flag threshold is 0.55%.

Target	Policy	Restricted delay	Censored	Pre-change rate
SD	F0	147,603.1	120	0.4906%
SD	F1	200,000.0	500	0.4920%
SD	L	6,800.4	0	0.5414%
SD	C	1,225.1	0	0.4086%
SD	Full	878.2	0	100.0000%
WV	F0	141,809.6	102	0.5062%

Appendix

Table 8: Five ACS policies at budget 0.5% and

A=10^{6}

. The censoring horizon is 200,000 post-change rounds.

Budget	SD: L/C	WV: L/C	MT: L/C
1.0%	5.653 [5.175, 6.136]	5.127 [4.692, 5.572]	5.564 [5.111, 6.044]
0.5%	8.262 [7.580, 8.979]	9.610 [8.674, 10.595]	10.412 [9.485, 11.355]
0.2%	19.350 [17.610, 21.198]	19.484 [17.704, 21.440]	22.169 [20.200, 24.174]

Appendix

Table 9: ACS harmful-shift delay ratios for the SR detector at

A=10^{6}

. Brackets give descriptive paired 95% bootstrap intervals.

Nominal channel budget	Sleep	Design-risk rate	Source-pool rate
1.0%	1,067	0.9993%	1.0155%
0.5%	2,144	0.4998%	0.5080%
0.2%	5,375	0.2000%	0.2032%
0.25% (Hybrid channel)	4,298	0.2500%	0.2540%

Appendix

Table 10: Sleep lengths and long-run label-channel audit rates for L N , computed from the renewal argument rather than measured.

Risk	L	L N	Hybrid	H N	C
0.210	977,023 (472)	103,511 (0)	989,544 (483)	146,702 (0)	999,181 (499)
0.215	585,064 (107)	61,774 (0)	765,428 (251)	95,387 (0)	998,892 (497)
0.230	41,367 (0)	28,456 (0)	74,331 (0)	50,348 (0)	914,112 (353)
0.250	14,451 (0)	16,536 (0)	25,553 (0)	34,360 (0)	522,283 (31)
0.27003	8,417 (0)	13,068 (0)	16,151 (0)	23,359 (0)	288,842 (4)

Appendix

Table 11: Input-preserving loss-flip scan at

A=10^{6}

, nominal budget 0.5%, and post-change horizon

H=10^{6}

. Entries are restricted mean delays in rounds, with censoring counts out of 500 in parentheses.

Risk	L	L N	Hybrid	H N	C
0.210	199,377 (496)	101,852 (18)	199,853 (498)	128,025 (102)	200,000 (500)
0.215	188,880 (432)	61,503 (3)	194,726 (466)	91,068 (33)	200,000 (500)

Appendix

Table 12: The two small-exceedance cells restricted to the original post-change horizon

H=200{,}000

, using the same stored runs. Entries are restricted means (censored streams out of 500).

Risk	L/L N	H N /L N
0.210	9.44 [9.04, 9.84]	1.42 [1.33, 1.51]
0.215	9.47 [8.80, 10.15]	1.54 [1.42, 1.67]
0.230	1.45 [1.31, 1.61]	1.77 [1.59, 1.97]
0.250	0.87 [0.79, 0.97]	2.08 [1.83, 2.35]
0.27003	0.64 [0.57, 0.72]	1.79 [1.58, 2.03]

Appendix

Table 13: Paired ratios of restricted mean delays for the scan at

H=10^{6}

, with pointwise 95% bootstrap intervals. Heavy censoring in the original controllers limits interpretation as uncensored mean-delay ratios.

State	Budget	L	L N	C	L N /L	L N /C
SD	1.0%	4,260	6,676	1,156	1.57 [1.41, 1.74]	5.78 [5.34, 6.22]
WV	1.0%	3,728	6,450	1,063	1.73 [1.56, 1.92]	6.07 [5.59, 6.54]
MT	1.0%	6,937	8,831	2,030	1.27 [1.16, 1.40]	4.35 [4.02, 4.71]
SD	0.5%	6,800	11,312	1,225	1.66 [1.49, 1.87]	9.23 [8.42, 10.07]
WV	0.5%	7,264	11,082	1,131	1.53 [1.36, 1.71]	9.80 [8.99, 10.69]
MT	0.5%	13,099	16,042	2,118	1.22 [1.09, 1.37]	7.57 [6.88, 8.26]

Appendix

Table 14: Interstate sensitivity at

A=10^{6}

and

H=200{,}000

. Delays are restricted means over 500 paired streams. Brackets give pointwise 95% paired bootstrap intervals. L and C are uncensored; L N has 2/500 censored streams in WV and MT at budget 0.2% and none elsewhere.

State	C	Hybrid	H N	H N /C
SD	1,225.1	1,253.0	1,257.2	1.026 [1.015, 1.039]
WV	1,130.9	1,154.1	1,160.8	1.027 [1.016, 1.039]
MT	2,118.4	2,153.6	2,148.3	1.014 [1.006, 1.023]

Appendix

Table 15: Hybrid sensitivity at nominal budget 0.5%,

A=10^{6}

and

H=200{,}000

. All 500 streams per cell are uncensored. Brackets give pointwise 95% paired bootstrap intervals.

Budget	Target	C	DE τ	DE τ+.02	DE τ+.06	Best/C
1%	SD	1,155.5	4,655.4	4,093.2	3,971.9	3.437
1%	WV	1,062.9	4,315.8	3,673.4	3,792.4	3.456
1%	MT	2,029.7	7,437.6	7,255.2	7,379.4	3.574
0.5%	SD	1,225.1	9,122.0	7,009.5	7,027.8	5.722
0.5%	WV	1,130.9	8,288.4	6,719.6	6,644.4	5.876
0.5%	MT	2,118.4	13,438.3	11,978.5	12,407.5	5.655

Appendix

Table 16: Strengthened label-history controls at

A=10^{6}

. Entries are restricted mean delays. “Best/C” is the post hoc minimum mean across the three fixed designs divided by C’s mean, an oracle envelope without a selected-baseline confidence interval.

Budget	Target	Uniform delay	C delay	Uniform censored	Uniform median
1%	SD	82,920.0	1,155.5	1/500 (0.2%)	80,110
1%	WV	80,486.1	1,062.9	0/500 (0.0%)	77,860
1%	MT	161,889.8	2,029.7	190/500 (38.0%)	178,865
0.5%	SD	155,557.0	1,225.1	152/500 (30.4%)	164,202
0.5%	WV	150,605.6	1,130.9	130/500 (26.0%)	153,936
0.5%	MT	194,496.5	2,118.4	432/500 (86.4%)	$\mathrm{NR}$

Appendix

Table 17: Uniform acquisition with the common CUSUM at

A=10^{6}

. Delays are restricted means through 200,000 post-change rounds, conditional on pre-change survival. C has no censored streams in these cells. NR means that the Kaplan–Meier median is not reached within the horizon.

Figure 4: Kaplan–Meier post-change survival curves at budget 0.5% and

A=10^{6}

, conditional on each policy’s pre-change survival. Curves compare the fixed DE τ design, Uniform and C. A plus at the horizon marks remaining administrative censoring. The horizontal axis is logarithmic.

Budget	DE τ delay	H D delay	H D /DE τ	H D censored
1%	5,272.9	9,614.0	1.823	0/500
0.5%	9,652.7	17,597.2	1.823	0/500
0.2%	22,424.2	41,934.2	1.870	4/500

Appendix

Table 18: The new capped H D allocation check at

A=10^{6}

. Entries are ratios of restricted means; this is distinct from original Hybrid and uncapped H N .

Target	H D /C	Pointwise 95% interval	Faster/equal/slower	Extra labels
SD	0.99553	[0.99219,0.99836]	16/475/9	5.146
WV	0.99504	[0.98909,1.00058]	20/460/20	7.716
MT	0.99904	[0.99694,1.00102]	8/479/13	4.728

Appendix

Table 19: Small finite-path differences for Income H D versus C. The interval estimates a ratio of paired mean delays, not a mean of per-path ratios. Faster/equal/slower counts compare paired alarm times. Extra labels are H D -only changed labels before C’s first changed-label request, averaged over all 500 paths.

Budget	Best label (12)	C	Label/C	Best DEfull	DEfull/C
1%	4,692	1,385	3.39	4,944	3.57
0.5%	8,457	1,308	6.46	8,785	6.71
0.2%	20,601	1,366	15.08	21,243	15.55

Appendix

Table 20: Synthetic delays with target trigger power 0.99. All-12 selects the fastest evaluated label controller; DEfull restricts the comparison to the exact-budget subfamily.

Pool	Records	Error risk	Mean shift	Proxy bias
CA	39133	17.147%	0.019	-0.208%
SD	2450	27.796%	0.682	-9.230%
WV	4052	28.036%	0.729	-10.215%
MT	2732	25.403%	0.752	-6.714%
DC	2094	14.852%	0.761	2.091%
OK	8959	23.005%	0.489	-5.490%

Appendix

Table 21: Evaluation pools selected before detection. Mean-shift scores summarize whitened means; proxy bias is error-prediction bias. All risks are unweighted finite-pool risks.

Policy	Acquisition	Detector input
F0	Uniform audit probability $b$	Source prediction plus IPW residual, globally safe betting scale
F1	Fixed source-trained input-dependent audit probability with floor $0.1b$	Same augmented form, scale limited by the global floor
L	Full-budget label-history controller	Full-audit factors when active; 1 when sleeping
C	Previous-block covariate trigger	Full-audit factors in high state; 1 otherwise
Full	Audit every label	Full-audit factors

Appendix

Table 22: ACS acquisition policies and detector inputs. L, C, Full, and Hybrid share their full-audit betting scale; F0 and F1 use propensity-corrected factors.

Budget	Target	L delay	C delay	L/C [95% CI]	Censored L/C
1.0%	SD	4,260.4	1,155.5	3.687 [3.392, 3.990]	0/0
1.0%	WV	3,727.6	1,062.9	3.507 [3.251, 3.789]	0/0
1.0%	MT	6,937.4	2,029.7	3.418 [3.180, 3.669]	0/0
0.5%	SD	6,800.4	1,225.1	5.551 [5.122, 6.019]	0/0
0.5%	WV	7,263.6	1,130.9	6.423 [5.896, 7.017]	0/0
0.5%	MT	13,099.1	2,118.4	6.183 [5.672, 6.674]	0/0

Appendix

Table 23: ACS harmful-shift results for CUSUM at

A=10^{6}

. Delays are restricted means; the last column reports censoring counts for L and C.

Figure 5: CUSUM restricted mean delay at

A=10^{6}

on harmful ACS targets. Label-only delay increases as the nominal source budget shrinks, while covariate-triggered delay remains comparatively stable. C and its reset variant coincide in all displayed cells; only C is plotted.

Budget	L delay	C delay	C censored
1.0%	4,906.1	136,060.0	153/500
0.5%	8,417.4	171,413.0	316/500
0.2%	18,957.4	193,621.7	458/500

Appendix

Table 24: Unchanged-input label perturbations at

A=10^{6}

: restricted delay and censoring for the label and covariate controllers.

Scenario	L delay	C delay	Hybrid delay	Censored L/C/H	Paired ratio [95% CI]
SD	6,800.4	1,225.1	1,253.0	0/0/0	Hybrid/C 1.023 [1.011, 1.037]
WV	7,263.6	1,130.9	1,154.1	0/0/0	Hybrid/C 1.021 [1.010, 1.032]
MT	13,099.1	2,118.4	2,153.6	0/0/0	Hybrid/C 1.017 [1.010, 1.024]
Label flip	8,417.4	171,413.0	16,150.8	0/316/0	Hybrid/L 1.919 [1.723, 2.125]

Appendix

Table 25: Hybrid at budget 0.5% and

A=10^{6}

. Brackets give paired 95% bootstrap intervals.

Figure 6: Parallel acquisition at budget 0.5% and

A=10^{6}

. All bars are restricted mean delays with a 200,000-round post-change horizon. The hatched C bar in the right panel includes 316/500 censored streams. Hybrid stays close to C on harmful interstate shifts and restores detection under unchanged-input label perturbations.

Policy	Threshold	Mean run length	Censored
L	27.13	153,625.0	0/500
C	22.78	191,621.0	0/500

Appendix

Table 26: Source run-length calibration at budget 0.5%: independent validation after targeting mean run length 200,000.

Pool	Records	Risk	Risk $-\tau$	Input shift
CA	34,502	30.946%	$-3.496$	Source
NE	4,767	43.675%	$+9.234$	2.121
KS	6,914	42.609%	$+8.168$	2.026
ND	1,930	43.472%	$+9.030$	2.256
AK	1,387	28.983%	$-5.458$	2.246

Appendix

Table 27: TravelTime evaluation pools, opened after the selection rule was locked. Risks are unweighted empirical losses. Exceedance is in percentage points. The shift is the norm of the saved whitened mean, not a risk statistic.

Label channel	$b=1\%$	$b=0.5\%$	$b=0.2\%$
L	1,521	3,057	7,665
DE τ	1,521	3,057	7,665
DE τ+.02	838	1,683	4,220
DE τ+.06	577	1,158	2,904
Hybrid L	3,057	6,129	15,346

Appendix

Table 28: TravelTime calibrated sleep rounds. L and Hybrid L are uncapped. Every DE keeps physical cap 10 (integer cap 100000); Hybrid L uses half the nominal budget.

Policy	$b=1\%$	$b=0.5\%$	$b=0.2\%$
NE
L (uncapped)	5,622.9 (0)	9,883.4 (0)	22,962.6 (0)
C	837.9 (0)	846.5 (0)	857.6 (0)
DE τ	5,622.9 (0)	9,883.4 (0)	22,962.6 (0)
DE τ+.02	4,194.0 (0)	8,606.2 (0)	21,992.5 (0)
DE τ+.06	3,714.6 (0)	6,419.7 (0)	16,681.7 (0)

Appendix

Table 29: All eight TravelTime policies on harmful and input-preserving scenarios at

A=10^{6}

. Entries are restricted means with censored paths out of 500 in parentheses. All cells have 500 pre-change survivors.

Scenario	$b$	RMST	Censored	KM median
NE	1%	77,356.2	1/500 (0.2%)	73,782
NE	0.5%	142,961.4	121/500 (24.2%)	145,247
NE	0.2%	193,591.5	446/500 (89.2%)	NR
KS	1%	101,390.4	8/500 (1.6%)	100,649
KS	0.5%	169,603.2	243/500 (48.6%)	198,754
KS	0.2%	198,461.0	483/500 (96.6%)	NR

Appendix

Table 30: TravelTime Uniform censoring and KM medians at the same 200,000-round horizon. NR denotes a median not reached within the horizon.

$b$	C labels	Hybrid labels	Difference [95% CI]
1%	199,968.4	199,968.1	-0.320 [-0.640, -0.064]
0.5%	199,960.2	199,960.1	-0.160 [-0.400,0.000]
0.2%	199,952.4	199,950.9	-1.496 [-2.202, -0.880]

Appendix

Table 31: AK post-change label counts through 200,000 rounds and Hybrid minus C, with paired 95% intervals. Near-zero differences do not imply low absolute labeling cost.

Figure 7: TravelTime no-alarm survival for the first locked harmful target NE. All eight fixed policies use the same CUSUM; Uniform retains substantial administrative censoring at low budgets.

Figure 8: TravelTime input-preserving flip survival. C loses power as its unchanged-input trigger becomes rare; L and Hybrid restore detection, with 49/500 Hybrid paths still censored at the lowest budget.

Budget	C delay	Best DE/C	Uniform censored
1%	1,063–2,030	3.44–3.57	0.0–38.0%
0.5%	1,131–2,118	5.65–5.88	26.0–86.4%
0.2%	1,245–2,230	12.15–12.92	92.0–99.4%

Appendix

Table 32: Income, all three harmful targets at

A=10^{6}

. Ranges run over SD, WV and MT. Best DE is the post hoc minimum over the three designs of Table 16 . Uniform uses the common detector and a 200,000-round censoring horizon.

$b$	Target	L	C	Best DE	Hybrid	Full	Cens.
1%	NE	5,623	838	3,715	841	803	0/0/0
1%	KS	6,619	1,029	4,739	1,031	985	0/0/0
1%	ND	5,896	904	4,096	907	867	0/0/0
1%	Flip	11,052	191,466	9,120	18,634	2,700	0/411/0
0.5%	NE	9,883	847	6,420	848	803	0/0/0
0.5%	KS	11,056	1,042	7,799	1,043	985	0/0/0

Appendix

Table 33: TravelTime at

A=10^{6}

: restricted mean post-change rounds through 200,000 rounds, with 500 paired paths per cell. Best DE is the post hoc minimum over the three fixed designs. Cens. gives L/C/Hybrid counts; all paths survive pre-change time. Uniform and every fixed DE, KM and cost result are in Appendix G.12 .

Feature	Cost-constrained design	Present model
Null	Specified nominal experiment laws	Conditional risk at most $\tau$ , including safe input changes
Observation	$E_{0}=X$ , $E_{1}=(X,L)$ embeds past-based requests	Also permits current-input-dependent requests
Resource comparison	Pre-change experiment fractions	Source-prefix allowance; one prefix in a stated range suffices at the critical budget
Relevant conclusion	Constructive delay comparisons	Policy-uniform additive converse and critical-budget separation

Appendix

Table 34: Observation and resource assumptions in cost-constrained experiment design and in the present model.

Change-Point Detection

Change-point detection identifies when the statistical behavior or underlying regime of a sequence changes.

See all 87 papers

Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias
Jun 12, 2026Annie Ulichney, Amanda CostonDouble ML Distribution Shift Robustness
Department of Statistics, University of California, Berkeley
Certify or Refuse: A Cross-Model Map for Selective Risk Control with Coverage Floors under Covariate Shift
Aug 11, 2026Jiamiao Liu, Dewen Qiao, Yu Zhang +1Selective Prediction Covariate Shift
Department of Information, Xinqiao Hospital Army Medical University (Third Military Medical University) Chongqing 400037, China
How Many Labels Does Model Choice Need? Certificates and Budgets for Selective Prediction
Sep 16, 2026Tetsuji KuboyamaModel Selection Selective Prediction
Computer Centre, Gakushuin University

Jun 12, 2026J/K move · Enter open · S save

Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias

Annie Ulichney, Amanda Coston

Department of Statistics, University of California, Berkeley

Double ML Distribution Shift Robustness Distribution Shift Covariate Shift Selection Bias Doubly Robust Estimation+1

Open paper AskPDF arXiv

Understanding how a prediction model will perform in a new environment before deployment is essential to preventing harm when algorithms inform decision-making. Two common sources of model performance degradation are (i) covariate shift, where the target covariate distribution differs from the source, and (ii) selective labels, where the observability of outcomes depends on historical decisions. We study pre-deployment model evaluation under the joint presence of covariate shift and labeling of outcomes selectively based on observed features. In particular, we present a double machine learning procedure for estimating the target risk of an arbitrary black-box prediction model under a general loss function. We show identification of this estimand under standard assumptions and derive a bias-corrected estimator based on the influence function of the target risk. Finally, we evaluate our estimator through experiments using the eICU electronic health records database, showing that it tracks the true target risk more accurately than methods that address either selective labels or covariate shift alone, as well as baselines that combine standard plug-in approaches.

Abstract

Change-Point Detection

Explore similar work

Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias

Certify or Refuse: A Cross-Model Map for Selective Risk Control with Coverage Floors under Covariate Shift

How Many Labels Does Model Choice Need? Certificates and Budgets for Selective Prediction