Reward models used for off-policy evaluation (OPE) are commonly selected using held-out data. When sample size and context dimension grow proportionally, the difference in test risk between models fitted on the full sample and on a fixed fraction of it need not vanish. Held-out evaluation can therefore consistently estimate the test risk of the model trained on the smaller sample, but because the relative ranking of candidate reward models can change after full-data refitting, held-out selection can be suboptimal. We develop an Onsager-corrected, target-policy-weighted criterion that estimates the full-data reward model's out-of-sample risk under the target policy using only its training observations. This enables early stopping along the full-data gradient-descent trajectory without sample splitting. For neural networks with fixed depth and hidden width under Gaussian contexts, single-index rewards, context-dependent multi-index target policies, and known context-independent logging, we prove uniform consistency of our criterion over a fixed finite set of gradient-descent iterates and a near-oracle guarantee for the selected reward model. The same correction yields a doubly robust estimator that consistently estimates the true policy value without sample splitting, even when reward-model prediction error does not vanish. Experiments show decreasing risk-estimation error as dimension grows at a fixed sample-to-dimension ratio, and improved reward-model selection relative to held-out baselines. Empirically, our correction also reduces the root mean squared error of doubly robust value estimates computed from the same data used to fit and select the reward models.
Figures & tables
Figure 1: Risk estimation and stopping-time selection with m=2d and 200 datasets per dimension. Left: RMSE in estimating the full-data models’ conditional arm-wise test risks, with squared errors averaged over arms, candidate iterations, and datasets. Right: mean total target-policy prediction-risk regret of the selected full-data model, on a logarithmic scale. Bars give pointwise 95% Monte Carlo intervals.
Figure 2: Effect of target-policy weighting on stopping decisions for identical full-data trajectories. Points show the mean paired difference across datasets in selection regret between the corrected weighted criterion and the corrected unweighted criterion (left) or target-weighted ten-fold CV (right). Negative values favor the corrected weighted criterion. Bars give pointwise 95% Monte Carlo intervals over 200 datasets per dimension. All four target policies are shown.
Figure 3: Value estimation for identical reward models selected by the corrected criterion, with m=2d . Left: empirical bias of DR and ADR relative to the true target-policy value. Right: RMSE of DM, ADM, and cross-arm DM relative to the population value induced by the fitted model in each dataset. Bars give pointwise 95% Monte Carlo intervals over 200 datasets per dimension.
Figure 4: Policy-value RMSE with m=2d . ADR and uncorrected same-sample DR use the same full-data fits selected by the corrected risk criterion. CF10 uses either corrected stopping on each complete training complement or inner validation followed by refitting. Bars give pointwise 95% Monte Carlo intervals over the same 200 datasets per dimension.
Figure 5: Policy-value MSE normalized by σeff2/m . Left: fixed dimension d=256 with increasing sample size. Right: increasing dimension and sample-to-dimension ratio. The dashed line marks the oracle DR benchmark. Bars give pointwise 95% Monte Carlo intervals.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
d
Corrected
Ten-fold CV
h=⌈m⌉
h=⌈m2/3⌉
128
0.3466 / 0.0274
0.4100 / 0.0393
1.2601 / 0.3710
0.8611 / 0.2505
256
0.2481 / 0.0141
0.2898 / 0.0186
1.0045 / 0.2128
0.6390 / 0.1207
512
0.1695 / 0.0095
0.2155 / 0.0130
0.7814 / 0.1738
0.4726 / 0.0512
1024
0.1161 / 0.0043
0.1726 / 0.0093
0.6772 / 0.0991
0.3782 / 0.0296
2048
0.0811 / 0.0025
0.1537 / 0.0082
0.5527 / 0.0621
0.3017 / 0.0191
Appendix
Table 1: Risk estimation and full-data stopping selection with m=2d . Each entry yields risk-estimation RMSE / mean selection regret over the same 200 datasets per dimension.
d
Inner validation
Corrected stopping
Difference [ 95% interval]
128
0.8242
0.4754
−0.3488[−0.3735,−0.3241]
256
0.7276
0.4703
−0.2574[−0.2814,−0.2334]
512
0.6874
0.4777
−0.2097[−0.2273,−0.1922]
1024
0.6173
0.4678
−0.1495[−0.1640,−0.1350]
2048
0.5828
0.4705
−0.1123[−0.1258,−0.0988]
Appendix
Table 2: Prediction error of the selected complement-trained models with m=2d . The last column is corrected stopping minus inner validation, with a paired pointwise 95% Monte Carlo interval over 200 datasets.
d
Method
Squared bias
Variance
MSE
95% interval for MSE
128
ADR
0.00007
1.352
1.352
[1.072,1.632]
CF10
0.00134
1.398
1.399
[1.117,1.681]
IPS
0.00106
1.364
1.365
[1.088,1.643]
SNIPS
0.00163
1.334
1.335
[1.062,1.609]
256
ADR
0.00058
1.491
1.491
[1.203,1.779]
CF10
0.00031
1.625
1.625
[1.310,1.939]
Appendix
Table 3: Empirical bias–variance decomposition for policy-value estimation with m=2d and 200 datasets per dimension. CF10 uses corrected stopping. Squared bias, variance, and MSE are multiplied by m . The final column gives pointwise 95% Monte Carlo intervals for MSE; squared bias and variance are descriptive empirical components, defined in the text.
Figure 6: Importance-weighted arm-wise risk along training at d=512 and m=1024 . Lines are averages over 200 datasets; the test-risk reference includes reward noise and is conditional on each fitted network. Shading gives pointwise 95% Monte Carlo intervals across datasets.
Figure 7: Prediction discrepancy from a network retrained after omitting the evaluated observation. Every layer is retrained, with the original arm denominator Na retained. At each dimension, we use 100 datasets and sample 32 rows per dataset. Lines show root mean squared discrepancy over rows and datasets; shading gives pointwise 95% Monte Carlo intervals.
Figure 8: Risk estimation, stopping-time selection, and value bias under context-independent logging, with μ(2)=1−μ(1) , m=2d , and 200 datasets per setting. Top: root mean squared error in estimating the full-data models’ total target-policy test risk, with squared errors averaged over all pairs of stopping times and datasets. Middle: mean total prediction-risk regret of the selected full-data models. These two rows use logarithmic vertical axes. Bottom: empirical bias of DR and ADR at identical models and stopping times selected by the corrected criterion. Bars give pointwise 95% Monte Carlo intervals across datasets.
Figure 9: Risk estimation and value estimation under three context distributions, with m=2d and 200 datasets per setting. Top: root mean squared error in estimating the full-data models’ conditional arm-wise test risks, with squared errors averaged over both arms, all candidate iterations, and datasets. These risks include reward noise and use wa(X)=2ν(a∣X) ; the vertical axis is logarithmic. Bottom: empirical bias of DR and ADR relative to the true policy value, at identical models and stopping times selected by the corrected criterion. Bars give pointwise 95% Monte Carlo intervals across datasets.
Context law
d
Corrected regret
Corrected minus CV
95% interval
Gaussian
256
0.0141(0.0020)
−0.0045
[−0.0082,−0.0008]
1024
0.0043(0.0007)
−0.0050
[−0.0070,−0.0031]
2048
0.0025(0.0005)
−0.0057
[−0.0076,−0.0038]
Rademacher
256
0.0138(0.0018)
−0.0066
[−0.0106,−0.0025]
1024
0.0054(0.0007)
−0.0063
[−0.0086,−0.0041]
2048
0.0042(0.0006)
−0.0059
[−0.0077,−0.0041]
Appendix
Table 4: Stopping-time selection under three context distributions, with m=2d and 200 datasets per setting. Regret is measured by the total target-policy prediction risk Rν∘ relative to the best pair of stopping times on the full-data trajectories. Parentheses give Monte Carlo standard errors. The last two columns report the paired regret difference between the corrected rule and ten-fold cross-validation, with pointwise 95% Monte Carlo intervals.
d
m
ADR
Ten-fold CV + refit
CF10 (inner validation)
CF10 (corrected stopping)
512
1024
2.55
1.44
1.34
13.97
2048
4096
8.15
16.68
13.26
66.74
1024
32768
51.84
86.02
133.29
462.31
Appendix
Table 5: Median runtime in seconds for complete selection and value estimation on an Intel Xeon E5-2680 v3. Both CF10 procedures exclude each evaluation fold from fitting and selection. CV includes full-data refitting and raw DR scoring.
d
m
ADR
Ten-fold CV + refit
Holdout + refit h=⌈m⌉
Holdout + refit h=⌈m2/3⌉
512
1024
1.66
0.71
0.13
0.14
2048
4096
7.99
12.97
1.82
1.69
1024
32768
54.68
71.61
13.61
14.02
Appendix
Table 6: Median runtime in seconds in a separate matched holdout study on an Intel Xeon Gold 6230R. All procedures include selection and final scoring; CV and holdouts also include full-data refitting.