Reward models used for off-policy evaluation (OPE) are commonly selected using held-out data. When sample size and context dimension grow proportionally, the difference in test risk between models fitted on the full sample and on a fixed fraction of it need not vanish. Held-out evaluation can therefore consistently estimate the test risk of the model trained on the smaller sample, but because the relative ranking of candidate reward models can change after full-data refitting, held-out selection can be suboptimal. We develop an Onsager-corrected, target-policy-weighted criterion that estimates the full-data reward model's out-of-sample risk under the target policy using only its training observations. This enables early stopping along the full-data gradient-descent trajectory without sample splitting. For neural networks with fixed depth and hidden width under Gaussian contexts, single-index rewards, context-dependent multi-index target policies, and known context-independent logging, we prove uniform consistency of our criterion over a fixed finite set of gradient-descent iterates and a near-oracle guarantee for the selected reward model. The same correction yields a doubly robust estimator that consistently estimates the true policy value without sample splitting, even when reward-model prediction error does not vanish. Experiments show decreasing risk-estimation error as dimension grows at a fixed sample-to-dimension ratio, and improved reward-model selection relative to held-out baselines. Empirically, our correction also reduces the root mean squared error of doubly robust value estimates computed from the same data used to fit and select the reward models.
Figures & tables
Figure 1: Risk estimation and stopping-time selection with m=2d and 200 datasets per dimension. Left: RMSE in estimating the full-data models’ conditional arm-wise test risks, with squared errors averaged over arms, candidate iterations, and datasets. Right: mean total target-policy prediction-risk regret of the selected full-data model, on a logarithmic scale. Bars give pointwise 95% Monte Carlo intervals.
Figure 2: Effect of target-policy weighting on stopping decisions for identical full-data trajectories. Points show the mean paired difference across datasets in selection regret between the corrected weighted criterion and the corrected unweighted criterion (left) or target-weighted ten-fold CV (right). Negative values favor the corrected weighted criterion. Bars give pointwise 95% Monte Carlo intervals over 200 datasets per dimension. All four target policies are shown.
Figure 3: Value estimation for identical reward models selected by the corrected criterion, with m=2d . Left: empirical bias of DR and ADR relative to the true target-policy value. Right: RMSE of DM, ADM, and cross-arm DM relative to the population value induced by the fitted model in each dataset. Bars give pointwise 95% Monte Carlo intervals over 200 datasets per dimension.
Figure 4: Policy-value RMSE with m=2d . ADR and uncorrected same-sample DR use the same full-data fits selected by the corrected risk criterion. CF10 uses either corrected stopping on each complete training complement or inner validation followed by refitting. Bars give pointwise 95% Monte Carlo intervals over the same 200 datasets per dimension.
Figure 5: Policy-value MSE normalized by σeff2/m . Left: fixed dimension d=256 with increasing sample size. Right: increasing dimension and sample-to-dimension ratio. The dashed line marks the oracle DR benchmark. Bars give pointwise 95% Monte Carlo intervals.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
d
Corrected
Ten-fold CV
h=⌈m⌉
h=⌈m2/3⌉
128
0.3466 / 0.0274
0.4100 / 0.0393
1.2601 / 0.3710
0.8611 / 0.2505
256
0.2481 / 0.0141
0.2898 / 0.0186
1.0045 / 0.2128
0.6390 / 0.1207
512
0.1695 / 0.0095
0.2155 / 0.0130
0.7814 / 0.1738
0.4726 / 0.0512
1024
0.1161 / 0.0043
0.1726 / 0.0093
0.6772 / 0.0991
0.3782 / 0.0296
2048
0.0811 / 0.0025
0.1537 / 0.0082
0.5527 / 0.0621
0.3017 / 0.0191
Appendix
Table 1: Risk estimation and full-data stopping selection with m=2d . Each entry yields risk-estimation RMSE / mean selection regret over the same 200 datasets per dimension.
d
Inner validation
Corrected stopping
Difference [ 95% interval]
128
0.8242
0.4754
−0.3488[−0.3735,−0.3241]
256
0.7276
0.4703
−0.2574[−0.2814,−0.2334]
512
0.6874
0.4777
−0.2097[−0.2273,−0.1922]
1024
0.6173
0.4678
−0.1495[−0.1640,−0.1350]
2048
0.5828
0.4705
−0.1123[−0.1258,−0.0988]
Appendix
Table 2: Prediction error of the selected complement-trained models with m=2d . The last column is corrected stopping minus inner validation, with a paired pointwise 95% Monte Carlo interval over 200 datasets.
d
Method
Squared bias
Variance
MSE
95% interval for MSE
128
ADR
0.00007
1.352
1.352
[1.072,1.632]
CF10
0.00134
1.398
1.399
[1.117,1.681]
IPS
0.00106
1.364
1.365
[1.088,1.643]
SNIPS
0.00163
1.334
1.335
[1.062,1.609]
256
ADR
0.00058
1.491
1.491
[1.203,1.779]
CF10
0.00031
1.625
1.625
[1.310,1.939]
Appendix
Table 3: Empirical bias–variance decomposition for policy-value estimation with m=2d and 200 datasets per dimension. CF10 uses corrected stopping. Squared bias, variance, and MSE are multiplied by m . The final column gives pointwise 95% Monte Carlo intervals for MSE; squared bias and variance are descriptive empirical components, defined in the text.
Figure 6: Importance-weighted arm-wise risk along training at d=512 and m=1024 . Lines are averages over 200 datasets; the test-risk reference includes reward noise and is conditional on each fitted network. Shading gives pointwise 95% Monte Carlo intervals across datasets.
Figure 7: Prediction discrepancy from a network retrained after omitting the evaluated observation. Every layer is retrained, with the original arm denominator Na retained. At each dimension, we use 100 datasets and sample 32 rows per dataset. Lines show root mean squared discrepancy over rows and datasets; shading gives pointwise 95% Monte Carlo intervals.
Figure 8: Risk estimation, stopping-time selection, and value bias under context-independent logging, with μ(2)=1−μ(1) , m=2d , and 200 datasets per setting. Top: root mean squared error in estimating the full-data models’ total target-policy test risk, with squared errors averaged over all pairs of stopping times and datasets. Middle: mean total prediction-risk regret of the selected full-data models. These two rows use logarithmic vertical axes. Bottom: empirical bias of DR and ADR at identical models and stopping times selected by the corrected criterion. Bars give pointwise 95% Monte Carlo intervals across datasets.
Figure 9: Risk estimation and value estimation under three context distributions, with m=2d and 200 datasets per setting. Top: root mean squared error in estimating the full-data models’ conditional arm-wise test risks, with squared errors averaged over both arms, all candidate iterations, and datasets. These risks include reward noise and use wa(X)=2ν(a∣X) ; the vertical axis is logarithmic. Bottom: empirical bias of DR and ADR relative to the true policy value, at identical models and stopping times selected by the corrected criterion. Bars give pointwise 95% Monte Carlo intervals across datasets.
Context law
d
Corrected regret
Corrected minus CV
95% interval
Gaussian
256
0.0141(0.0020)
−0.0045
[−0.0082,−0.0008]
1024
0.0043(0.0007)
−0.0050
[−0.0070,−0.0031]
2048
0.0025(0.0005)
−0.0057
[−0.0076,−0.0038]
Rademacher
256
0.0138(0.0018)
−0.0066
[−0.0106,−0.0025]
1024
0.0054(0.0007)
−0.0063
[−0.0086,−0.0041]
2048
0.0042(0.0006)
−0.0059
[−0.0077,−0.0041]
Appendix
Table 4: Stopping-time selection under three context distributions, with m=2d and 200 datasets per setting. Regret is measured by the total target-policy prediction risk Rν∘ relative to the best pair of stopping times on the full-data trajectories. Parentheses give Monte Carlo standard errors. The last two columns report the paired regret difference between the corrected rule and ten-fold cross-validation, with pointwise 95% Monte Carlo intervals.
d
m
ADR
Ten-fold CV + refit
CF10 (inner validation)
CF10 (corrected stopping)
512
1024
2.55
1.44
1.34
13.97
2048
4096
8.15
16.68
13.26
66.74
1024
32768
51.84
86.02
133.29
462.31
Appendix
Table 5: Median runtime in seconds for complete selection and value estimation on an Intel Xeon E5-2680 v3. Both CF10 procedures exclude each evaluation fold from fitting and selection. CV includes full-data refitting and raw DR scoring.
d
m
ADR
Ten-fold CV + refit
Holdout + refit h=⌈m⌉
Holdout + refit h=⌈m2/3⌉
512
1024
1.66
0.71
0.13
0.14
2048
4096
7.99
12.97
1.82
1.69
1024
32768
54.68
71.61
13.61
14.02
Appendix
Table 6: Median runtime in seconds in a separate matched holdout study on an Intel Xeon Gold 6230R. All procedures include selection and final scoring; CV and holdouts also include full-data refitting.
Off-policy evaluation (OPE) for contextual bandit policies becomes challenging when action-level importance weighting incurs excessive variance. Doubly robust (DR) estimation remains unbiased under common support but retains these high-variance action-level weights. A prior estimator, Off-policy evaluation with Conjunct Effect Model (OffCEM), replaces them with more stable cluster-level weights, at the cost of relying on local correctness of the reward model. In this paper, we show that, under the assumptions required by DR and OffCEM, there exists an unbiased family of estimators that interpolates between OffCEM and DR. Building on this result, we propose the Variance Optimal-CEM (VOCEM) estimator, which selects the interpolation coefficient to minimize variance. We derive the population-optimal coefficient in closed form and show that the resulting estimator has variance no larger than either endpoint, OffCEM or DR. Experiments in controlled synthetic settings and on two large-action benchmarks show that VOCEM improves upon both endpoints in all 23 evaluated conditions, exhibiting greater stability and empirical robustness.
Nicolò Felicioni, Michael Benigni, Maurizio Ferrari Dacrema +1
We study off-policy evaluation in the setting of contextual bandits, where we aim to evaluate a new policy using historical data that consists of contexts, actions and received rewards. This historical data typically does not faithfully represent action distribution of the new policy accurately. A common approach, inverse probability weighting (IPW), adjusts for these discrepancies in action distributions. However, this method often suffers from high variance due to the probability being in the denominator. The doubly robust (DR) estimator reduces variance through modeling reward but does not directly address variance from IPW. In this work, we address the limitation of IPW by proposing a Nonparametric Weighting (NW) approach that constructs weights using a nonparametric model. Our NW approach achieves low bias like IPW but typically exhibits significantly lower variance. To further reduce variance, we incorporate reward predictions -- similar to the DR technique -- resulting in the Model-assisted Nonparametric Weighting (MNW) approach. The MNW approach yields accurate value estimates by explicitly modeling and mitigating bias from reward modeling, without aiming to guarantee the standard doubly robust property. Extensive empirical comparisons show that our approaches consistently outperform existing techniques, achieving lower variance in value estimation while maintaining low bias.
Rong J. B. Zhu
Institute of Science and Technology for Brain-inspired Intelligence, Fudan University
Off-policy evaluation (OPE) is essential for assessing decision policies before deployment when online experimentation is costly or unsafe. Accurate estimation remains difficult when behavior-policy data are limited and the target policy favors states and actions rarely observed in those data. We propose InCOPE, a prior-data fitted network that learns a shared policy-value prediction rule across synthetic OPE tasks. We construct an OPE task prior over Markov decision processes and behavior-target policy pairs, generating behavior-policy datasets and exact target-policy values for pretraining. For a new tabular decision problem, the pretrained network uses the behavior-policy dataset, target policy, and known task settings as context to predict the policy's expected return without task-specific parameter updates. Experiments across four tabular domains, including a semi-synthetic clinical setting, demonstrate InCOPE's effectiveness for policy-value estimation.
Shuguang Yu, Ting Li, Fan Zhou
The Hong Kong Polytechnic University, Hong Kong, China · Shanghai University of Finance and Economics, Shanghai, China