Model Selection for Off-Policy Evaluation via Pairwise Bellman Validation

Lars van der Laan; Vasilis Syrgkanis; Nathan Kallus

stat.MLOct 26, 2026

Model Selection for Off-Policy Evaluation via Pairwise Bellman Validation

Authors: Lars van der Laan, Vasilis Syrgkanis, Nathan Kallus

Organizations: Stanford University · Netflix and Cornell University

Abstract

Model selection and hyperparameter tuning pose a central challenge in off-policy evaluation (OPE) because errors of fitted Q-functions and occupancy-ratio estimates cannot be directly evaluated from held-out transitions. Bellman-residual and minimax validation methods often require auxiliary regressions or optimization over separate critic classes, adding computational and tuning costs. We introduce pairwise Bellman validation (PBV), a tuning-free minimax procedure for selecting among finite classes of fitted Q-functions and occupancy-ratio estimators using held-out transitions. PBV constructs critics from pairwise candidate differences, requiring only finitely many empirical moment calculations without auxiliary validation models or separate critic optimization. We establish finite-sample oracle inequalities relative to the best available candidate, even under candidate-class misspecification. Our key insight is that Q-function errors and log ratios between candidate and true occupancy ratios provide oracle critics whose Bellman and adjoint Bellman moments control estimation error. Pairwise candidate differences approximate these unknown critics, while penalties account for comparator approximation error. For Q-function selection, we first develop unweighted PBV, which evaluates pairwise Bellman moments under the offline distribution and admits oracle guarantees under a pairwise coercivity condition relating the offline distribution to the target-policy dynamics. We then develop target-aligned PBV (TA-PBV), which uses marginalized importance weighting to provide error control without pairwise coercivity. For occupancy-ratio selection, we develop adjoint PBV (A-PBV), which evaluates adjoint Bellman moments against log-ratio critics. On Hopper and Deep OPE, unweighted PBV performs competitively, while target-occupancy weighting can improve performance in settings with controlled overlap.

Figures & tables

Setting	Selector	Absolute value error		Initial-state value RMSE
Setting	Selector	Hopper	Deep OPE	Hopper	Deep OPE
Original	Unweighted PBV	4.2	265.6	9.4	350.2
	Unweighted PBV (no penalty)	4.8	265.1	9.5	350.1
	TA-PBV ( $T=5$ )	5.2	262.1	10.7	347.3
	LSTD-Tournament	5.6	258.8	11.4	341.4
	Squared TD error	9.4	276.2	17.6	363.4

Table 1: Selected-

Q

performance at

n_{Q}=3{,}200

(lower is better). Absolute policy-value error is the primary metric. Initial-state value RMSE is the square root of the mean estimated state-value MSE under the reset-state law. Both metrics average over settings and validation resamples. Bold marks the smallest displayed mean within each setting, benchmark, and metric.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

	$\displaystyle\widehat{\mathcal{S}}_{ij}^{\mathrm{adj}}:={}$	$\displaystyle\frac{1}{n}\sum_{t=1}^{n}w_{i}(X_{t})\{\ell_{ij}(X_{t})-\gamma(\pi\ell_{ij})(S_{t}^{\prime})\}-\frac{1-\gamma}{n}\sum_{t=1}^{n}\ell_{ij}(X_{0,t})$
		$\displaystyle-\frac{1-\gamma}{4n}\sum_{t=1}^{n}\{\sqrt{w_{i}(X_{t})}-\sqrt{w_{j}(X_{t})}\}^{2}.$

Appendix

Algorithm S1 Adjoint PBV selector for occupancy ratios

Setting	Selector	Hopper	Deep OPE
Original	Unweighted PBV	9.4 [4.2,14.1] MCSE 0.4	350.2 [276.5,417.5] MCSE 0.1
	Unweighted PBV (no penalty)	9.5 [4.9,13.8] MCSE 0.4	350.1 [276.3,418.3] MCSE 0.1
	TA-PBV ( $T=5$ )	10.7 [5.6,15.0] MCSE 0.4	347.3 [275.3,413.3] MCSE 0.1
	LSTD-Tournament	11.4 [5.9,15.9] MCSE 0.4	341.4 [264.5,409.4] MCSE 0.1
	Squared TD error	17.6 [10.2,23.5] MCSE 0.7	363.4 [283.9,433.3] MCSE 0.1
Controlled overlap	Unweighted PBV	25.9 [13.0,39.1] MCSE 0.4	139.3 [43.5,212.7] MCSE 0.1

Appendix

Table S1 : Initial-state value RMSE at

n_{Q}=3{,}200

, with 90% hierarchical bootstrap intervals in brackets. The third line gives the Monte Carlo standard error of RMSE, obtained by the delta method from the MSE standard error. Bootstrap intervals hold the fitted candidates and reference simulations fixed; Monte Carlo standard errors account for the shared evaluation samples. Every target uses 1,024 initial states. Bold marks the smallest displayed mean within each setting and benchmark.

Figure S1: Policy-level difference in absolute value error relative to LSTD-Tournament; negative values favor PBV. All panels use $n_{Q}=3{,}200$ . Panels (a)–(b) show unweighted PBV on the original benchmarks, and panels (c)–(d) show TA-PBV with $T=5$ on the controlled-overlap benchmarks. Margins show means across rows and target policies.

Figure S2: Policy-level difference in absolute value error between TA-PBV with $T=5$ and unweighted PBV; negative values favor TA-PBV. All panels use $n_{Q}=3{,}200$ . Panels (a)–(b) show the original benchmarks, and panels (c)–(d) show the controlled-overlap benchmarks. Margins show row and target-policy means.

Setting	Benchmark	Unweighted PBV minus LSTD-Tournament	TA-PBV minus LSTD-Tournament	TA-PBV minus unweighted PBV
Original	Hopper	$-1.41$ $[-2.66,-0.37]$	$-0.34$ $[-1.62,0.88]$	$1.07$ $[0.14,2.43]$
Original	Deep OPE	$6.81$ $[-19.37,31.80]$	$3.26$ $[-20.20,25.07]$	$-3.55$ $[-9.98,2.88]$
Controlled overlap	Hopper	$-0.55$ $[-5.23,3.11]$	$-1.75$ $[-8.36,3.19]$	$-1.20$ $[-5.83,2.69]$
Controlled overlap	Deep OPE	$-4.66$ $[-20.42,7.78]$	$-9.61$ $[-28.71,2.18]$	$-4.95$ $[-14.40,1.54]$

Appendix

Table S2 : Paired differences in mean absolute value error at

n_{Q}=3{,}200

. Each entry reports the mean difference with its 90% hierarchical bootstrap interval below. Differences are the first selector minus the second; negative values favor the first selector. TA-PBV uses

T=5

. Inner validation resamples are averaged before resampling the benchmark hierarchy, with methods paired in every bootstrap draw.

Study	Selector	Hopper	Deep OPE
Original	Unweighted PBV	1.4 [0.6,4.4]	190.6 [90.5,373.3]
benchmarks	Unweighted PBV (no penalty)	1.8 [0.7,6.2]	197.8 [86.2,373.2]
	TA-PBV ( $T=5$ )	1.7 [0.6,6.5]	197.3 [88.6,366.9]
	LSTD-Tournament	1.6 [0.5,5.4]	184.1 [85.3,410.0]
	Squared TD error	1.9 [0.4,17.2]	197.0 [77.6,455.5]
Controlled	Unweighted PBV	8.8 [3.0,20.7]	31.8 [13.3,75.8]

Appendix

Table S3 : Setting-level absolute value error at

n_{Q}=3{,}200

, reported as median [25th, 75th percentile]. Inner validation resamples are averaged before computing quantiles. Lower is better; bold marks the smallest median at the displayed precision within each study and benchmark.

Hopper
	$n_{Q}$
Selector	3,200	6,400	12,800	25,600	51,200
Unweighted PBV	15.4 [7.9,25.1]	15.2 [7.7,25.2]	14.4 [7.7,23.9]	14.4 [7.4,23.4]	14.3 [7.4,23.9]
TA-PBV ( $T=5$ )	14.2 [8.2,20.8]	12.9 [7.3,19.2]	12.7 [6.8,19.3]	12.6 [6.7,19.1]	12.5 [6.7,19.1]
LSTD-Tournament	15.9 [7.3,26.9]	14.1 [6.4,24.2]	13.4 [6.1,22.8]	13.3 [5.9,22.8]	13.0 [5.9,21.9]
Deep OPE

Appendix

Table S4 : Absolute value error for unweighted PBV, TA-PBV, and LSTD-Tournament on the controlled-overlap benchmarks across the

Q

-validation sample-size grid. Each entry reports the mean with its 90% hierarchical bootstrap interval below. Lower is better; boldface marks the smallest mean in each column.

	Value error		ESS fraction
$T$	Hopper	Deep OPE	Hopper	Deep OPE
1	16.2	90.3	.099 ( $<$ .001)	.022 ( $<$ .001)
2.5	15.6	68.6	.106 (.066)	.447 ( $<$ .001)
5	14.2	65.1	.198 (.167)	.757 ( $<$ .001)
7.5	13.6	65.0	.409 (.387)	.868 (.002)
10	13.4	65.5	.610 (.571)	.918 (.053)

Appendix

Table S5 : TA-PBV temperature sensitivity on the controlled-overlap benchmarks at

n_{Q}=3{,}200

. Value-error entries are means over benchmark settings and validation resamples. ESS entries are median (minimum) relative effective sample sizes,

\mathrm{ESS}/n

, across benchmark settings. All temperatures use the primary benchmark draws from Table 1 . The prespecified

T=5

specification follows Sinha et al. (2022) .

Off-Policy Evaluation

Off-policy evaluation estimates a policy’s value from data collected by another policy, using weighting, models, and uncertainty-aware methods.

See all 74 papers

Dec 29, 2025

Fitted Q-Evaluation without Bellman Completeness via Occupancy Weighting

Fitted

Q

-evaluation (FQE) is a standard regression-based method for off-policy evaluation, but under distribution shift, value-function realizability alone does not ensure convergence, and existing analyses often require Bellman completeness. We trace this instability to a geometric mismatch: standard FQE projects Bellman targets in the norm induced by the offline distribution, which need not preserve Bellman contraction. We therefore study occupancy-weighted FQE, which changes only the regression weights. Weighting by a target-policy discounted occupancy ratio aligns the projection norm with the target-policy dynamics and restores contraction of the population projected Bellman operator. We derive finite-sample guarantees with estimated occupancy ratios and function-class misspecification, separating finite-iteration, statistical, approximation, and ratio-estimation errors. Exact occupancy weighting removes the need for Bellman completeness; with estimated weights, approximate completeness and value-function realizability reduce sensitivity to ratio-estimation error, with exact realizability yielding higher-order dependence. Combining occupancy-weighted FQE with fitted occupancy-ratio evaluation gives an end-to-end guarantee governed by the complexities and direct approximation errors of the value-function and occupancy-ratio classes. Under coverage, joint realizability of these two classes suffices for consistent estimation without Bellman or critic-side completeness. Controlled experiments illustrate the projection-norm mechanism and the finite-sample tradeoff between contraction and coverage.

Lars van der Laan, Nathan Kallus

Department of Statistics, University of Washington · Netflix · Cornell University

Off-Policy Evaluation Value Function Estimation

Ask PDF

Jul 6, 2026

Fitted Occupancy-Ratio Evaluation without Bellman Completeness

Occupancy ratios correct distribution shift in offline reinforcement learning and are central to off-policy evaluation. Existing primal-dual and minimax methods typically estimate these ratios by enforcing occupancy-balance moments over a critic class. We propose fitted occupancy-ratio evaluation (FORE), a fitted fixed-point method that characterizes the discounted occupancy ratio through an adjoint Bellman recursion. At each iteration, FORE solves a single-level density-ratio objective on one-step-transition data, thereby projecting the adjoint Bellman image onto a log-ratio class in Kullback--Leibler (KL) divergence. Unlike analyses of fitted Q-evaluation, which typically require value-function realizability together with Bellman completeness or projected-operator stability, our central approximation condition is just realizability of the discounted occupancy ratio itself. Under this condition, the population KL-projected recursion contracts in relative entropy toward the true ratio by virtue of the adjoint Bellman operator being a KL-contraction. For the empirical recursion, we establish finite-sample regret bounds that yield convergence in KL up to log-ratio approximation error and a statistical error governed by the complexity of the ratio hypothesis class. The fitted ratio supports direct value estimation by reward reweighting, occupancy-weighted fitted Q-evaluation, and doubly robust estimation that combines the fitted ratio with a fitted Q-function. Together, these results identify discounted occupancy-ratio realizability as a sufficient condition for offline policy evaluation without any completeness assumptions.

Lars van der Laan, Nathan Kallus

Stanford University · Netflix · Cornell University

Off-Policy Evaluation Offline RL Density Ratio Estimation

Ask PDF

Aug 25, 2026

Bellman Calibration for Marginalized Importance Weighting in Offline Reinforcement Learning

Marginalized importance weighting evaluates a target policy by reweighting offline state-action samples with its discounted occupancy ratio, characterized by an adjoint Bellman equation. Existing minimax, primal-dual, and fitted fixed-point estimators can leave residual occupancy-balance violations because of function-class approximation, regularization, or incomplete optimization. These violations are difficult to diagnose and reduce because the objectives generally lack a direct supervised validation loss for hyperparameter tuning, model selection, and early stopping. We introduce isotonic Bellman calibration, a one-dimensional, model-agnostic post-processing method that reduces these violations while preserving the ranking information in any initial occupancy-ratio estimate. The method corrects the estimate's scale and shape by applying fitted occupancy-ratio evaluation (FORE) over a one-dimensional class of nondecreasing transformations. We characterize Bellman calibration as a conditional fixed-point property equivalent to occupancy-balance against every test function of the calibrated ratio. More generally, we derive a calibration-refinement bound showing that any fitted ratio with small calibration error performs nearly as well as the best post-processing based on its fitted values. For isotonic Bellman calibration, we establish finite-sample calibration guarantees and a KL oracle inequality relative to the best monotone transformation of the initial estimate. Consequently, isotonic Bellman calibration achieves small calibration error and KL risk within statistical error of the best monotone correction, with guarantees for downstream target-occupancy functionals, including policy-value estimation.

Lars van der Laan, Nathan Kallus

Stanford University · Netflix and Cornell University

Off-Policy Evaluation Offline RL Importance Sampling

Ask PDF

Abstract

Off-Policy Evaluation

Explore similar work

Fitted Q-Evaluation without Bellman Completeness via Occupancy Weighting

Fitted Occupancy-Ratio Evaluation without Bellman Completeness

Bellman Calibration for Marginalized Importance Weighting in Offline Reinforcement Learning