Model Selection for Off-Policy Evaluation via Pairwise Bellman Validation

Lars van der Laan; Vasilis Syrgkanis; Nathan Kallus

stat.MLOct 26, 2026

Model Selection for Off-Policy Evaluation via Pairwise Bellman Validation

Authors: Lars van der Laan, Vasilis Syrgkanis, Nathan Kallus

Organizations: Stanford University · Netflix and Cornell University

Abstract

Model selection and hyperparameter tuning pose a central challenge in off-policy evaluation (OPE) because errors of fitted Q-functions and occupancy-ratio estimates cannot be directly evaluated from held-out transitions. Bellman-residual and minimax validation methods often require auxiliary regressions or optimization over separate critic classes, adding computational and tuning costs. We introduce pairwise Bellman validation (PBV), a tuning-free minimax procedure for selecting among finite classes of fitted Q-functions and occupancy-ratio estimators using held-out transitions. PBV constructs critics from pairwise candidate differences, requiring only finitely many empirical moment calculations without auxiliary validation models or separate critic optimization. We establish finite-sample oracle inequalities relative to the best available candidate, even under candidate-class misspecification. Our key insight is that Q-function errors and log ratios between candidate and true occupancy ratios provide oracle critics whose Bellman and adjoint Bellman moments control estimation error. Pairwise candidate differences approximate these unknown critics, while penalties account for comparator approximation error. For Q-function selection, we first develop unweighted PBV, which evaluates pairwise Bellman moments under the offline distribution and admits oracle guarantees under a pairwise coercivity condition relating the offline distribution to the target-policy dynamics. We then develop target-aligned PBV (TA-PBV), which uses marginalized importance weighting to provide error control without pairwise coercivity. For occupancy-ratio selection, we develop adjoint PBV (A-PBV), which evaluates adjoint Bellman moments against log-ratio critics. On Hopper and Deep OPE, unweighted PBV performs competitively, while target-occupancy weighting can improve performance in settings with controlled overlap.

Figures & tables

Setting	Selector	Absolute value error		Initial-state value RMSE
Setting	Selector	Hopper	Deep OPE	Hopper	Deep OPE
Original	Unweighted PBV	4.2	265.6	9.4	350.2
	Unweighted PBV (no penalty)	4.8	265.1	9.5	350.1
	TA-PBV ( $T=5$ )	5.2	262.1	10.7	347.3
	LSTD-Tournament	5.6	258.8	11.4	341.4
	Squared TD error	9.4	276.2	17.6	363.4

Table 1: Selected-

Q

performance at

n_{Q}=3{,}200

(lower is better). Absolute policy-value error is the primary metric. Initial-state value RMSE is the square root of the mean estimated state-value MSE under the reset-state law. Both metrics average over settings and validation resamples. Bold marks the smallest displayed mean within each setting, benchmark, and metric.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

	$\displaystyle\widehat{\mathcal{S}}_{ij}^{\mathrm{adj}}:={}$	$\displaystyle\frac{1}{n}\sum_{t=1}^{n}w_{i}(X_{t})\{\ell_{ij}(X_{t})-\gamma(\pi\ell_{ij})(S_{t}^{\prime})\}-\frac{1-\gamma}{n}\sum_{t=1}^{n}\ell_{ij}(X_{0,t})$
		$\displaystyle-\frac{1-\gamma}{4n}\sum_{t=1}^{n}\{\sqrt{w_{i}(X_{t})}-\sqrt{w_{j}(X_{t})}\}^{2}.$

Appendix

Algorithm S1 Adjoint PBV selector for occupancy ratios

Setting	Selector	Hopper	Deep OPE
Original	Unweighted PBV	9.4 [4.2,14.1] MCSE 0.4	350.2 [276.5,417.5] MCSE 0.1
	Unweighted PBV (no penalty)	9.5 [4.9,13.8] MCSE 0.4	350.1 [276.3,418.3] MCSE 0.1
	TA-PBV ( $T=5$ )	10.7 [5.6,15.0] MCSE 0.4	347.3 [275.3,413.3] MCSE 0.1
	LSTD-Tournament	11.4 [5.9,15.9] MCSE 0.4	341.4 [264.5,409.4] MCSE 0.1
	Squared TD error	17.6 [10.2,23.5] MCSE 0.7	363.4 [283.9,433.3] MCSE 0.1
Controlled overlap	Unweighted PBV	25.9 [13.0,39.1] MCSE 0.4	139.3 [43.5,212.7] MCSE 0.1

Appendix

Table S1 : Initial-state value RMSE at

n_{Q}=3{,}200

, with 90% hierarchical bootstrap intervals in brackets. The third line gives the Monte Carlo standard error of RMSE, obtained by the delta method from the MSE standard error. Bootstrap intervals hold the fitted candidates and reference simulations fixed; Monte Carlo standard errors account for the shared evaluation samples. Every target uses 1,024 initial states. Bold marks the smallest displayed mean within each setting and benchmark.

Figure S1: Policy-level difference in absolute value error relative to LSTD-Tournament; negative values favor PBV. All panels use $n_{Q}=3{,}200$ . Panels (a)–(b) show unweighted PBV on the original benchmarks, and panels (c)–(d) show TA-PBV with $T=5$ on the controlled-overlap benchmarks. Margins show means across rows and target policies.

Figure S2: Policy-level difference in absolute value error between TA-PBV with $T=5$ and unweighted PBV; negative values favor TA-PBV. All panels use $n_{Q}=3{,}200$ . Panels (a)–(b) show the original benchmarks, and panels (c)–(d) show the controlled-overlap benchmarks. Margins show row and target-policy means.

Setting	Benchmark	Unweighted PBV minus LSTD-Tournament	TA-PBV minus LSTD-Tournament	TA-PBV minus unweighted PBV
Original	Hopper	$-1.41$ $[-2.66,-0.37]$	$-0.34$ $[-1.62,0.88]$	$1.07$ $[0.14,2.43]$
Original	Deep OPE	$6.81$ $[-19.37,31.80]$	$3.26$ $[-20.20,25.07]$	$-3.55$ $[-9.98,2.88]$
Controlled overlap	Hopper	$-0.55$ $[-5.23,3.11]$	$-1.75$ $[-8.36,3.19]$	$-1.20$ $[-5.83,2.69]$
Controlled overlap	Deep OPE	$-4.66$ $[-20.42,7.78]$	$-9.61$ $[-28.71,2.18]$	$-4.95$ $[-14.40,1.54]$

Appendix

Table S2 : Paired differences in mean absolute value error at

n_{Q}=3{,}200

. Each entry reports the mean difference with its 90% hierarchical bootstrap interval below. Differences are the first selector minus the second; negative values favor the first selector. TA-PBV uses

T=5

. Inner validation resamples are averaged before resampling the benchmark hierarchy, with methods paired in every bootstrap draw.

Study	Selector	Hopper	Deep OPE
Original	Unweighted PBV	1.4 [0.6,4.4]	190.6 [90.5,373.3]
benchmarks	Unweighted PBV (no penalty)	1.8 [0.7,6.2]	197.8 [86.2,373.2]
	TA-PBV ( $T=5$ )	1.7 [0.6,6.5]	197.3 [88.6,366.9]
	LSTD-Tournament	1.6 [0.5,5.4]	184.1 [85.3,410.0]
	Squared TD error	1.9 [0.4,17.2]	197.0 [77.6,455.5]
Controlled	Unweighted PBV	8.8 [3.0,20.7]	31.8 [13.3,75.8]

Appendix

Table S3 : Setting-level absolute value error at

n_{Q}=3{,}200

, reported as median [25th, 75th percentile]. Inner validation resamples are averaged before computing quantiles. Lower is better; bold marks the smallest median at the displayed precision within each study and benchmark.

Hopper
	$n_{Q}$
Selector	3,200	6,400	12,800	25,600	51,200
Unweighted PBV	15.4 [7.9,25.1]	15.2 [7.7,25.2]	14.4 [7.7,23.9]	14.4 [7.4,23.4]	14.3 [7.4,23.9]
TA-PBV ( $T=5$ )	14.2 [8.2,20.8]	12.9 [7.3,19.2]	12.7 [6.8,19.3]	12.6 [6.7,19.1]	12.5 [6.7,19.1]
LSTD-Tournament	15.9 [7.3,26.9]	14.1 [6.4,24.2]	13.4 [6.1,22.8]	13.3 [5.9,22.8]	13.0 [5.9,21.9]
Deep OPE

Appendix

Table S4 : Absolute value error for unweighted PBV, TA-PBV, and LSTD-Tournament on the controlled-overlap benchmarks across the

Q

-validation sample-size grid. Each entry reports the mean with its 90% hierarchical bootstrap interval below. Lower is better; boldface marks the smallest mean in each column.

	Value error		ESS fraction
$T$	Hopper	Deep OPE	Hopper	Deep OPE
1	16.2	90.3	.099 ( $<$ .001)	.022 ( $<$ .001)
2.5	15.6	68.6	.106 (.066)	.447 ( $<$ .001)
5	14.2	65.1	.198 (.167)	.757 ( $<$ .001)
7.5	13.6	65.0	.409 (.387)	.868 (.002)
10	13.4	65.5	.610 (.571)	.918 (.053)

Appendix

Table S5 : TA-PBV temperature sensitivity on the controlled-overlap benchmarks at

n_{Q}=3{,}200

. Value-error entries are means over benchmark settings and validation resamples. ESS entries are median (minimum) relative effective sample sizes,

\mathrm{ESS}/n

, across benchmark settings. All temperatures use the primary benchmark draws from Table 1 . The prespecified

T=5

specification follows Sinha et al. (2022) .

Off-Policy Evaluation

Off-policy evaluation estimates a policy’s value from data collected by another policy, using weighting, models, and uncertainty-aware methods.

See all 74 papers

Fitted Q-Evaluation without Bellman Completeness via Occupancy Weighting
Dec 29, 2025Lars van der Laan, Nathan KallusOff-Policy Evaluation Value Function Estimation
Department of Statistics, University of Washington · Netflix · Cornell University
Fitted Occupancy-Ratio Evaluation without Bellman Completeness
Jul 6, 2026Lars van der Laan, Nathan KallusOff-Policy Evaluation Offline RL
Stanford University · Netflix · Cornell University
Bellman Calibration for Marginalized Importance Weighting in Offline Reinforcement Learning
Aug 25, 2026Lars van der Laan, Nathan KallusOff-Policy Evaluation Offline RL
Stanford University · Netflix and Cornell University

Dec 29, 2025J/K move · Enter open · S save

Fitted Q-Evaluation without Bellman Completeness via Occupancy Weighting

Lars van der Laan, Nathan Kallus

Department of Statistics, University of Washington · Netflix · Cornell University

Off-Policy Evaluation Value Function Estimation

Open paper AskPDF arXiv

Fitted

Q

-evaluation (FQE) is a standard regression-based method for off-policy evaluation, but under distribution shift, value-function realizability alone does not ensure convergence, and existing analyses often require Bellman completeness. We trace this instability to a geometric mismatch: standard FQE projects Bellman targets in the norm induced by the offline distribution, which need not preserve Bellman contraction. We therefore study occupancy-weighted FQE, which changes only the regression weights. Weighting by a target-policy discounted occupancy ratio aligns the projection norm with the target-policy dynamics and restores contraction of the population projected Bellman operator. We derive finite-sample guarantees with estimated occupancy ratios and function-class misspecification, separating finite-iteration, statistical, approximation, and ratio-estimation errors. Exact occupancy weighting removes the need for Bellman completeness; with estimated weights, approximate completeness and value-function realizability reduce sensitivity to ratio-estimation error, with exact realizability yielding higher-order dependence. Combining occupancy-weighted FQE with fitted occupancy-ratio evaluation gives an end-to-end guarantee governed by the complexities and direct approximation errors of the value-function and occupancy-ratio classes. Under coverage, joint realizability of these two classes suffices for consistent estimation without Bellman or critic-side completeness. Controlled experiments illustrate the projection-norm mechanism and the finite-sample tradeoff between contraction and coverage.

Abstract

Off-Policy Evaluation

Explore similar work

Fitted Q-Evaluation without Bellman Completeness via Occupancy Weighting

Fitted Occupancy-Ratio Evaluation without Bellman Completeness

Bellman Calibration for Marginalized Importance Weighting in Offline Reinforcement Learning