Bellman Calibration for Marginalized Importance Weighting in Offline Reinforcement Learning

Lars van der Laan; Nathan Kallus

cs.LGAug 25, 2026

Bellman Calibration for Marginalized Importance Weighting in Offline Reinforcement Learning

Authors: Lars van der Laan, Nathan Kallus

Organizations: Stanford University · Netflix and Cornell University

Abstract

Marginalized importance weighting evaluates a target policy by reweighting offline state-action samples with its discounted occupancy ratio, characterized by an adjoint Bellman equation. Existing minimax, primal-dual, and fitted fixed-point estimators can leave residual occupancy-balance violations because of function-class approximation, regularization, or incomplete optimization. These violations are difficult to diagnose and reduce because the objectives generally lack a direct supervised validation loss for hyperparameter tuning, model selection, and early stopping. We introduce isotonic Bellman calibration, a one-dimensional, model-agnostic post-processing method that reduces these violations while preserving the ranking information in any initial occupancy-ratio estimate. The method corrects the estimate's scale and shape by applying fitted occupancy-ratio evaluation (FORE) over a one-dimensional class of nondecreasing transformations. We characterize Bellman calibration as a conditional fixed-point property equivalent to occupancy-balance against every test function of the calibrated ratio. More generally, we derive a calibration-refinement bound showing that any fitted ratio with small calibration error performs nearly as well as the best post-processing based on its fitted values. For isotonic Bellman calibration, we establish finite-sample calibration guarantees and a KL oracle inequality relative to the best monotone transformation of the initial estimate. Consequently, isotonic Bellman calibration achieves small calibration error and KL risk within statistical error of the best monotone correction, with guarantees for downstream target-occupancy functionals, including policy-value estimation.

Figures & tables

	$\displaystyle\widehat{h}_{k+1}$	$\displaystyle\in\argmin_{h\in\mathcal{F}_{{\rm iso},n}}\Biggl\{\log\left[\frac{1}{n}\sum_{i=1}^{n}\exp\{h(\widehat{w}_{\pi}(X_{i}))\}\right]-(1-\gamma)\frac{1}{m}\sum_{j=1}^{m}h\{\widehat{w}_{\pi}(X_{0,j})\}$
		$\displaystyle-\gamma\frac{1}{n}\sum_{i=1}^{n}\widehat{\omega}^{(k)}(X_{i})h\{\widehat{w}_{\pi}(X_{i}^{+})\}\Biggr\}.$

Algorithm 1 Isotonic Bellman calibration for marginalized importance weighting

	Projected calibration error			Absolute policy-value error
Benchmark	Uncalibrated	Calibrated	$\Delta$ [95% CI]	Uncalibrated	Calibrated	$\Delta$ [95% CI]
D4RL	16.1	0.120	$-16.0\;[-44.3,-1.02]$	2.99	0.710	$-2.28\;[-3.28,-1.39]$
CartPole	0.00207	$1.96\!\times\!10^{-4}$	$-0.00187\;[-0.00248,-0.00133]$	0.176	0.110	$-0.0658\;[-0.0733,-0.0580]$

Table 1: Cross-calibrated estimator versus the uncalibrated estimator fitted on the full training sample. Lower is better. Projected calibration error is evaluated on an independent evaluation sample using the sample-split estimator in Appendix B.3 .

\Delta

is calibrated minus uncalibrated; brackets give paired 95% cluster bootstrap confidence intervals.

Figure 1: D4RL policy-value error across tasks and ESS strata. (a) Mean absolute policy-value error for each matched dataset–policy pair; lines connect paired uncalibrated and calibrated estimators. (b) Relative change in cluster-balanced mean policy-value error across strata defined by the uncalibrated estimator’s ESS fraction. Positive values favor calibration; bars give paired 95% cluster bootstrap confidence intervals.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Estimator	Settings
Neural FORE	Two hidden layers of width 128; at most 300 outer iterations; 30 variational steps per iteration; batch size 8,192; learning rate $10^{-3}$ ; weight decay $0.1$ ; early stopping by Bellman loss on a 20% validation subset of the training fold; early stopping begins after 10 iterations and occurs after 20 iterations without improvement.
DualDICE	Google Research implementation; 5,000 updates; batch size 256.
SCOPE-RL MWL	State–action minimax objective; 10,000 updates; hidden width 128; batch size 128; training-fold standardization; median kernel bandwidth.
NeuralDICE	Google Research DICE-RL configuration; primal, dual, and normalization regularizers $(0,1,1)$ ; positive $\zeta$ output; reward term included; two hidden layers of width 64; 5,000 updates; batch size 256; learning rate $10^{-4}$ .

Appendix

Table 2: Base occupancy-ratio estimators and fixed hyperparameters.

Benchmark	Base estimator	Projected calibration error	Policy-value error
D4RL	Neural FORE	$63.9\rightarrow 0.238$	$0.465\rightarrow 1.03$
D4RL	DualDICE	$0.308\rightarrow 0.0686$	$1.90\rightarrow 0.495$
D4RL	SCOPE-RL MWL	$0.228\rightarrow 0.113$	$0.685\rightarrow 0.579$
D4RL	NeuralDICE	$0.0633\rightarrow 0.0608$	$8.91\rightarrow 0.741$
CartPole	Neural FORE	$6.05\!\times\!10^{-4}\rightarrow 3.28\!\times\!10^{-4}$	$0.123\rightarrow 0.133$
CartPole	DualDICE	$0.00673\rightarrow 1.40\!\times\!10^{-4}$	$0.212\rightarrow 0.124$

Appendix

Table 3: Mean error by base estimator. Each arrow compares the uncalibrated estimator with the ten-fold cross-calibrated estimator. Lower is better.

Dataset–policy pair	Uncalibrated	Calibrated
HalfCheetah–expert	20.1	0.179
HalfCheetah–medium	2.84	0.0280
HalfCheetah–medium-high	3.87	3.07
HalfCheetah–random	0.139	0.00425
Hopper–expert	1.20	0.0435
Hopper–medium	1.02	1.22

Appendix

Table 4: D4RL error by matched dataset–policy pair, averaged over sample sizes, replications, and base estimators.

Off-Policy Evaluation

Off-policy evaluation estimates a policy’s value from data collected by another policy, using weighting, models, and uncertainty-aware methods.

See all 74 papers

Fitted Occupancy-Ratio Evaluation without Bellman Completeness
Jul 6, 2026Lars van der Laan, Nathan KallusOff-Policy Evaluation Offline RL
Stanford University · Netflix · Cornell University
Model Selection for Off-Policy Evaluation via Pairwise Bellman Validation
Oct 26, 2026Lars van der Laan, Vasilis Syrgkanis, Nathan KallusOff-Policy Evaluation Value Function Estimation
Stanford University · Netflix and Cornell University
Fitted Q-Evaluation without Bellman Completeness via Occupancy Weighting
Dec 29, 2025Lars van der Laan, Nathan KallusOff-Policy Evaluation Value Function Estimation
Department of Statistics, University of Washington · Netflix · Cornell University

Jul 6, 2026J/K move · Enter open · S save

Fitted Occupancy-Ratio Evaluation without Bellman Completeness

Lars van der Laan, Nathan Kallus

Stanford University · Netflix · Cornell University

Off-Policy Evaluation Offline RL Density Ratio Estimation

Open paper AskPDF arXiv

Occupancy ratios correct distribution shift in offline reinforcement learning and are central to off-policy evaluation. Existing primal-dual and minimax methods typically estimate these ratios by enforcing occupancy-balance moments over a critic class. We propose fitted occupancy-ratio evaluation (FORE), a fitted fixed-point method that characterizes the discounted occupancy ratio through an adjoint Bellman recursion. At each iteration, FORE solves a single-level density-ratio objective on one-step-transition data, thereby projecting the adjoint Bellman image onto a log-ratio class in Kullback--Leibler (KL) divergence. Unlike analyses of fitted Q-evaluation, which typically require value-function realizability together with Bellman completeness or projected-operator stability, our central approximation condition is just realizability of the discounted occupancy ratio itself. Under this condition, the population KL-projected recursion contracts in relative entropy toward the true ratio by virtue of the adjoint Bellman operator being a KL-contraction. For the empirical recursion, we establish finite-sample regret bounds that yield convergence in KL up to log-ratio approximation error and a statistical error governed by the complexity of the ratio hypothesis class. The fitted ratio supports direct value estimation by reward reweighting, occupancy-weighted fitted Q-evaluation, and doubly robust estimation that combines the fitted ratio with a fitted Q-function. Together, these results identify discounted occupancy-ratio realizability as a sufficient condition for offline policy evaluation without any completeness assumptions.

Abstract

Off-Policy Evaluation

Explore similar work

Fitted Occupancy-Ratio Evaluation without Bellman Completeness

Model Selection for Off-Policy Evaluation via Pairwise Bellman Validation

Fitted Q-Evaluation without Bellman Completeness via Occupancy Weighting