Offline Policy Evaluation via Mixed Bellman Residuals and Adaptive Critic Representations

Amitakshar Biswas; Yuhan Li; Ruoqing Zhu

stat.MLSep 27, 2026

Offline Policy Evaluation via Mixed Bellman Residuals and Adaptive Critic Representations

Authors: Amitakshar Biswas, Yuhan Li, Ruoqing Zhu

Organizations: Department of Statistics, University of Illinois at Urbana-Champaign

Abstract

Evaluating a target policy using data generated by a different behavior policy remains a fundamental challenge in reinforcement learning. While most existing work relies on the standard one-step Bellman residual, we consider a convex combination of one-step and two-step residuals with a fixed mixing weight. In ideal settings, this mixed Bellman formulation can provide a natural bias--variance trade-off between approximation error under a restricted value-function class and the increased variance arising from multi-step importance weighting. To solve this mixed residual optimization, we adopt a minimax formulation involving a critic function. Unlike standard approaches that rely on a fixed functional class, we construct a data-dependent critic representation using predicted future feature directions which effectively induces a kernel adapted to the underlying transition dynamics. This allows the critic to focus on directions that are most relevant for the estimated Bellman error. To control overfitting, we use sample splitting to construct the critic and estimate the value function on separate data subsets. Simulation studies and MetaWorld tasks illustrate the effect of the mixing parameter and show that intermediate residual combinations can improve value estimation in challenging settings.

Figures & tables

Figure 1: Frames from the MetaWorld door-open-v3 task. The robot arm approaches the handle, makes contact, and pulls the door open.

Figure 2: Pointwise squared bias, variance, and MSE in the MetaWorld door-open-v3 task as functions of $\alpha$ . The two-step contribution reduces squared bias but increases variance, with a shallow MSE minimum at $\alpha=0.5$ .

Pointwise Value MSE
Method	Env I	Env II	Env III	Env IV	Env V	Env VI
FQE	$\mathbf{.01777{\scriptstyle\pm.00089}}$	$.02217{\scriptstyle\pm.00033}$	$.69464{\scriptstyle\pm.01138}$	$.08911{\scriptstyle\pm.00460}$	$.01105{\scriptstyle\pm.00029}$	$1.95669{\scriptstyle\pm.02731}$
Kernel BRM/MQL	$.11884{\scriptstyle\pm.00225}$	$.03378{\scriptstyle\pm.00023}$	$1.22100{\scriptstyle\pm.10387}$	$2.62950{\scriptstyle\pm.26174}$	$.13920{\scriptstyle\pm.01169}$	$8.17740{\scriptstyle\pm.63216}$
Mixed-AC ( $\alpha=0$ )	$.14404{\scriptstyle\pm.00612}$	$.01850{\scriptstyle\pm.00039}$	$.08361{\scriptstyle\pm.00819}$	$\mathbf{.05861{\scriptstyle\pm.00814}}$	$.003393{\scriptstyle\pm.000634}$	$.14376{\scriptstyle\pm.01713}$
Mixed-AC (best fixed $\alpha$ )	$.14404{\scriptstyle\pm.00612}$	$\mathbf{.01828{\scriptstyle\pm.00035}}$	$\mathbf{.08180{\scriptstyle\pm.00808}}$	$\mathbf{.05861{\scriptstyle\pm.00814}}$	$\mathbf{.003186{\scriptstyle\pm.000397}}$	$\mathbf{.11888{\scriptstyle\pm.01035}}$
$J^{\pi}$ Squared Error

Table 1: Pointwise MSE and scalar policy-value squared error, reported as mean

\pm

SE over

100

repetitions. The best fixed

\alpha

is chosen post hoc using true MSE and is included only as a diagnostic.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Env.	$d_{\mathrm{RFF}}$	$d_{\mathrm{MLP}}$	$\eta$	$\lambda_{\theta}$	$(h_{\mathrm{cond}},\lambda_{\mathrm{cond}})$
I	64	32	5	10	$(32,10^{-4})$
II	128	16	1	$10^{-5}$	$(32,10^{-5})$
III	64	16	100	10	$(64,10^{-5})$
IV	64	16	10	$10^{-1}$	$(32,10^{-5})$
V	64	16	10	1	$(32,10^{-3})$
VI	128	32	10	1	$(32,10^{-4})$

Appendix

Table 2: Selected Mixed-AC configurations.

Env.	Hidden	LR	WD	Iterations	Steps
I	128	.001	.001	100	25
II	128	.001	$10^{-5}$	50	25
III	64	.001	.001	100	10
IV	128	.001	.001	100	25
V	128	.001	$10^{-4}$	50	25
VI	128	.001	$10^{-4}$	50	25

Appendix

Table 3: Selected FQE configurations.

Env.	Mixed-AC $-$ FQE
I	$.12627$ $[.11442,.13812]$
II	$-.00367$ $[-.00454,-.00281]$
III	$-.61103$ $[-.64061,-.58145]$
IV	$-.03050$ $[-.05042,-.01057]$
V	$-.007656$ $[-.009002,-.006311]$
VI	$-1.81293$ $[-1.88040,-1.74545]$

Appendix

Table 4: Paired pointwise MSE differences between Mixed-AC at

\alpha=0

and FQE. Brackets give 95% confidence intervals. Negative values favor Mixed-AC.

	$0$	$.25$	$.50$	$.75$	$1$
Env I	$.14404\pm.00612$	$.18737\pm.01684$	$.33502\pm.03250$	$.55987\pm.04945$	$.83283\pm.06611$
Env II	$.01850\pm.00039$	$.01828\pm.00035$	$.01943\pm.00037$	$.02166\pm.00043$	$.02452\pm.00053$
Env III	$.08361\pm.00819$	$.08184\pm.00799$	$.08180\pm.00808$	$.08250\pm.00829$	$.08356\pm.00853$
Env IV	$.05861\pm.00814$	$.05977\pm.00829$	$.06432\pm.00859$	$.07048\pm.00894$	$.07708\pm.00928$
Env V	$.003393\pm.000634$	$.003186\pm.000397$	$.003693\pm.000423$	$.004238\pm.000466$	$.004695\pm.000497$
Env VI	$.14376\pm.01713$	$.11888\pm.01035$	$.12987\pm.01041$	$.14536\pm.01165$	$.16010\pm.01278$

Appendix

Table 5: Pointwise MSE, mean

\pm

SE over

100

repetitions.

$\alpha$	Bias 2	Variance	MSE
0	.04601	.03798	.08361
.25	.04230	.03994	.08184
.50	.04013	.04209	.08180
.75	.03888	.04406	.08250
1	.03824	.04578	.08356

Appendix

Table 6: Pointwise squared bias, variance, and MSE for door-open-v3 .

	$\alpha=0$	$\alpha=.25$	$\alpha=.50$	$\alpha=.75$	$\alpha=1$
Env III (Door)	$-0.5$	$-0.6$	$-0.8$	$-0.9$	$-1.2$
Env IV (Drawer)	$\mathbf{24.3}$	$13.4$	$1.2$	$-10.3$	$-19.2$
Env V (Button)	$\mathbf{83.4}$	$60.6$	$12.8$	$-33.9$	$-53.4$
Env VI (Faucet)	$\mathbf{85.2}$	$69.1$	$33.2$	$-9.3$	$-36.0$

Appendix

Table 7: Relative MSE improvement (%) from using the adaptive critic instead of the current-state critic. Positive values favor the adaptive critic.

Env.	Selected MSE	Oracle MSE	Excess MSE [95% CI]	Agreement
I	$.15918$	$.12798$	$.03121\ [.01771,.04470]$	$59\%$
II	$.01899$	$.01823$	$.00076\ [.00044,.00109]$	$53\%$
III	$.10619$	$.08980$	$.01639\ [.01142,.02137]$	$39\%$
IV	$.09221$	$.07846$	$.01374\ [.00627,.02122]$	$65\%$
V	$.004124$	$.002030$	$.002094\ [.000608,.003580]$	$55\%$
VI	$.22188$	$.11568$	$.10620\ [.07193,.14047]$	$41\%$

Appendix

Table 8: Performance of the critic-scaled validation rule. Excess MSE is the difference between the selected estimator and the oracle candidate in the same repetition.

Env.	Selected MSE	$\bm{\alpha=0}$ MSE	Oracle MSE	Agreement	Spearman
I	$.81944$	$.14799$	$.11823$	$3\%$	$-.736$
II	$.01962$	$.01819$	$.01763$	$9\%$	$.144$
III	$.10686$	$.10366$	$.08984$	$39\%$	$-.039$
IV	$.05880$	$.04967$	$.04223$	$38\%$	$.031$
V	$.004233$	$.002717$	$.001387$	$12\%$	$-.503$
VI	$.15029$	$.14344$	$.06845$	$12\%$	$-.230$

Appendix

Table 9: Performance of the common-studentized validation rule over

100

repetitions. The

\alpha=0

and oracle columns are computed under the same final-refit protocol. Ground truth is used only for post-selection evaluation.

Method	I	II	III	IV	V	VI
Mixed-AC	$.04725$ $(.00775)$	$\mathbf{.000945}$ $\mathbf{(.000134)}$	$\mathbf{.11495}$ $\mathbf{(.01546)}$	$\mathbf{.07719}$ $\mathbf{(.00867)}$	$\mathbf{.003524}$ $\mathbf{(.000711)}$	$\mathbf{.15939}$ $\mathbf{(.02217)}$
DualDICE	$.11974$ $(.02682)$	$.12567$ $(.00758)$	$.64101$ $(.00824)$	$.16485$ $(.00402)$	$.01224$ $(.00020)$	$.82602$ $(.01283)$
GenDICE	$1.78283$ $(.16016)$	$.002522$ $(.000198)$	$.12537$ $(.01305)$	$.08068$ $(.00941)$	$.008640$ $(.000738)$	$.32441$ $(.03335)$
BestDICE	$\mathbf{.003943}$ $\mathbf{(.000465)}$	$.007350$ $(.000410)$	$.43554$ $(.01625)$	$.10980$ $(.00479)$	$.015265$ $(.000447)$	$.69331$ $(.01481)$

Appendix

Table 10: Squared scalar policy-value error over

100

repetitions. Entries report mean with SE in parentheses. Bold denotes the lowest sample mean in each environment.

Env.	Mixed-AC $-$ DualDICE	95% CI
I	$-.07249$	$[-.12826,-.01673]$
II	$-.12473$	$[-.13976,-.10969]$
III	$-.52605$	$[-.55486,-.49725]$
IV	$-.08766$	$[-.10775,-.06757]$
V	$-.008714$	$[-.01013,-.00730]$
VI	$-.66663$	$[-.71669,-.61657]$

Appendix

Table 11: Paired differences in squared scalar policy-value error between Mixed-AC and DualDICE.

Env.	Mixed-AC $-$ GenDICE	95% CI	Mixed-AC $-$ BestDICE	95% CI
I	$-1.73558$	$[-2.05330,-1.41785]$	$.043307$	$[.028012,.058602]$
II	$-.001577$	$[-.002045,-.001110]$	$-.006405$	$[-.007228,-.005582]$
III	$-.010414$	$[-.046102,.025274]$	$-.32059$	$[-.36459,-.27658]$
IV	$-.003484$	$[-.026470,.019503]$	$-.032608$	$[-.053661,-.011555]$
V	$-.005116$	$[-.006922,-.003310]$	$-.011741$	$[-.013352,-.010130]$
VI	$-.16502$	$[-.24426,-.08579]$	$-.53392$	$[-.58324,-.48461]$

Appendix

Table 12: Paired differences in squared scalar policy-value error for GenDICE and BestDICE. Negative values favor Mixed-AC.

	Unnormalized		Self-normalized
	$\widehat{\pi}/\widehat{b}$	$\pi/\widehat{b}$	$\widehat{\pi}/\widehat{b}$	$\pi/\widehat{b}$
Env III	$123.80$	$204.38$	$.2368$	$.2364$
Env IV	$66.23$	$82.10$	$.1219$	$.1196$
Env V	$4.377$	$5.756$	$.003571$	$.003646$
Env VI	$225.38$	$280.98$	$.1202$	$.1209$

Appendix

Table 13: Mean pointwise MSE in the MetaWorld importance-ratio sensitivity experiment.

	1k	2k	5k	7.5k	10k	20k
Env I	.673	.587	.783	.940	1.078	–
Env II	.714	.640	.935	1.185	1.453	–
Env III	1.001	.726	.852	.955	1.024	1.464
Env IV	.998	.709	.802	.933	1.000	1.369
Env V	1.030	.743	.850	.948	1.050	1.438
Env VI	1.241	.978	1.196	1.408	1.571	2.285

Appendix

Table 14: Mean total fitting time in seconds.

	1k	2k	5k	7.5k	10k	20k
Env I	11.32	11.23	27.84	41.72	55.32	–
Env II	13.11	16.41	40.76	61.08	81.02	–
Env III	13.42	13.28	32.22	48.40	64.10	127.11
Env IV	13.42	12.98	31.55	47.42	62.81	124.62
Env V	13.42	12.98	31.55	47.42	62.81	124.62
Env VI	15.76	21.86	53.18	80.00	105.97	210.27

Appendix

Table 15: Peak Python-attributed memory in MB.

	1k	2k	5k	7.5k	10k	20k
Env I	2.29	4.33	10.96	16.57	21.92	–
Env II	3.44	6.50	16.44	24.85	32.88	–
Env III	1.729	3.673	9.075	13.828	18.365	36.515
Env IV	1.729	3.673	9.075	13.828	18.365	36.515
Env V	1.729	3.673	9.075	13.828	18.365	36.515
Env VI	3.457	7.346	18.149	27.656	36.731	73.030

Appendix

Table 16: Feature storage in MB.

Off-Policy Evaluation

Off-policy evaluation estimates a policy’s value from data collected by another policy, using weighting, models, and uncertainty-aware methods.

See all 74 papers

Deep Weighted Bellman Residual Minimization for $Q^*$ Estimation
Sep 28, 2026Lican Kang, Jerry Zhijian Yang, Cheng Yuan +1Off-Policy Evaluation Value Function Estimation
Institute for Math and AI, Wuhan University, Wuhan, 430072, China · Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, 430072, China · School of Artificial Intelligence, Wuhan University, Wuhan, 430072, China +1
Residuals-based Offline Reinforcement Learning
Apr 1, 2026Qing Zhu, Xian YuOffline RL Deep Q-Learning
Department of Integrated Systems Engineering, The Ohio State University, Columbus, OH, USA
Lifted Bellman Linear Programming for Offline Reinforcement Learning
Sep 21, 2026Hyukjun Yang, Jongchan Park, Narim Jeong +1Reinforcement Learning Offline RL
Korea Advanced Institute of Science and Technology(KAIST) · NAVER LABS

Sep 28, 2026J/K move · Enter open · S save

Deep Weighted Bellman Residual Minimization for $Q^*$ Estimation

Lican Kang, Jerry Zhijian Yang, Cheng Yuan, Chen Zhong

Institute for Math and AI, Wuhan University, Wuhan, 430072, China · Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, 430072, China · School of Artificial Intelligence, Wuhan University, Wuhan, 430072, China+1

Off-Policy Evaluation Value Function Estimation Offline RL

Open paper AskPDF arXiv

Off-policy evaluation is a foundational component of offline reinforcement learning, aiming to assess and optimize policy performance using pre-collected datasets. However, such datasets often suffer from pronounced challenges, including distribution shift,

Q

-value overestimation, and low sample utilization efficiency. To address these issues, this paper introduces a weighted Bellman residual minimization framework that incorporates density ratio weighting by effectively integrating expert demonstrations with behavioral data. The proposed weighting scheme departs from the conventional completeness assumption commonly imposed in the theoretical analysis of deep reinforcement learning. We establish a sharp convergence rate for density ratio estimation and derive the convergence rate for the excess risk of resulting deep

Q^*

estimator. Extensive empirical evaluations demonstrate that, compared to existing methods, our method achieves significant improvements in numerical performance and policy generalization, providing specific guidance for the rational utilization of expert demonstrations.

Abstract

Off-Policy Evaluation

Explore similar work

Deep Weighted Bellman Residual Minimization for Q∗Q^*Q∗ Estimation

Residuals-based Offline Reinforcement Learning

Lifted Bellman Linear Programming for Offline Reinforcement Learning

Deep Weighted Bellman Residual Minimization for $Q^*$ Estimation