Off-Policy Evaluation

Follow this topic Ask about this topic

Off-policy evaluation (OPE) estimates how a new decision policy would perform using data collected by a different policy, avoiding costly or risky deployment trials. Common approaches reweight logged actions with importance sampling, combine weights with reward models in doubly robust estimators, fit value functions with fitted Q-evaluation, or learn dynamics and occupancy ratios; bootstrap and conformal methods help quantify uncertainty. Applications span contextual bandits and sequential decision processes, including recommendation, healthcare, robotics, and language-model agents, with benchmarks testing policy shift, sparse support, and missing outcomes. Estimates can become biased or unstable when target actions or histories are poorly represented, so overlap diagnostics, sensitivity checks, and calibrated uncertainty matter alongside point estimates.

Latest papers

74 papers17 papers in the last four weeks, up 113% on the four weeks before. 0.1% of all new papers.

Q-MMR: Off-Policy Evaluation via Recursive Reweighting and Moment Matching
May 7, 2026Xiang Li, Nan JiangOff-Policy Evaluation Value Function Estimation
Nanjing University · UIUC
SOPE: Stabilizing Off-Policy Evaluation for Online RL with Prior Data
May 7, 2026Carlo Romeo, Girolamo Macaluso, Alessandro Sestini +1Off-Policy Evaluation Offline-To-Online RL
1Media Integration and Communication Center – University of Florence · 2SEED – Electronic Arts
The Partial Testimony of Logs: Evaluation of Language Model Generation under Confounded Model Choice
May 2, 2026Jikai Jin, Vasilis SyrgkanisLLM Evaluation Off-Policy Evaluation
Robustness of Refugee-Matching Gains to Off-Policy Evaluation Choices
Apr 25, 2026Kirk Bansak, Elisabeth Paulson, Dominik Rothenhäusler +3Counterfactual Evaluation Causal Effect Estimation
Immigration Policy Lab, Stanford University · Department of Political Science, University of California, Berkeley · Technology and Operations Management Unit, Harvard Business School +3
CASP: Support-Aware Offline Policy Selection for Two-Stage Recommender Systems
Apr 24, 2026Nilson ChapagainOff-Policy Evaluation Recommender Systems
Texas A&M University College Station, TX, USA
Distributional Off-Policy Evaluation with Deep Quantile Process Regression
Apr 20, 2026Qi Kuang, Chao Wang, Yuling Jiao +1Distributional RL Quantile Regression
School of Statistics and Data Science, and Philosophy and Social Sciences Laboratory of Data Science in Finance and Economics at the Ministry of Education,2026 Jiangxi University of Finance and Economics · School of Statistics and Data Science, Shanghai University of Finance and EconomicsApr · School of Artificial Intelligence, and Hubei Key Laboratory of Computational Science, 24 Wuhan University
Off-Policy Evaluation and Learning for Survival Outcomes under Censoring
Mar 24, 2026Kohsuke Kubota, Mitsuhiro Takahashi, Yuta SaitoOff-Policy Evaluation Off-Policy RL
NTT DOCOMO, INC. · Hanjuku-kaso, Co, Ltd.
Off-Policy Evaluation for Ranking Policies under Deterministic Logging Policies
Mar 23, 2026Koichi Tanaka, Kazuki Kawamura, Takanori Muroi +6Off-Policy Evaluation Inverse Probability Weighting
Keio University · Sony Group Corporation · Yale University +1
From Weighting to Modeling: A Nonparametric Estimator for Off-Policy Evaluation
Mar 10, 2026Rong J. B. ZhuOff-Policy Evaluation Inverse Probability Weighting
Institute of Science and Technology for Brain-inspired Intelligence, Fudan University
Unifying On- and Off-Policy Variance Reduction Methods
Mar 9, 2026Olivier JeunenCausal Effect Estimation A/b Testing
aampe Antwerp, Belgium
A Covering Framework for Offline POMDPs Learning using Belief Space Metric
Mar 3, 2026Youheng Zhu, Yiping LuOff-Policy Evaluation Offline RL
Northwestern University
Additive Control Variates Dominate Self-Normalisation in Off-Policy Evaluation
Feb 16, 2026Olivier Jeunen, Shashank GuptaOff-Policy Evaluation Variance Reduction
aampe Antwerp, Belgium · Microsoft AI Bangalore, India
Fitted Q-Evaluation without Bellman Completeness via Occupancy Weighting
Dec 29, 2025Lars van der Laan, Nathan KallusOff-Policy Evaluation Value Function Estimation
Department of Statistics, University of Washington · Netflix · Cornell University
PERRY: Policy Evaluation with Confidence Intervals using Auxiliary Data
Jul 26, 2025Aishwarya Mandyam, Jason Meng, Ge Gao +4Off-Policy Evaluation Healthcare
Stanford University · Gladstone Institutes · Stanford University[cs.LG]

May 7, 2026J/K move · Enter open · S save

Q-MMR: Off-Policy Evaluation via Recursive Reweighting and Moment Matching

Xiang Li, Nan Jiang

Nanjing University · UIUC

Off-Policy Evaluation Value Function Estimation Distribution Matching Offline RL

Open paper AskPDF arXiv

We present a novel theoretical framework, Q-MMR, for off-policy evaluation in finite-horizon MDPs. Q-MMR learns a set of scalar weights, one for each data point, such that the reweighted rewards approximate the expected return under the target policy. The weights are learned inductively in a top-down manner via a moment matching objective against a value-function discriminator class. Notably, and perhaps surprisingly, a data-dependent finite-sample guarantee for general function approximation can be established under only the realizability of

Q^π

, with a dimension-free bound -- that is, the error does not depend on the statistical complexity of the function class. We also establish connections to several existing methods, such as importance sampling and linear FQE. Further theoretical analyses shed new light on the nature of coverage, a concept of fundamental importance to offline RL.