Off-Policy Evaluation

Follow this topic Ask about this topic

Off-policy evaluation (OPE) estimates how a new decision policy would perform using data collected by a different policy, avoiding costly or risky deployment trials. Common approaches reweight logged actions with importance sampling, combine weights with reward models in doubly robust estimators, fit value functions with fitted Q-evaluation, or learn dynamics and occupancy ratios; bootstrap and conformal methods help quantify uncertainty. Applications span contextual bandits and sequential decision processes, including recommendation, healthcare, robotics, and language-model agents, with benchmarks testing policy shift, sparse support, and missing outcomes. Estimates can become biased or unstable when target actions or histories are poorly represented, so overlap diagnostics, sensitivity checks, and calibrated uncertainty matter alongside point estimates.

Latest papers

74 papers17 papers in the last four weeks, up 113% on the four weeks before. 0.1% of all new papers.

Model Selection for Off-Policy Evaluation via Pairwise Bellman Validation
Oct 26, 2026Lars van der Laan, Vasilis Syrgkanis, Nathan KallusOff-Policy Evaluation Value Function Estimation
Stanford University · Netflix and Cornell University
Prior-Data Fitted Networks for In-Context Off-Policy Evaluation
Oct 25, 2026Shuguang Yu, Ting Li, Fan ZhouOff-Policy Evaluation Prior-Data Fitted Networks
The Hong Kong Polytechnic University, Hong Kong, China · Shanghai University of Finance and Economics, Shanghai, China
Doubly Robust Distributionally Robust Offline Contextual Pricing
Oct 22, 2026Min Xu, Xinyi Yin, Yunfan Zhang +3Off-Policy Evaluation Distributionally Robust Optimization
School of Management and Engineering, Nanjing University, Nanjing, China · School of Mathematics, Nanjing University, Nanjing, China · Stern School of Business, New York University, New York, USA
The Certification Exponent: Instance-Optimal Sequential Certification of Policy Improvement from History Dependent Logs
Oct 17, 2026Rui WangSequential Hypothesis Testing Off-Policy Evaluation
Microsoft Vancouver, BC, Canada
Coverage Cliffs in Off-Policy Quantile Certification
Oct 15, 2026Pauline Bourigault, Emmanuelle BourigaultOff-Policy Evaluation Contextual Bandits
Explore Once, Evaluate Many: Multiple Policy Evaluation for Stochastic Linear Bandits
Oct 15, 2026Spyros Dragazis, Yilei Chen, Aldo PacchianoPolicy Evaluation Linear Contextual Bandits
Boston University · Broad Institute of MIT and Harvard
RoboFQE: Rollout-Free Evaluation of Visual Robot Policies from Offline Trajectories
Oct 13, 2026Yunfu Deng, Sangwoo Shin, Brahma S. Pavse +1Robotic Policy Evaluation Off-Policy Evaluation
Department of Computer Sciences, University of Wisconsin–Madison
agent-risk: Risk Assessment for Changes to AI Agents
Oct 12, 2026Yue Zhao, Zixiang Xu, Angela Huyue Zhang +2AI Risk Management AI Agent Safety
University of Southern California · London Business School · Arizona State University
What a Data Release Must Preserve for Off-Policy Evaluation
Oct 11, 2026Kyungbok Lee, Yating Zou, Michael R. KosorokOff-Policy Evaluation Partial Identification
Department of Biostatistics, Gillings School of Global Public Health University of North Carolina at Chapel Hill
Budgeted Multi-Source Counterfactual Annotation for Off-Policy Evaluation
Oct 7, 2026Biao Xiang, Ali Eshragh, Yuexing Li +1Off-Policy Evaluation Counterfactual Evaluation
Pennsylvania State University, State College, PA · Johns Hopkins Carey Business School, Washington, DC · International Computer Science Institute, Berkeley, CA +1
Amortized Off-Policy Evaluation for LLMs
Oct 7, 2026Younwoo Choi, Leo Feng, Vincent Liu +1Off-Policy Evaluation LLM Evaluation
RBC Borealis · University of Toronto · Vector Institute
Efficient Best-of-N policy evaluation for inference-time alignment
Oct 7, 2026Jonas Schweisthal, Yuxin Wang, Athiya Deviyani +2Best-of-N Sampling Off-Policy Evaluation
LMU Munich & MCML · Carnegie Mellon University · Meta (work in personal capacity)
Variance-Optimal Off-Policy Evaluation with Conjunct Effect Modeling
Oct 6, 2026Nicolò Felicioni, Michael Benigni, Maurizio Ferrari Dacrema +1Causal Effect Estimation Off-Policy Evaluation
Spotify · Politecnico di Milano
Evaluation of Active Feature Acquisition Policies with Tabular Foundation Models
Oct 5, 2026Yuta Kobayashi, Divyam Madaan, Shalmali JoshiOff-Policy Evaluation Value Function Estimation
Department of Biomedical Informatics, Columbia University
Revealing After Overwriting: An Exponential POMDP OPE Lower Bound under History-Dependent Logging
Oct 4, 2026Youyu Luo, Pengzhan Zhou, Zhida Qin +4Off-Policy Evaluation Partially Observable Markov Decision Processes
Chongqing University · Beijing Institute of Technology · The Hong Kong Polytechnic University
OTROPE: Optimal Transport-based Robust Off-policy Evaluation for Large Language Models
Sep 28, 2026Liner Xiang, Wenbo Zhang, Hengrui CaiLLM Evaluation Off-Policy Evaluation
Department of Statistics, University of California, Irvine
Recommendation Ranking Off-Policy Evaluation under Ranking-Dependent Examination via Examination-Relevance Decomposition
Sep 28, 2026Riki Okamura, Toshiharu SugawaraOff-Policy Evaluation Doubly Robust Estimation
Department of Computer Science and Communications Engineering, Waseda University, Tokyo, Japan
Deep Weighted Bellman Residual Minimization for $Q^*$ Estimation
Sep 28, 2026Lican Kang, Jerry Zhijian Yang, Cheng Yuan +1Off-Policy Evaluation Value Function Estimation
Institute for Math and AI, Wuhan University, Wuhan, 430072, China · Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, 430072, China · School of Artificial Intelligence, Wuhan University, Wuhan, 430072, China +1
Offline Policy Evaluation via Mixed Bellman Residuals and Adaptive Critic Representations
Sep 27, 2026Amitakshar Biswas, Yuhan Li, Ruoqing ZhuOff-Policy Evaluation Representation Learning for RL
Department of Statistics, University of Illinois at Urbana-Champaign
BranchDrive: A Branch-Structured Dataset for Action-Conditioned Driving Prediction
Sep 23, 2026Feeza Khan Khanzada, Sudarshan Sridhar, Jaerock KwonAutonomous Driving Datasets Autonomous Driving Benchmarks
Department of Electrical and Computer Engineering, University of Michigan-Dearborn, 4901 Evergreen Rd, Dearborn, MI 48128, United States
Optimal Sequential Annotations for Off-Policy Evaluation
Sep 22, 2026Woojin Chae, Ezinne Nwankwo, Haitong Qin +1Off-Policy Evaluation Learning with Missing Data
University of Southern California · University of California, Berkeley · University of Washington, Seattle
Counterfactual Tool Ranking under Utility, Cost, and Privilege Constraints
Sep 19, 2026Jiapeng LiCounterfactual Evaluation Tool-Use Evaluation
Microsoft
Model-based Bootstrap for Offline Policy Evaluation in Tabular Reinforcement Learning
Sep 17, 2026Weiwei Wang, Yuqiang Li, Xianyi Wu +1Reinforcement Learning Off-Policy Evaluation
Department of Statistics and Data Science, Southern University of Science and Technology · School of Statistics & KLATASDS-MOE, East China Normal University · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen +1
Exponential Hardness of Off-Policy Evaluation under History-Dependent Logging
Sep 16, 2026Pranaya JajooOff-Policy Evaluation Sample Complexity
University of Alberta
Rank Without an Oracle: Deviation-Aware Interaction-Rank Selection from Offline Multi-Agent Logs
Sep 8, 2026Xiangwu Wang, Chengwei Cao, Hongyuan TangOff-Policy Evaluation Game Theory
University of Hong Kong · University of California, San Diego · Carnegie Mellon University
Confounding Masquerading as Improvement: A Systematic Evaluation of Offline Reinforcement Learning for Stroke Antithrombotic Treatment in a 129,000-Patient Registry
Aug 31, 2026Kihun RheeOff-Policy Evaluation Offline RL
AI Research Center, JLK Inc., Seoul, South Korea
Adaptive Doubly Robust Off-Policy Evaluation for Ranking Policies under Diverse User Behavior
Aug 30, 2026Kosuke Iguchi, Ren KishimotoOff-Policy Evaluation Doubly Robust Estimation
Institute of Science Tokyo Tokyo, Japan
Off-Policy Evaluation for Semantic ID Recommenders: Does the Model's Own Code Hierarchy Help?
Aug 28, 2026Artem BetleiOff-Policy Evaluation Semantic ID Recommendation
Criteo AI Lab Grenoble, France
Bellman Calibration for Marginalized Importance Weighting in Offline Reinforcement Learning
Aug 25, 2026Lars van der Laan, Nathan KallusOff-Policy Evaluation Offline RL
Stanford University · Netflix and Cornell University
Robust Data-Collection Policy Learning for Low-Variance Online Policy Evaluation
Aug 25, 2026Claire Chen, Shuze Daniel Liu, Licheng Luo +3Policy Learning Policy Evaluation
California Institute of Technology · Massachusetts Institute of Technology · Purdue University +3

Oct 26, 2026J/K move · Enter open · S save

Model Selection for Off-Policy Evaluation via Pairwise Bellman Validation

Lars van der Laan, Vasilis Syrgkanis, Nathan Kallus

Stanford University · Netflix and Cornell University

Off-Policy Evaluation Value Function Estimation

Open paper AskPDF arXiv

Model selection and hyperparameter tuning pose a central challenge in off-policy evaluation (OPE) because errors of fitted Q-functions and occupancy-ratio estimates cannot be directly evaluated from held-out transitions. Bellman-residual and minimax validation methods often require auxiliary regressions or optimization over separate critic classes, adding computational and tuning costs. We introduce pairwise Bellman validation (PBV), a tuning-free minimax procedure for selecting among finite classes of fitted Q-functions and occupancy-ratio estimators using held-out transitions. PBV constructs critics from pairwise candidate differences, requiring only finitely many empirical moment calculations without auxiliary validation models or separate critic optimization. We establish finite-sample oracle inequalities relative to the best available candidate, even under candidate-class misspecification. Our key insight is that Q-function errors and log ratios between candidate and true occupancy ratios provide oracle critics whose Bellman and adjoint Bellman moments control estimation error. Pairwise candidate differences approximate these unknown critics, while penalties account for comparator approximation error. For Q-function selection, we first develop unweighted PBV, which evaluates pairwise Bellman moments under the offline distribution and admits oracle guarantees under a pairwise coercivity condition relating the offline distribution to the target-policy dynamics. We then develop target-aligned PBV (TA-PBV), which uses marginalized importance weighting to provide error control without pairwise coercivity. For occupancy-ratio selection, we develop adjoint PBV (A-PBV), which evaluates adjoint Bellman moments against log-ratio critics. On Hopper and Deep OPE, unweighted PBV performs competitively, while target-occupancy weighting can improve performance in settings with controlled overlap.