Off-Policy Evaluation

Follow this topic Ask about this topic

Momentum

8 papers in the last four weeks, up 167% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 46

All topics

Efficient Best-of-N policy evaluation for inference-time alignment
Oct 7, 2026Jonas Schweisthal, Yuxin Wang, Athiya Deviyani +2Best-of-N Sampling Off-Policy Evaluation
LMU Munich & MCML · Carnegie Mellon University · Meta (work in personal capacity)
Variance-Optimal Off-Policy Evaluation with Conjunct Effect Modeling
Oct 6, 2026Nicolò Felicioni, Michael Benigni, Maurizio Ferrari Dacrema +1Causal Effect Estimation Off-Policy Evaluation
Spotify · Politecnico di Milano
Evaluation of Active Feature Acquisition Policies with Tabular Foundation Models
Oct 5, 2026Yuta Kobayashi, Divyam Madaan, Shalmali JoshiOff-Policy Evaluation Value Function Estimation
Department of Biomedical Informatics, Columbia University
Revealing After Overwriting: An Exponential POMDP OPE Lower Bound under History-Dependent Logging
Oct 4, 2026Youyu Luo, Pengzhan Zhou, Zhida Qin +4Off-Policy Evaluation Partially Observable Markov Decision Processes
Chongqing University · Beijing Institute of Technology · The Hong Kong Polytechnic University
OTROPE: Optimal Transport-based Robust Off-policy Evaluation for Large Language Models
Sep 28, 2026Liner Xiang, Wenbo Zhang, Hengrui CaiLLM Evaluation Off-Policy Evaluation
Department of Statistics, University of California, Irvine
Recommendation Ranking Off-Policy Evaluation under Ranking-Dependent Examination via Examination-Relevance Decomposition
Sep 28, 2026Riki Okamura, Toshiharu SugawaraOff-Policy Evaluation Doubly Robust Estimation
Department of Computer Science and Communications Engineering, Waseda University, Tokyo, Japan
Deep Weighted Bellman Residual Minimization for $Q^*$ Estimation
Sep 28, 2026Lican Kang, Jerry Zhijian Yang, Cheng Yuan +1Off-Policy Evaluation Value Function Estimation
Institute for Math and AI, Wuhan University, Wuhan, 430072, China · Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, 430072, China · School of Artificial Intelligence, Wuhan University, Wuhan, 430072, China +1
Optimal Sequential Annotations for Off-Policy Evaluation
Sep 22, 2026Woojin Chae, Ezinne Nwankwo, Haitong Qin +1Off-Policy Evaluation Learning with Missing Data
University of Southern California · University of California, Berkeley · University of Washington, Seattle
Model-based Bootstrap for Offline Policy Evaluation in Tabular Reinforcement Learning
Sep 17, 2026Weiwei Wang, Yuqiang Li, Xianyi Wu +1Reinforcement Learning Off-Policy Evaluation
Department of Statistics and Data Science, Southern University of Science and Technology · School of Statistics & KLATASDS-MOE, East China Normal University · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen +1
Exponential Hardness of Off-Policy Evaluation under History-Dependent Logging
Sep 16, 2026Pranaya JajooOff-Policy Evaluation Sample Complexity
University of Alberta
Rank Without an Oracle: Deviation-Aware Interaction-Rank Selection from Offline Multi-Agent Logs
Sep 8, 2026Xiangwu Wang, Chengwei Cao, Hongyuan TangOff-Policy Evaluation Game Theory
University of Hong Kong · University of California, San Diego · Carnegie Mellon University
Confounding Masquerading as Improvement: A Systematic Evaluation of Offline Reinforcement Learning for Stroke Antithrombotic Treatment in a 129,000-Patient Registry
Aug 31, 2026Kihun RheeOff-Policy Evaluation Offline RL
AI Research Center, JLK Inc., Seoul, South Korea
When Can You Trust Offline Evaluation of Equal-Cost Top-k Allocation? A Controlled, Reproducible Benchmark and Practitioner's Guide
Aug 12, 2026Binshuang LiInverse Probability Weighting Causal Effect Estimation
Independent Researcher
When Offline Evaluation Misleads: A Diagnostic Protocol for Reward and Policy Selection in Delayed-Feedback Contextual Bandits
Aug 12, 2026Sang Su Lee, Vineeth Loganathan, Shishir Dash +1Multi-Armed Bandits Off-Policy Evaluation
Thumbtack, Inc. San Francisco, CA, USA
On-Policy and Off-Policy Learning for Large Action Spaces
Jul 30, 2026Imad AoualiThompson Sampling Multi-Armed Bandits
Institut Polytechnique de Paris · Ecole nationale de la statistique et de l’administration économique · Ecole doctorale n◦574 Ecole Doctorale de Mathématique Hadamard (EDMH)
Learning from the Unseen: Offline Reinforcement Learning with Hidden Actions
Jul 28, 2026Zeyu Bian, Ying Zhou, Yifan CuiCausal Effect Estimation Off-Policy Evaluation
Department of Statistics, Florida State University · Department of Statistics, University of Connecticut · Center for Data Science, Zhejiang University
Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits
Jul 24, 2026Yuta Natsubori, Masataka Ushiku, Yuta SaitoOff-Policy Evaluation Contextual Bandits
Hakuhodo DY Holdings, Inc. · Cornell University
Kernel weighted importance sampling for off-policy evaluation in contextual bandits
Jul 16, 2026Joshua Spear, Matthieu Komorowski, Rebecca Pope +2Off-Policy Evaluation Importance Sampling
Institute of Child Health, University College London · Strive Health Ltd. · National Institute for Health Research +3
Fitted Occupancy-Ratio Evaluation without Bellman Completeness
Jul 6, 2026Lars van der Laan, Nathan KallusOff-Policy Evaluation Offline RL
Stanford University · Netflix · Cornell University
Critical Interval MSE: Toward Reliable Offline Validation for Robot Manipulation Policies
Jun 29, 2026Haoxu Huang, Tongsam Zheng, Yifan Chen +2Robot Policy Learning Off-Policy Evaluation
Tsinghua University · Shanghai Qi Zhi Institute · Spirit AI
Statistical Inference for Misspecified Contextual Bandits
Jun 21, 2026Yongyi Guo, Ziping XuInverse Probability Weighting Adaptive Inference
Department of Statistics, University of Wisconsin–Madison · School of Data Science and Society, University of North Carolina at Chapel Hill
Off-Policy Evaluation for Missingness-Aware Policies in MDPs with Rewards Missing Not at Random
Jun 18, 2026Ziheng Wei, Annie Qu, Rui MiaoReinforcement Learning Off-Policy Evaluation
Department of Statistics, University of Michigan at Ann Arbor · Department of Statistics and Applied Probability, University of California at Santa Barbara · Department of Mathematical Sciences, University of Texas at Dallas
Off-Policy Evaluation with Strategic Agents via Local Disclosure
Jun 5, 2026Kiet Q. H. Vo, Abbavaram Gowtham Reddy, Julian Rodemann +2Strategic Classification Off-Policy Evaluation
Rational Intelligence Lab, CISPA Helmholtz Center for Information Security, Saarbrücken, Germany · Relational Machine Learning Lab, CISPA Helmholtz Center for Information Security, Saarbrücken, Germany · Department of Statistics, LMU Munich, Germany +1
Autoregressive Diffusion World Models for Off-Policy Evaluation of LLM Agents
Jun 4, 2026Kaixuan Liu, Guojun Xiong, Weinan Zhang +1World Model Learning Off-Policy Evaluation
Department of Computer Science Emory University Atlanta, GA, USA · School of Computer Science Shanghai Jiao Tong University Shanghai, China
X4Val: Learning Neural Surrogates for Variance-Reduced Policy Evaluation
Jun 3, 2026Rachel Luo, Michael Watson, Apoorva Sharma +6Off-Policy Evaluation Robotic Policy Evaluation
NVIDIA Research, Santa Clara, CA, USA · Harvard University, Cambridge, MA, USA · Stanford University, Stanford, CA, USA
Bandit Simulation for Average Reward Inference
May 30, 2026Samya Praharaj, Chih-Yu Chang, Koulik Khamaru +1Multi-Armed Bandits Simulation-Based Inference
Rutgers University · Imperial College London
Certified Policy Optimisation for Nested Causal Bandits via PAC-Bayes Risk
May 28, 2026Tim Woydt, Paul-David ZuercherPAC-Bayesian Generalization Bounds Multi-Armed Bandits
ProdAxon · Department of Computer Science Technical University Darmstadt · Institute for Manufacturing Department of Engineering University of Cambridge
Quotient DAGs for Off-Policy Evaluation:Forward-Flow Importance Sampling and Exact Slate Propensities
May 28, 2026Ziwen Xie, Shaowen Xiang, Hongyu He +1Off-Policy Evaluation Importance Sampling
Shanghai Jiao Tong University · National University of Singapore
Insurance Pricing Optimization via Off-Policy Evaluation
May 27, 2026Sascha Günther, Dimitri Semenovich, Mario V. WüthrichOff-Policy Evaluation Policy Optimization
Department of Mathematics, ETH Zurich
Support-aware offline policy selection for advertising marketplaces
May 20, 2026Prashant Shekhar, Caroline HowardOff-Policy Evaluation Online Advertising
Department of Mathematics Embry-Riddle Aeronautical University, Daytona Beach, FL, USA

Oct 7, 2026, cs.LGJ/K move · Enter open · S save

Efficient Best-of-N policy evaluation for inference-time alignment

Jonas Schweisthal, Yuxin Wang, Athiya Deviyani, Stefan Feuerriegel+1

LMU Munich & MCML · Carnegie Mellon University · Meta (work in personal capacity)

Best-of-N Sampling Off-Policy Evaluation Inference-Time Language Model Alignment Doubly Robust Estimation

Open paper AskPDF arXiv

Best-of-N (BoN) is a common inference-time alignment method that selects the highest-scoring response among N samples from a reference model. Evaluating BoN policies from logged data is challenging under sample-only access because standard off-policy estimators require density ratios that depend on unavailable response likelihoods. In this paper, we propose a sample-only framework for evaluating and selecting BoN policies without access to these likelihoods. We show that the order-statistic structure of BoN allows the required density ratios to be expressed through score-rank probabilities that are estimable from samples alone. We then develop a doubly robust estimator of the BoN policy value (BoN-DR) that efficiently reuses a shared auxiliary sample pool across candidate budgets. We establish valid asymptotic inference even under reward estimator misspecification and prove the efficiency of our BoN-DR estimator. Since larger budgets can amplify errors in the score function and lead to reward overoptimization, we derive two selection rules: (i) maximizing the estimated policy value and (ii) maximizing a lower confidence bound on the improvement over the reference policy, which accounts for estimation uncertainty and provides a no-harm guarantee. Across synthetic experiments and GSM8K with multiple reference and reward models, our framework accurately estimates BoN policy values and selects effective sampling budgets.

Momentum

Related topics

Latest papers 46