Kernel weighted importance sampling for off-policy evaluation in contextual bandits

Joshua Spear; Matthieu Komorowski; Rebecca Pope; Neil J Sebire; Erica E. M. Moodie

cs.LGJul 16, 2026

Kernel weighted importance sampling for off-policy evaluation in contextual bandits

Authors: Joshua Spear, Matthieu Komorowski, Rebecca Pope, Neil J Sebire, Erica E. M. Moodie

Organizations: Institute of Child Health, University College London · Strive Health Ltd. · National Institute for Health Research · University College London, Biomedical Research Council · Great Ormond Street Hospital · Department of Epidemiology, Biostatistics and Occupational Health, McGill University

Abstract

This article presents a novel estimator for performing off-policy evaluation using only offline data for contextual bandits. The proposed estimator, Kernel-WIS is demonstrated to be asymptotically consistent and to empirically outperform strong baselines (including vanilla weighted importance sampling), particularly under complex conditions including behaviour policy miss-specification. The benefit of Kernel-WIS is derived from combining the bounded property of vanilla weighted importance sampling with the linearity of vanilla importance sampling.

Explore similar work

Statistical Inference for Misspecified Contextual Bandits
Jun 21, 2026Yongyi Guo, Ziping XuLinear Contextual Bandits Statistical Inference
Active Context Selection Improves Simple Regret in Contextual Bandits
May 19, 2026Mohammad Shahverdikondori, Jalal Etesami, Negar KiyavashOptimal Bandit Algorithms Linear Regret
Pessimistic Risk-Aware Policy Learning in Contextual Bandits
May 15, 2026Yilong Wan, Yuqiang Li, Xianyi WuLinear Contextual Bandits Lipschitz Constant

Jun 21, 2026 · stat.MLJ/K move · Enter open · S save

Statistical Inference for Misspecified Contextual Bandits

Yongyi Guo, Ziping Xu

Department of Statistics, University of Wisconsin–Madison · School of Data Science and Society, University of North Carolina at Chapel Hill

Contextual bandit algorithms have transformed modern experimentation by enabling real-time adaptation for personalized treatment. Yet these advantages create challenges for statistical inference due to adaptivity. We study inference with contextual-bandit data without assuming a well-specified outcome model. In this setting, we show a previously overlooked issue: standard algorithms such as LinUCB may fail to stabilize under misspecified working models, leading to non-Gaussian estimator behavior and invalid inference. This issue is practically important, as misspecified working models -- such as approximations of complex dynamical systems -- are often employed by online agents in real-world adaptive experiments to balance reward, computational tractability, and robustness. We develop an inverse-probability-weighted Z-estimation framework for a broad class of marginal moment targets, including projection parameters, structural parameters with noisy contexts, and off-policy values. We identify a stability condition tailored to this framework, scaled inverse-propensity convergence, under which the IPW-Z estimator is consistent and asymptotically normal with a consistent sandwich variance estimator. We further establish sufficient conditions for scaled inverse-propensity convergence for several policy classes, including multi-armed bandit algorithms and smooth contextual allocation policies. Simulations and a HeartSteps V1 real-data-calibrated application show reliable coverage and competitive performance across multiple targets. Overall, our results highlight the importance of stability-aware adaptive design for valid post-experiment inference.

Linear Contextual Bandits Statistical Inference Propensity

Open paper Ask about it PDF

Abstract

Explore similar work

Statistical Inference for Misspecified Contextual Bandits

Active Context Selection Improves Simple Regret in Contextual Bandits

Pessimistic Risk-Aware Policy Learning in Contextual Bandits