We study the Inverse Contextual Bandit (ICB) problem, in which a learner seeks to optimize a policy while an observer, who cannot access the learner's rewards and only observes actions, aims to recover the underlying problem parameters. During the learning process, the learner's behavior naturally transitions from exploration to exploitation, resulting in non-stationary action data that poses significant challenges for the observer. To address this issue, we propose a simple and effective framework called Two-Phase Suffix Imitation. The framework discards data from an initial burn-in phase and performs empirical risk minimization using only data from a subsequent imitation phase. We derive a predictive decision loss bound that explicitly characterizes the bias-variance trade-off induced by the choice of burn-in length. Despite the severe information deficit, we show that a reward-free observer can achieve a convergence rate of O~(1/N), matching the asymptotic efficiency of a fully reward-aware learner. This result demonstrates that a passive observer can effectively uncover the optimal policy from actions alone, attaining performance comparable to that of the learner itself.
Figure 1: Performance comparison with full horizon layout ( d=50 , K =200). The left block illustrates LinTS performance, while the right block displays LinUCB results.
Figure 2: Performance comparison of Observer strategies against the Learner baseline ( d=50 , K =200). Figure (a) shows the results for LinTS, and Figure (b) for LinUCB. Both metrics demonstrate that the Observer (Best Achieved) outperforms the online Learner.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Diagnostic verification of Assumption 1 : (a) learner actions become increasingly predictable over time; (b) a late-trained observer agrees more with the learner on late test windows than on early ones (95% CIs over 20 seeds).
Figure 4: Performance comparison with full horizon layout ( d=20 , K=50 )
Figure 5: Performance comparison with full horizon layout ( d=20 , K=100 )
Figure 6: Performance comparison with full horizon layout ( d=20 , K=200 )
Figure 7: Performance comparison with full horizon layout ( d=50 , K=50 )
Figure 8: Performance comparison with full horizon layout ( d=50 , K=100 )
Figure 9: Performance comparison of Observer strategies against the Learner baseline ( d=20 ) under varying numbers of arms K∈{50,100,200} . Each row corresponds to a fixed K ; left: LinTS, right: LinUCB.
Figure 10: Performance comparison of Observer strategies against the Learner baseline ( d=50 ) under varying numbers of arms K∈{50,100} . Each row corresponds to a fixed K ; left: LinTS, right: LinUCB.
Classical inverse bandit methods typically aim to recover true reward parameters. In contrast, we infer a demonstrator's subjective beliefs, specifically its internal estimates of suboptimality gaps. We consider no-regret demonstrators that follow randomized policies, such as Thompson sampling, and characterize a family of algorithms for which per-step inversion is possible. This enables gap estimates to be recovered directly from the arm-selection distribution. Our analysis identifies an inherent stability-coverage trade-off: inversion becomes ill-conditioned as the demonstrator concentrates its probability mass on a single arm, while the number of informative rounds depends on the rate at which its policy converges. We establish conditions under which inversion is reliable and provide theoretical guarantees on the resulting estimation error. Experiments on simulated data support the analysis and illustrate how these conditions affect gap recovery in practice.
Michela Petriconi, Nicolas Menet, Claire Vernade
University of Tübingen · University of Technology Nuremberg · ETH Zurich
Contextual bandit algorithms have transformed modern experimentation by enabling real-time adaptation for personalized treatment. Yet these advantages create challenges for statistical inference due to adaptivity. We study inference with contextual-bandit data without assuming a well-specified outcome model. In this setting, we show a previously overlooked issue: standard algorithms such as LinUCB may fail to stabilize under misspecified working models, leading to non-Gaussian estimator behavior and invalid inference. This issue is practically important, as misspecified working models -- such as approximations of complex dynamical systems -- are often employed by online agents in real-world adaptive experiments to balance reward, computational tractability, and robustness. We develop an inverse-probability-weighted Z-estimation framework for a broad class of marginal moment targets, including projection parameters, structural parameters with noisy contexts, and off-policy values. We identify a stability condition tailored to this framework, scaled inverse-propensity convergence, under which the IPW-Z estimator is consistent and asymptotically normal with a consistent sandwich variance estimator. We further establish sufficient conditions for scaled inverse-propensity convergence for several policy classes, including multi-armed bandit algorithms and smooth contextual allocation policies. Simulations and a HeartSteps V1 real-data-calibrated application show reliable coverage and competitive performance across multiple targets. Overall, our results highlight the importance of stability-aware adaptive design for valid post-experiment inference.
Yongyi Guo, Ziping Xu
Department of Statistics, University of Wisconsin–Madison · School of Data Science and Society, University of North Carolina at Chapel Hill
We study contextual bandits in which a surrogate is observed after the action but before the learner decides whether to acquire the primary outcome that defines action value and regret. The value of acquiring the primary outcome depends on both decision relevance (how much the current outcome matters for comparing policies) and the residual uncertainty after observing the surrogate. The Audited Surrogate Bandit (ASB) learns a contextual policy while allocating a budget of B primary-outcome acquisitions over T rounds. ASB sets a pre-surrogate acquisition level from current decision relevance and, after observing the surrogate, redistributes that level using an estimate of that residual uncertainty. For a finite class of N policies over K actions, ASB incurs O[KTlogN{1+T/B}] regret relative to the best policy in the class. In a two-action family where the surrogate does not reveal the better action, a learner that observes the surrogate before deciding whether to acquire can achieve bounded regret, whereas any learner that must decide before seeing the surrogate incurs Ω(T/B) worst-case regret under the same budget. Synthetic experiments show that both acquisition factors matter: ASB has lower regret than variants using only decision relevance or only residual uncertainty. On a KuaiRec benchmark of user-video interactions, the regret gap relative to relevance-only acquisition widens and then narrows as the budget grows.
Kyungbok Lee, Michael R. Kosorok
Department of Biostatistics University of North Carolina at Chapel Hill Chapel Hill, NC 27599-7420, USA