Inverse Contextual Bandits without Rewards: Learning from a Non-Stationary Learner via Suffix Imitation

Yuqi Kong; Xiao Zhang; Weiran Shen

cs.LGMar 4, 2026

Inverse Contextual Bandits without Rewards: Learning from a Non-Stationary Learner via Suffix Imitation

Authors: Yuqi Kong, Xiao Zhang, Weiran Shen

Organizations: Gaoling School of Artificial Intelligence, Renmin University of China

Abstract

We study the Inverse Contextual Bandit (ICB) problem, in which a learner seeks to optimize a policy while an observer, who cannot access the learner's rewards and only observes actions, aims to recover the underlying problem parameters. During the learning process, the learner's behavior naturally transitions from exploration to exploitation, resulting in non-stationary action data that poses significant challenges for the observer. To address this issue, we propose a simple and effective framework called Two-Phase Suffix Imitation. The framework discards data from an initial burn-in phase and performs empirical risk minimization using only data from a subsequent imitation phase. We derive a predictive decision loss bound that explicitly characterizes the bias-variance trade-off induced by the choice of burn-in length. Despite the severe information deficit, we show that a reward-free observer can achieve a convergence rate of $\tilde O(1/\sqrt{N})$ , matching the asymptotic efficiency of a fully reward-aware learner. This result demonstrates that a passive observer can effectively uncover the optimal policy from actions alone, attaining performance comparable to that of the learner itself.

Figures & tables

\tilde{\theta}\in\argmin_{\theta\in\mathbb{R}^{d}}\frac{1}{L(N)}\sum_{t=T(N)+1}^{N}\mathbf{1}\!\left[\pi_{\theta}(\mathcal{A}_{t},X_{t})\neq\hat{a}_{t}\right].

Algorithm 1 Suffix Imitation via ERM

Panel of Figure 1 — Figure 1: Performance comparison with full horizon layout ( $d=50$ , $K$ =200). The left block illustrates LinTS performance, while the right block displays LinUCB results.

Panel of Figure 2 — Figure 2: Performance comparison of Observer strategies against the Learner baseline ( $d=50$ , $K$ =200). Figure (a) shows the results for LinTS, and Figure (b) for LinUCB. Both metrics demonstrate that the Observer (Best Achieved) outperforms the online Learner.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Diagnostic verification of Assumption 1 : (a) learner actions become increasingly predictable over time; (b) a late-trained observer agrees more with the learner on late test windows than on early ones (95% CIs over 20 seeds).

Panel of Figure 4 — Figure 4: Performance comparison with full horizon layout ( $d=20$ , $K=50$ )

Panel of Figure 5 — Figure 5: Performance comparison with full horizon layout ( $d=20$ , $K=100$ )

Panel of Figure 6 — Figure 6: Performance comparison with full horizon layout ( $d=20$ , $K=200$ )

Panel of Figure 7 — Figure 7: Performance comparison with full horizon layout ( $d=50$ , $K=50$ )

Panel of Figure 8 — Figure 8: Performance comparison with full horizon layout ( $d=50$ , $K=100$ )

Panel of Figure 9 — Figure 9: Performance comparison of Observer strategies against the Learner baseline ( $d=20$ ) under varying numbers of arms $K\in\{50,100,200\}$ . Each row corresponds to a fixed $K$ ; left: LinTS, right: LinUCB.

Panel of Figure 10 — Figure 10: Performance comparison of Observer strategies against the Learner baseline ( $d=50$ ) under varying numbers of arms $K\in\{50,100\}$ . Each row corresponds to a fixed $K$ ; left: LinTS, right: LinUCB.

Contextual Bandits

Contextual bandits learn which action to take from features of each situation when only the chosen action’s reward is observed.

See all 152 papers

Learning from a Bandit Learner
Oct 19, 2026Michela Petriconi, Nicolas Menet, Claire VernadeMulti-Armed Bandits Learning from Suboptimal Demonstrations
University of Tübingen · University of Technology Nuremberg · ETH Zurich
Statistical Inference for Misspecified Contextual Bandits
Jun 21, 2026Yongyi Guo, Ziping XuInverse Probability Weighting Adaptive Inference
Department of Statistics, University of Wisconsin–Madison · School of Data Science and Society, University of North Carolina at Chapel Hill
The Surrogate Is Not the Reward: Post-Surrogate Primary-Outcome Acquisition in Contextual Bandits
Oct 5, 2026Kyungbok Lee, Michael R. KosorokContextual Bandits
Department of Biostatistics University of North Carolina at Chapel Hill Chapel Hill, NC 27599-7420, USA

Oct 19, 2026J/K move · Enter open · S save

Learning from a Bandit Learner

Michela Petriconi, Nicolas Menet, Claire Vernade

University of Tübingen · University of Technology Nuremberg · ETH Zurich

Multi-Armed Bandits Learning from Suboptimal Demonstrations

Open paper AskPDF arXiv

Classical inverse bandit methods typically aim to recover true reward parameters. In contrast, we infer a demonstrator's subjective beliefs, specifically its internal estimates of suboptimality gaps. We consider no-regret demonstrators that follow randomized policies, such as Thompson sampling, and characterize a family of algorithms for which per-step inversion is possible. This enables gap estimates to be recovered directly from the arm-selection distribution. Our analysis identifies an inherent stability-coverage trade-off: inversion becomes ill-conditioned as the demonstrator concentrates its probability mass on a single arm, while the number of informative rounds depends on the rate at which its policy converges. We establish conditions under which inversion is reliable and provide theoretical guarantees on the resulting estimation error. Experiments on simulated data support the analysis and illustrate how these conditions affect gap recovery in practice.

Abstract

Contextual Bandits

Explore similar work

Learning from a Bandit Learner

Statistical Inference for Misspecified Contextual Bandits

The Surrogate Is Not the Reward: Post-Surrogate Primary-Outcome Acquisition in Contextual Bandits