Contextual Bandits

Follow this topic Ask about this topic

Contextual bandits study how to choose an action using information about the current situation when feedback reveals the reward only for the action taken. Common approaches include linear and generalized linear models with UCB or Thompson sampling, alongside kernel, neural, and policy-based methods for richer reward patterns. Offline learning and evaluation, nonstationary or corrupted feedback, and constraints such as budgets, privacy, and safety extend these methods to settings including recommendation, pricing, and LLM routing. A central challenge is balancing exploration with reliable decisions when models are misspecified, feedback is sparse or delayed, and computational or operational costs matter.

Latest papers

152 papers20 papers in the last four weeks, up 150% on the four weeks before. 0.1% of all new papers.

When Do We Need LLMs? A Diagnostic for Language-Driven Bandits
Apr 7, 2026Uljad Berdica, Fernando Acero, Anton Ipsen +3Contextual Bandits LLM Decision-Making
University of Oxford · J.P. Morgan AI Research
Jump Start or False Start? A Theoretical and Empirical Evaluation of LLM-initialized Bandits
Apr 2, 2026Adam Bayley, Xiaodan Zhu, Raquel Aoki +2Contextual Bandits Large Language Model-Based Recommendation
Department of Electrical and Computer Engineering, Queen’s University · RBC Borealis
ParetoBandit: Budget-Paced Adaptive Routing for Non-Stationary LLM Serving
Mar 31, 2026Annette Taberner-MillerAdaptive Model Routing LLM Routing
Independent Researcher.
DAK-UCB: Diversity-Aware Prompt Routing for LLMs and Generative Models
Mar 24, 2026Donya Jafari, Farzan FarniaLLM Routing Contextual Bandits
Sharif University of Technology · The Chinese University of Hong Kong
Simple Projection-Free Algorithm for Contextual Recommendation with Logarithmic Regret and Robustness
Mar 21, 2026Shinsaku SakaueContextual Bandits Recommender Systems
Center for Advanced Intelligence Project, RIKEN, Tokyo, Japan
Kernel Single-Index Bandits: Estimation, Inference, and Learning
Mar 19, 2026Sakshi Arya, Satarupa Bhattacharjee, Bharath K. SriperumbudurContextual Bandits Single-Index Models
Department of Mathematics, Applied Mathematics, and Statistics, Case Western Reserve University · Department of Statistics, University of Florida · Department of Statistics, The Pennsylvania State University
Off-Policy Learning with Limited Supply
Mar 19, 2026Koichi Tanaka, Ren Kishimoto, Bushun Kawagishi +4Contextual Bandits Off-Policy RL
Institute of Science Tokyo Tokyo, Japan · Meiji University Tokyo, Japan · Yale University New Haven, CT, USA +2
A Practical Algorithm for Feature-Rich, Non-Stationary Bandit Problems
Mar 17, 2026Wei Min Loh, Sajib Kumer Sinha, Ankur Agarwal +1Bandit Learning Multi-Armed Bandits
University of Waterloo, Vector Institute · Manulife Financial
Robust and Computationally Efficient Linear Contextual Bandits under Adversarial Corruption and Heavy-Tailed Noise
Mar 16, 2026Naoto Tani, Futoshi FutamiContextual Bandits Linear Contextual Bandits
The University of Osaka · RIKEN AIP · The University of Tokyo
Mixing Makes Markovian Contexts Cheap for Linear Bandits
Mar 13, 2026Kaan Buyukkalayci, Osama Hanna, Christina FragouliMarkovian Bandits Contextual Bandits
University of California, Los Angeles · Meta, Superintelligence Lab
RIE-Greedy: Regularization-Induced Exploration for Contextual Bandits
Mar 11, 2026Tong Li, Thiago de Queiroz Casanova, Eric M. Schwartz +3Contextual Bandits Exploration-Exploitation Tradeoff
University of Toronto Canada · Braze United States · University of Michigan United States
What Do We Care About in Bandits with Noncompliance? BRACE: Bandits with Recommendations, Abstention, and Certified Effects
Mar 10, 2026Nicolás Della PennaContextual Bandits Instrumental Variable Estimation
Grouplang
From Weighting to Modeling: A Nonparametric Estimator for Off-Policy Evaluation
Mar 10, 2026Rong J. B. ZhuOff-Policy Evaluation Inverse Probability Weighting
Institute of Science and Technology for Brain-inspired Intelligence, Fudan University
Learning When to Trust in Contextual Social Bandits
Mar 9, 2026Majid Ghasemi, Mark CrowleyLLM Sycophancy Corruption Robustness
Department of Electrical & Computer Engineering University of Waterloo, Canada
Beam-aware Kernelized Contextual Bandits for User Association and Beamforming in mmWave Vehicular Networks
Mar 8, 2026Xiaoyang He, Manabu TsukadaWireless Communications Wireless Resource Allocation
Graduate School of Electronics and Communication Engineering, Sun Yat-sen University, China · Graduate School of Information Science and Technology, The University of Tokyo, Japan
Bilateral Trade Under Heavy-Tailed Valuations: Minimax Regret without a Variance Bound
Mar 6, 2026Hangyi ZhaoMinimax Regret Regret Minimization
Adapter-Augmented Bandits for Online Multi-Constrained Multi-Modal Inference Scheduling
Mar 6, 2026Xianzhi Zhang, Yue Xu, Yinlin Zhu +4LLM Inference Scheduling Multi-Armed Bandits
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, 510006, China · School of Computing, Macquarie University, NSW 2109, Australia
Structured Exploration vs. Generative Flexibility: A Field Study Comparing Bandit and LLM Architectures for Personalised Health Behaviour Interventions
Mar 6, 2026Dominik P. Hofer, Haochen Song, Rania Islambouli +5Personalized Language Models Contextual Bandits
Ludwig Boltzmann Institute for Digital Health and Prevention, Salzburg, Austria · University of Toronto, Toronto, Ontario, Canada · Stanford University, Stanford, California, United States of America
Online Learning for Multi-Layer Hierarchical Inference under Partial and Policy-Dependent Feedback
Mar 4, 2026Haoran Zhang, Seohyeon Cha, Hasan Burhan Beytur +3Adaptive Model Routing Contextual Bandits
Department of Electrical and Computer Engineering, The University of Texas at Austin · DEVCOM Army Research Laboratory
Steering Frozen LLMs: Adaptive Social Alignment via Online Prompt Routing
Mar 4, 2026Zeyu Zhang, Xiangxiang Dai, Ziyi Han +2LLM Alignment LLM Safety Alignment
The Chinese University of Hong Kong · University of Washington
Inverse Contextual Bandits without Rewards: Learning from a Non-Stationary Learner via Suffix Imitation
Mar 4, 2026Yuqi Kong, Xiao Zhang, Weiran ShenContextual Bandits Imitation Learning
Gaoling School of Artificial Intelligence, Renmin University of China
No One Size Fits All: QueryBandits for Hallucination Mitigation
Feb 23, 2026Nicole Cho, William Watson, Alec Koppel +2LLM Hallucination Mitigation Query Reformulation
JPMorgan AI Research, New York, NY, USA
A Jointly Efficient and Near-Optimal Algorithm for Heteroskedastic Generalized Linear Bandits with Adversarial Corruptions
Feb 11, 2026Sanghwa Kim, Junghyun Lee, Se-Young YunMulti-Armed Bandits Online Mirror Descent
Kim Jaechul Graduate School of AI, KAIST Seoul 02455, Republic of Korea
Taming the Monster Every Context: Complexity Measure and Unified Framework for Offline-Oracle Efficient Contextual Bandits
Feb 10, 2026Hao Qin, Chicheng ZhangExploration-Exploitation Tradeoff Contextual Bandits
The University of Arizona
COBALT: Contextual Bandit Learning with Offline Trajectories for Multi-Turn Code Generation
Feb 3, 2026Ziru Chen, Dongdong Chen, Ruinan Jin +3Code Generation RL for Code Generation
The Ohio State University. · Microsoft.
Learning to Route and Schedule LLMs from User Retrials via Contextual Queueing Bandits
Feb 2, 2026Seoungbin Bae, Junyoung Son, Dabeen LeeLLM Serving LLM Routing
Department of Industrial & Systems Engineering, KAIST · Graduate School of Data Science, KAIST · Department of Mathematical Sciences, Seoul National University
Empirical Bayesian Multi-Bandit Learning
Oct 30, 2025Xia Jiang, Rong J. B. ZhuContextual Bandits Empirical Bayes
Fudan University, Institute of Science and Technology for Brain-inspired Intelligence, Shanghai, People’s Republic of China
Learning Peer Influence Probabilities with Linear Contextual Bandits
Oct 21, 2025Ahmed Sayeed Faruk, Mohammad Shahverdikondori, Elena ZhelevaMulti-Armed Bandits Contextual Bandits
University of Illinois Chicago, Chicago, USA · École Polytechnique Fédérale de Lausanne, Switzerland
Provable Anytime Ensemble Sampling Algorithms in Nonlinear Contextual Bandits
Oct 12, 2025Jiazheng Sun, Weixin Wang, Pan XuEnsemble Learning Contextual Bandits
Duke University
Best-of-Both Worlds for linear contextual bandits with paid observations
Oct 8, 2025Nathan Boyer, Dorian Baudry, Patrick RebeschiniMulti-Armed Bandits Follow-the-Regularized-Leader
ENS Paris-Saclay, France · Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LIG, 38000 Grenoble, France · Department of Statistics, University of Oxford

Apr 7, 2026J/K move · Enter open · S save

When Do We Need LLMs? A Diagnostic for Language-Driven Bandits

Uljad Berdica, Fernando Acero, Anton Ipsen, Parisa Zehtabi+2

University of Oxford · J.P. Morgan AI Research

Contextual Bandits LLM Decision-Making LLM Uncertainty Estimation

Open paper AskPDF arXiv

We study Contextual Multi-Armed Bandits (CMABs) for non-episodic decision-making problems where the context includes both textual and numerical information (e.g., recommendation systems, dynamic portfolio adjustments, offer selection; all frequent problems in finance). While Large Language Models (LLMs) are increasingly applied to these settings, utilizing LLMs for reasoning at every decision step is computationally expensive, and uncertainty estimates are difficult to obtain. To address this, we introduce LLMP-UCB, a bandit algorithm that derives uncertainty estimates from LLMs via repeated inference. However, our experiments demonstrate that lightweight numerical bandits operating on text embeddings (dense or Matryoshka) match or exceed the accuracy of LLM-based solutions at a fraction of their cost. We further show that embedding dimensionality is a practical lever on the exploration-exploitation balance, enabling cost-performance tradeoffs without prompt complexity. Finally, to guide practitioners, we propose a geometric diagnostic based on the arms' embeddings to decide when to use LLM-driven reasoning versus a lightweight numerical bandit. Our results provide a principled deployment framework for cost-effective, uncertainty-aware decision systems with broad applicability across AI use cases.