Contextual Bandits

Follow this topic Ask about this topic

Contextual bandits study how to choose an action using information about the current situation when feedback reveals the reward only for the action taken. Common approaches include linear and generalized linear models with UCB or Thompson sampling, alongside kernel, neural, and policy-based methods for richer reward patterns. Offline learning and evaluation, nonstationary or corrupted feedback, and constraints such as budgets, privacy, and safety extend these methods to settings including recommendation, pricing, and LLM routing. A central challenge is balancing exploration with reliable decisions when models are misspecified, feedback is sparse or delayed, and computational or operational costs matter.

Latest papers

151 papers20 papers in the last four weeks, up 150% on the four weeks before. 0.1% of all new papers.

Learning Which Device to Call: Online Tool Grounding for LLM Agents in IoT
Oct 18, 2026Yichen Liu, Xiaochong Jiang, Haoran Yu +1LLM Tool Use Tool Selection for Language Model Agents
Independent Researcher, Seattle, WA, USA · Independent Researcher, Boston, MA, USA
Learning Personalized Memory Gating from Delayed One-Sided Feedback
Oct 16, 2026Kaixuan Ji, Zixiang Chen, Wenting Zhao +9Memory Management for LLM Agents Contextual Bandits
Salesforce AI Research · University of California, Los Angeles · University of Illinois at Urbana-Champaign +1
Chance-Constrained Personalized LLM Routing with Online Preference Learning
Oct 15, 2026Ameya Anjarlekar, Rasoul Etesami, R SrikantLLM Routing Cost-Aware Inference
Coverage Cliffs in Off-Policy Quantile Certification
Oct 15, 2026Pauline Bourigault, Emmanuelle BourigaultOff-Policy Evaluation Contextual Bandits
High-Probability Regret Bounds for Feel-Good Thompson Sampling
Oct 14, 2026Xuheng Li, Qingyue Zhao, Quanquan GuThompson Sampling Contextual Bandits
Department of Computer Science, University of California, Los Angeles, CA 90095, USA
Optimal Online Regret for Context-Lumpable Bandits
Oct 13, 2026Chenyu Lu, Zihan ZhangContextual Bandits
The Hong Kong University of Science and Technology
Online LLM Personalization from Pairwise Preferences Across Users
Oct 12, 2026Romain Puech, Georgy Salakhutdinov, Dimitris BertsimasEducational QA LLM Personalization
Massachusetts Institute of Technology
Budgeted Multi-Source Counterfactual Annotation for Off-Policy Evaluation
Oct 7, 2026Biao Xiang, Ali Eshragh, Yuexing Li +1Off-Policy Evaluation Counterfactual Evaluation
Pennsylvania State University, State College, PA · Johns Hopkins Carey Business School, Washington, DC · International Computer Science Institute, Berkeley, CA +1
Amortized Off-Policy Evaluation for LLMs
Oct 7, 2026Younwoo Choi, Leo Feng, Vincent Liu +1Off-Policy Evaluation LLM Evaluation
RBC Borealis · University of Toronto · Vector Institute
Variance-Optimal Off-Policy Evaluation with Conjunct Effect Modeling
Oct 6, 2026Nicolò Felicioni, Michael Benigni, Maurizio Ferrari Dacrema +1Causal Effect Estimation Off-Policy Evaluation
Spotify · Politecnico di Milano
The Surrogate Is Not the Reward: Post-Surrogate Primary-Outcome Acquisition in Contextual Bandits
Oct 5, 2026Kyungbok Lee, Michael R. KosorokContextual Bandits
Department of Biostatistics University of North Carolina at Chapel Hill Chapel Hill, NC 27599-7420, USA
Sampling Allocation of LinUCB: Optimal Design Limits in the Small-Gap Regime
Oct 5, 2026Yujie Liu, Vincent Y. F. Tan, Yunbei XuAdaptive Inference Contextual Bandits
National University of Singapore
Learning While Scheduling Jobs under Context-Dependent Service Rates: An Anytime Rate-Optimal Algorithm
Oct 5, 2026Seoungbin Bae, Dabeen LeeMulti-Armed Bandits Contextual Bandits
Department of Industrial & Systems Engineering, KAIST · Department of Mathematical Sciences, Seoul National University
Bandits via Additive Quantized Representations
Oct 1, 2026Ami Tavory, Noam Touitou, Tal Sarig +2Contextual Bandits Residual VQ
Meta Platforms
Reserve-Aware Contrast Certificates for Conservative Bandits with Uncertain Baselines
Sep 30, 2026Qinchuan ChengMulti-Armed Bandits Contextual Bandits
School of Automation Science and Engineering, Xi’an Jiaotong University, Xi’an, China
Risk-Aware Adaptive Evaluation: Finding High-Impact Failures Under Limited Budgets
Sep 30, 2026Priyanath Maji, Spandan Ghose ChowdhuryThompson Sampling AI Agent Evaluation
Georgia Institute of Technology Atlanta, GA 30332
Nonpreemptive Scheduling While Learning Context-Dependent Service Rates
Sep 29, 2026Wansoo Choi, Seoungbin Bae, Dabeen LeeQueueing Theory Contextual Bandits
Department of Mathematical Sciences, Seoul National University · Department of Industrial & Systems Engineering, KAIST
Pointwise or Pairwise: When Do Pairwise Losses Help Reward Learning, Provably?
Sep 29, 2026Junghyun Lee, Minsoo Ha, Sanghwa Kim +4Offline-to-Online Contextual Bandits Contextual Bandits
Kim Jaechul Graduate School of AI, KAIST Seoul 02455, Republic of Korea · Graduate School of AI, POSTECH Pohang 37673, Republic of Korea · Samsung Research Seoul 06765, Republic of Korea +1
Corruption-Robust Sparse Linear Contextual Bandits with Knapsack Constraints
Sep 29, 2026Yige Wang, Hanyang Li, Yiming Zong +2Contextual Bandits Linear Contextual Bandits
Department of Industrial Engineering and Decision Analytics, The Hong Kong University of Science and Technology · Department of Statistics and Data Science, The Wharton School, University of Pennsylvania
Elicitation and Decision Geometry in Single-Index Bandits
Sep 28, 2026Sakshi Arya, Cheng Soon OngMulti-Armed Bandits Single-Index Models
Case Western Reserve University, USA · CSIRO, Australia
Vanilla Policy Optimization Is Both Optimal and Differentially Private for Stochastic Contextual Bandits
Sep 27, 2026Idan Attias, Orin Levy, Alexander Ryabchenko +2Policy Optimization Differential Privacy
University of Illinois Chicago · Toyota Technological Institute at Chicago · Tel Aviv University +3
Saturation-Insensitive Dueling Bandits with General Function Approximation
Sep 26, 2026Chenggong Zhang, Xuheng Li, Qiwei Di +2Contextual Bandits
Department of Computer Science, University of California, Los Angeles, CA 90095, USA · School of Data Science and Society, University of North Carolina, Chapel Hill, NC 27599, USA
Efficient Linear Bandits via Cluster-Aware Sketching
Sep 23, 2026Hantao Yang, Hong Xie, Defu LianMulti-Armed Bandits Randomized Sketching
University of Science and Technology of China
Nearly Minimax-Optimal Regret for Linear Contextual Bandits with Arbitrary Adaptive Action Sets
Sep 14, 2026Tianyuan JinMulti-Armed Bandits Minimax Regret
Data Science and Analytics Thrust The Hong Kong University of Science and Technology (Guangzhou)
When Greedy Sampling Explores: KL-Regularized Contextual Bandits without Eluder-Dimension Dependence
Sep 11, 2026Zichen Wang, Haoyang Hong, Huazheng WangContextual Bandits
Department of ECE and CSL, University of Illinois Urbana-Champaign · School of Electrical Engineering and Computer Science, Oregon State University
Meta-LinEXP3: Online-within-Online Learning for Adversarial Linear Contextual Bandits
Sep 9, 2026Hao Li, Jie Xu, Zheng XieMulti-Armed Bandits Contextual Bandits
College of Science, National University of Defense Technology, Changsha 410073, China
Drift-Aware LLM Routing with Sparse Contexts and Shared Budgets
Sep 1, 2026Cheung Hao Lee, Patrick WongCost-Aware Inference Non-Stationary Bandits
Nonparametric Contextual Pricing and Inventory Learning under Censored Demand
Aug 31, 2026Zean Han, Jing Liang, Ruihan Lin +2Inventory Control Learning with Missing Data
Department of Industrial Engineering and Decision Analytics The Hong Kong University of Science and Technology
Designing for the Next Click: Bandits for Real-Time Page Layout
Aug 30, 2026Bhavtosh Rath, Harshith Narasimhamurthy, Bob Eisinger +3Contextual Bandits E-Commerce
Target Corporation Minneapolis, MN, USA · Target Corporation Minneapolis, MN USA
Safety by Design: Realized-Cost Constraints for Contextual Bandits with Continuous Actions
Aug 27, 2026Spyros Dragazis, Aldo PacchianoContextual Bandits Multi-Armed Bandits
Boston University · Broad Institute of MIT and Harvard

Oct 18, 2026J/K move · Enter open · S save

Learning Which Device to Call: Online Tool Grounding for LLM Agents in IoT

Yichen Liu, Xiaochong Jiang, Haoran Yu, Lifei Liu

Independent Researcher, Seattle, WA, USA · Independent Researcher, Boston, MA, USA

LLM Tool Use Tool Selection for Language Model Agents Internet of Things Contextual Bandits

Open paper AskPDF arXiv

The Internet of Things (IoT) gives large language model (LLM) agents access to physical observations, but one requested capability is usually offered by several devices, and the agent's plan does not say which one to call. Existing work retrieves relevant tools and checks plan feasibility, without learning which eligible device returns a useful observation. Unlike previous works, we study online tool grounding: binding an already specified tool call to one physical endpoint under service cost and device participation constraints. We propose a framework that learns which physical device should execute each agent-issued call. It formulates selection as a contextual bandit problem and combines adaptive context partitions with hierarchical endpoint trees, so that execution feedback is reused across related requests and devices. Selection balances task success against service cost while respecting device capability, authorization, battery reserves, and availability. Gaussian reward aggregation provides an optional mechanism for protecting individual feedback values. Theoretical results establish a finite-horizon regret bound and differential privacy for released reward statistics under fixed, reward-independent assignments. In a simulated loading-dock workload, the proposed router achieves higher step success and better cost-adjusted service, at lower simulated device energy and latency, than nearest-device, advertised-quality, UCB1, and LinUCB selection.