Learning to Defer

Follow this topic Ask about this topic

Learning to defer studies how a system can combine its own predictions with expert advice while accounting for accuracy, cost, and the availability of help. Common approaches train a joint predictor and rejector, use cost-sensitive or calibrated scores with adjustable deferral thresholds, or learn routing policies through online bandit and reinforcement-learning methods. Applications include medical diagnosis, language-model tool use, and prediction with multiple experts whose skills and capacity vary. A central challenge is estimating who will be right on each case from selective or incomplete feedback, especially when expertise shifts, depends on hidden context, or is limited by workload and fatigue.

Latest papers

26 papers2 papers in the last four weeks, down 33% on the four weeks before. 0.0% of all new papers.

To Defer or Not to Defer? Online Learning with LLMs and Congested Experts
Oct 10, 2026Vasileios Saketos, William Reveillard, Alexandre Proutiere +2Learning to Defer Online Learning
KTH Royal Institute of Technology, Stockholm, Sweden
A Query Is Not a Commitment: Learning to Correct Expert Answers in Online Deferral
Oct 5, 2026Yannis Montreuil, Axel Carlier, Lai Xing Ng +1Cost-Sensitive Learning Learning to Defer
School of Computing, National University of Singapore · IRIT, Toulouse INP, France · Institute for Infocomm Research, A*STAR, Singapore
Learning to Defer with Guidance on Real World Medical Data
Sep 22, 2026Emma Sun, Joshua Strong, Alison NobleLearning to Defer Human-in-the-Loop AI
Department of Engineering Science, University of Oxford, UK
RACER: Role-Aligned Competence Estimation for Human-AI Routing
Sep 18, 2026Joshua Strong, Emma Sun, Alexander Capstick +3Learning to Defer Human-AI Decision Making
Department of Engineering Science, University of Oxford, UK
Cost-Aware Post-Hoc Deferral Under Calibration and Shift: An Environmental AI Case Study
Sep 7, 2026Haoran Yu, Lifei Liu, Danping ZhangLearning to Defer Cost-Aware Inference
University of Florida Gainesville, FL, USA · Wichita State University Wichita, KS, USA · Nanchang Hangkong University Nanchang, China
Let it Cook: Learning to Wait in Sequential Decision Making
Aug 11, 2026Christopher Watson, Arjun Krishna, Dinesh Jayaraman +1Sequential Decision Making Learning to Defer
Department of Computer and Information Science, University of Pennsylvania, USA
Multistage Defer Trees for Hybrid Interpretability: If at First You Can't Succeed, Tree Again
Jun 30, 2026Zakk Heile, Hayden McTavish, Margo Seltzer +1Ensemble Learning Learning to Defer
Department of Computer Science · Duke University · Durham, USA +2
Human-AI Teaming Through the Lens of Calibration
Jun 9, 2026Eric Nalisnick, Chi Zhang, Sophia Qian +1Learning to Defer Model Calibration
Department of Computer Science Johns Hopkins University · Department of Statistics University of Michigan
Oversight Has a Capacity: Calibrating Agent Guards to a Subjective, Fatiguing Human
Jun 8, 2026Emre TuranLearning to Defer LLM Guardrails
Independent Researcher
Learning to Assign Prediction Tasks to Agents with Capacity Constraints
May 27, 2026Shang Wu, Saatvik Kher, Padhraic SmythLearning to Defer Contextual Bandits
Department of Computer Science University of California, Irvine Irvine, CA 92697, USA
Knowing When to Ask: Segment-Level Credit Assignment for LLM Tool Use
May 27, 2026Abhijit Kumar, Zoey Wu, Mohit SuleyRL for LLM Agents Reward Design for RL
Microsoft AI Redmond, WA
Set-Valued Policy Learning
May 19, 2026Laura Fuentes-Vicente, Mathieu Even, Gaëlle Dormion +3Healthcare Learning to Defer
Inria PreMeDICaL, Inserm, Montpellier, France · Elixir Health, Paris, France · Université Paris Cité, CNRS, MAP5, F-75006 Paris, France +1
Density-Ratio Losses for Post-Hoc Learning to Defer
May 19, 2026Alexander Soen, Ragnar Thobaben, Joakim Jaldén +1Learning to Defer Density Ratio Estimation
KTH⋄ · Google Research†
Online Learning-to-Defer with Varying Experts
May 12, 2026Dang Hoang Duy, Yannis Montreuil, Maxime Meyer +3Learning to Defer Bandit Learning
School of Computing, National University of Singapore, Singapore · IPAL, IRL 2955, Singapore · Department of Mathematics, National University of Singapore, Singapore, 117543 +2
MPD $^2$ -Router: Mask-aware Multi-expert Prior-regularized Dual-head Deferral Router in Glaucoma Screening and Diagnosis
May 8, 2026Wenxin ZhanAI-Assisted Decision Making Learning to Defer
Baruch college
Coherent Hierarchical Multi-Label Learning to Defer for Medical Imaging
May 4, 2026Joshua Strong, Pramit Saha, Emma Sun +2Hierarchical Classification Multi-Label Classification
Department of Engineering Science, University of Oxford, UK · Nuffield Division of Anaesthetics, University of Oxford, The John Radcliffe Hospital, UK
Optimized Deferral for Imbalanced Settings
Apr 30, 2026Corinna Cortes, Anqi Mao, Mehryar Mohri +1Cost-Sensitive Learning Learning to Defer
Google Research, New York, NY · Courant Institute of Mathematical Sciences, New York, NY
People-Centred Medical Image Analysis via Fairness-Aware Human-AI Cooperation
Apr 28, 2026Zheng Zhang, Milad Masroor, Cuong Nguyen +6Medical Image Benchmarks Learning to Defer
CVSSP, PAI, University of Surrey, UK · Southampton Solent University, UK · AIML, Adelaide University, Australia +2
L2D-Clinical: Learning to Defer for Adaptive Model Selection in Clinical Text Classification
Apr 14, 2026Rishik Kondadadi, John E. OrtegaLearning to Defer Text Classification
University of Minnesota, Northeastern University, Minneapolis, Boston, USA
DeferredSeg:A Multi-Expert Deferral Framework for Medical Image Segmentation
Apr 14, 2026Qiuyu Tian, Haoliang Sun, Yunshan Wang +2Medical Image Segmentation Learning to Defer
Beyond Augmented-Action Surrogates for Multi-Expert Learning-to-Defer
Apr 10, 2026Yannis Montreuil, Axel Carlier, Lai Xing Ng +1Learning to Defer
School of Computing, National University of Singapore, Singapore, 117418 · Fédération ENAC ISAE-SUPAERO ONERA, Université de Toulouse, Toulouse, 31555, France · Agency for Science, Technology and Research, Institute for Infocomm Research, Singapore, 138634
Fatigue-Aware Learning to Defer via Constrained Optimisation
Apr 1, 2026Zheng Zhang, Cuong C. Nguyen, David Rosewarne +2Learning to Defer Human-in-the-Loop AI
Centre for Vision, Speech and Signal Processing University of Surrey Guildford, Surrey, UK
Consistent Learning-to-Defer with Expert-Conditional Advice
Mar 15, 2026Yannis Montreuil, Leïna Montreuil, Axel Carlier +2Learning to Defer Cost-Sensitive Learning
School of Computing National University of Singapore Singapore, 117418 · Département de Mathématiques Sorbonne University Paris, 75005, France · Fédération ENAC ISAE-SUPAERO ONERA Université de Toulouse Toulouse, 31555, France +1
Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning
Mar 9, 2026Wei Yang, Defu Cao, Jiacheng Pang +2Human-in-the-Loop AI Multi-Agent LLM Systems
University of Southern California
When More Experts Hurt: Underfitting in Multi-Expert Learning to Defer
Feb 19, 2026Shuqi Liu, Yuzhou Cao, Lei Feng +2Learning to Defer
Nanyang Technological University · Southeast University
Learning-to-Defer in Non-Stationary Time Series via Switching State-Space Models
Jan 30, 2026Yannis Montreuil, Letian Yu, Axel Carlier +2Cost-Sensitive Learning Adaptive Model Routing
School of Computing National University of Singapore · Université de Toulouse Fédération ENAC ISAE-SUPAERO ONERA · Institute for Infocomm Research A*STAR, Singapore

Oct 10, 2026J/K move · Enter open · S save

To Defer or Not to Defer? Online Learning with LLMs and Congested Experts

Vasileios Saketos, William Reveillard, Alexandre Proutiere, Sebastian Dalleiger+1

KTH Royal Institute of Technology, Stockholm, Sweden

Learning to Defer Online Learning

Open paper AskPDF arXiv

Modern AI systems increasingly combine large language models (LLMs) with human experts, using the LLM to handle routine queries while deferring uncertain cases for expert review. When expert capacity is limited, however, deferral decisions must account not only for the reliability of the LLM but also for congestion in the expert queue. We formulate this as an online sequential decision-making problem in which, for each query, the system observes the confidence signal of the LLM's answer and must decide whether to output the answer or defer the query to a capacity-constrained expert. The reliability of the confidence signal is initially unknown and must be learned from expert feedback, which is both selectively observed (only deferred queries are labeled) and delayed by queuing. We propose KL-PBD, a learning algorithm that uses Kullback--Leibler lower confidence bounds on the unknown reliability parameters to guide deferral decisions. For

K

confidence levels and horizon

T

, KL-PBD achieves

O(K\log T)

gap-dependent regret when the accuracy at every confidence level is bounded away from the deferral boundary, and

\widetilde{O}(\sqrt{KT})

worst-case regret. We establish matching lower bounds, showing that these rates are optimal up to logarithmic factors. Finally, we validate our approach on synthetic instances and real-world question-answering datasets, namely GSM8K, NuminaMath, and MedMCQA, using Qwen LLMs.