Beyond Augmented-Action Surrogates for Multi-Expert Learning-to-Defer

Yannis Montreuil; Axel Carlier; Lai Xing Ng; Wei Tsang Ooi

stat.MLApr 10, 2026

Beyond Augmented-Action Surrogates for Multi-Expert Learning-to-Defer

Authors: Yannis Montreuil, Axel Carlier, Lai Xing Ng, Wei Tsang Ooi

Organizations: School of Computing, National University of Singapore, Singapore, 117418 · Fédération ENAC ISAE-SUPAERO ONERA, Université de Toulouse, Toulouse, 31555, France · Agency for Science, Technology and Research, Institute for Infocomm Research, Singapore, 138634

Abstract

A learning-to-defer (L2D) system decides, for each input, whether to predict on its own or to hand it to one of several available experts. The very well established recipe trains classifier and router jointly by treating the $K$ classes and $J$ experts as competing actions in one shared $(K{+}J)$ -action geometry. Subsequent work has proposed a series of incremental fixes within this geometry; we show that each still suffers, to varying severity, from an optimization-level pathology (target distortion, gradient amplification, winner-take-all starvation, set-mass collapse, or class-expert coupling) even under statistical consistency. We step outside the augmented-action family entirely and propose a decoupled surrogate: a softmax classifier head and an independent sigmoid head per expert, mirroring the two natural objects of the problem. We show that per-sample updates are then coordinatewise and the class-expert Hessian block is identically zero, and prove an excess-risk bound with calibration constant $\max\{2\sqrt{2},\sqrt{2J/λ}\}$ -- to our knowledge the first multi-expert L2D guarantee whose constant does not grow with the expert pool when the per-expert weight is held fixed. On controlled synthetic studies and on CIFAR-10, CIFAR-10H, and Covertype, it is the only method in our comparison that remains stable as the expert pool grows, preserves rare specialists, and improves over a standalone classifier on every real-data benchmark.

Figures & tables

		Add. CE	PiCCE	Mao25	A-SM	OvA	Decoupled
Conditional minimizer recovers the Bayes pair $(\eta,\alpha)$	(i)	✗ ( 14 )	✗ ( E.3 )	✗ ( 6 )	✓ ( 17 )	✓ ( 19 )	✓ ( 10 )
Per-sample gradient and Hessian free of $O(\|\mathcal{J}\|)$ prefactor	(ii)	✗ ( 4 )	✓ ( E.4 )	✓ ( 6 )	✓ ( 7 )	✓ ( H.3 )	✓ ( 9 )
No winner-take-all suppression of correct experts	(ii)	✓ ( 4 )	✗ ( 5 )	✓ ( 6 )	✓ ( 7 )	✓ ( 19 )	✓ ( 9 )
No class–expert coupling in the per-sample gradient	(ii)	✗ ( 4 )	✗ ( 5 )	✗ ( 6 )	✗ ( 7 )	✓ ( H.3 )	✓ ( 12 )
Class head is a categorical posterior throughout training	(i)	✗ ( 3 )	✗ ( 3 )	✗ ( 3 )	✓ ( 17 )	✗ ( 19 )	✓ ( 8 )
Quantitative surrogate-to-target excess-risk bound	(th)	✓ †	✗	✓ ‡	✗	✗	✓ ( 10 )

Table 1: Structural properties of multi-expert L2D surrogates. ✓ = satisfied; ✗ = violated. Columns (i) / (ii) / (th) : statistical target / local geometry / theoretical guarantee. Each entry cites the proposition, definition, equation, or appendix subsection that establishes it.

	Sys. Acc. $\uparrow$			Cls. Acc. $\uparrow$
Method	$J{=}8$	$J{=}16$	$J{=}32$	$J{=}8$	$J{=}16$	$J{=}32$
Decoupled	$\mathbf{.918}_{\pm.001}$	$\mathbf{.919}_{\pm.003}$	$\mathbf{.919}_{\pm.002}$	$\underline{.909}_{\pm.001}$	$\underline{.909}_{\pm.002}$	$\mathbf{.908}_{\pm.003}$
Classif. only	$\underline{.910}_{\pm.002}$	$\underline{.911}_{\pm.001}$	$\underline{.908}_{\pm.002}$	$\mathbf{.910}_{\pm.002}$	$\mathbf{.911}_{\pm.001}$	$\mathbf{.908}_{\pm.002}$
OvA	$.901_{\pm.001}$	$.890_{\pm.002}$	$.880_{\pm.003}$	$.875_{\pm.001}$	$.859_{\pm.004}$	$.844_{\pm.003}$
PiCCE	$.858_{\pm.007}$	$.860_{\pm.011}$	$.859_{\pm.008}$	$.888_{\pm.008}$	$.895_{\pm.006}$	$.880_{\pm.014}$
A-SM	$.860_{\pm.002}$	$.827_{\pm.003}$	$.783_{\pm.006}$	$.831_{\pm.002}$	$.772_{\pm.003}$	$.680_{\pm.009}$

Table 2: CIFAR-10 with redundant synthetic experts (mean

\pm

std,

4

seeds). Best in bold ; second-best underlined .

	Sys. Acc. $\uparrow$			Cls. Acc. $\uparrow$
Method	$J{=}5$	$J{=}10$	$J{=}20$	$J{=}5$	$J{=}10$	$J{=}20$
Decoupled	$\mathbf{.960}_{\pm.005}$	$\mathbf{.958}_{\pm.003}$	$\mathbf{.961}_{\pm.004}$	$\mathbf{.892}_{\pm.001}$	$\mathbf{.892}_{\pm.003}$	$.889_{\pm.001}$
OvA	$\underline{.959}_{\pm.005}$	$.954_{\pm.008}$	$\underline{.956}_{\pm.006}$	$.889_{\pm.001}$	$.799_{\pm.203}$	$\mathbf{.890}_{\pm.002}$
A-SM	$.959_{\pm.004}$	$.952_{\pm.007}$	$.949_{\pm.005}$	$\underline{.890}_{\pm.002}$	$.747_{\pm.313}$	$.470_{\pm.435}$
PiCCE	$.954_{\pm.003}$	$\underline{.957}_{\pm.004}$	$.953_{\pm.002}$	$.889_{\pm.001}$	$\underline{.890}_{\pm.002}$	$\underline{.890}_{\pm.001}$
Add. CE	$.953_{\pm.004}$	$.952_{\pm.004}$	$.951_{\pm.003}$	$.889_{\pm.002}$	$.888_{\pm.001}$	$.742_{\pm.331}$

Table 3: CIFAR-10H with real human annotators (mean

\pm

std,

5

draws). Best in bold ; second-best underlined .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Exact defer regret	System accuracy
Decoupled	$\mathbf{0.0002\,{\scriptstyle(0.0001)}}$	$\mathbf{0.993\,{\scriptstyle(0.001)}}$
PiCCE	$0.238\,{\scriptstyle(0.116)}$	$0.755\,{\scriptstyle(0.118)}$
OvA	$0.294\,{\scriptstyle(0.020)}$	$0.699\,{\scriptstyle(0.020)}$
A-SM	$0.304\,{\scriptstyle(0.035)}$	$0.688\,{\scriptstyle(0.035)}$
Mao25	$0.323\,{\scriptstyle(0.035)}$	$0.669\,{\scriptstyle(0.036)}$
Add. CE	$0.348\,{\scriptstyle(0.038)}$	$0.646\,{\scriptstyle(0.036)}$

Appendix

Table 4: Nested redundant suite at

J{=}24

. The decoupled surrogate remains essentially Bayes-optimal; all baselines degrade severely. Means with std in parentheses over 3 seeds.

	$J=16$	$J=24$	$J=32$
Decoupled defer regret	$0.013$	$0.0002$	$0.007$
Best non-decoupled defer regret	$0.226$	$0.238$	$0.293$
Best non-decoupled method	PiCCE	PiCCE	OvA

Appendix

Table 5:

J

-sweep summary. The decoupled surrogate stays near Bayes-optimal as the expert pool grows, while the best non-decoupled regret increases monotonically.

Method	Defer regret	System acc.	Coverage	Spec. sel. on $R$	Shared-correct routing
Decoupled	$\mathbf{0.003\,{\scriptstyle(0.002)}}$	$0.875\,{\scriptstyle(0.002)}$	$0.849$	$\mathbf{0.993\,{\scriptstyle(0.007)}}$	$\mathbf{0.994\,{\scriptstyle(0.007)}}$
OvA	$0.003\,{\scriptstyle(0.001)}$	$0.875\,{\scriptstyle(0.003)}$	$0.850$	$0.988\,{\scriptstyle(0.023)}$	$0.987\,{\scriptstyle(0.026)}$
A-SM	$0.003\,{\scriptstyle(0.002)}$	$0.875\,{\scriptstyle(0.003)}$	$0.850$	$0.988\,{\scriptstyle(0.017)}$	$0.988\,{\scriptstyle(0.017)}$
Mao25	$0.026\,{\scriptstyle(0.002)}$	$0.852\,{\scriptstyle(0.004)}$	$1.000$	$0.000\,{\scriptstyle(0.000)}$	$0.000\,{\scriptstyle(0.000)}$
PiCCE	$0.035\,{\scriptstyle(0.001)}$	$0.843\,{\scriptstyle(0.002)}$	$0.821$	$0.000\,{\scriptstyle(0.000)}$	$0.000\,{\scriptstyle(0.000)}$
Add. CE	$0.324\,{\scriptstyle(0.067)}$	$0.554\,{\scriptstyle(0.069)}$	$0.128$	$1.000\,{\scriptstyle(0.000)}$	$1.000\,{\scriptstyle(0.000)}$

Appendix

Table 6: Rare-specialist suite. Coverage is

\Pr(r(X){=}\mathrm{classify})

; Bayes-optimal coverage is

0.85

. PiCCE defers but never to the specialist; Mao25 never defers at all; The decoupled surrogate, OvA, and A-SM recover the specialist almost perfectly.

Method	Defer regret	System acc.	Coverage	Best expert sel. on $D$
Decoupled	$\mathbf{0.0008\,{\scriptstyle(0.001)}}$	$\mathbf{0.957\,{\scriptstyle(0.003)}}$	$0.397$	$1.000$
A-SM	$0.064\,{\scriptstyle(0.020)}$	$0.894\,{\scriptstyle(0.018)}$	$0.330$	$1.000$
OvA	$0.082\,{\scriptstyle(0.076)}$	$0.876\,{\scriptstyle(0.078)}$	$0.349$	$1.000$
PiCCE	$0.129\,{\scriptstyle(0.045)}$	$0.829\,{\scriptstyle(0.043)}$	$0.383$	$0.661$
Add. CE	$0.151\,{\scriptstyle(0.033)}$	$0.808\,{\scriptstyle(0.031)}$	$0.241$	$1.000$
Mao25	$0.154\,{\scriptstyle(0.043)}$	$0.804\,{\scriptstyle(0.045)}$	$0.324$	$0.750$

Appendix

Table 7: Pair-shared ranking suite. Chance-level best-expert selection is

0.25

; uniform-among-correct is

0.48

. Mao25 reaches

0.750

, well above chance but far below the

1.000

achieved by the decoupled surrogate and three other baselines, confirming that the loss provides partial but degraded ranking signal.

Method	$\|\mathcal{J}\|$	Gradient norm $\\|\nabla_{a}\Phi\\|_{2}$	Top eigenvalue $\lambda_{\max}(\nabla_{a}^{2}\Phi)$
Add. CE	1	$0.586\,{\scriptstyle(0.305)}$	$0.821\,{\scriptstyle(0.084)}$
Add. CE	4	$0.609\,{\scriptstyle(0.266)}$	$1.093\,{\scriptstyle(0.067)}$
Add. CE	8	$0.616\,{\scriptstyle(0.292)}$	$1.101\,{\scriptstyle(0.116)}$
Add. CE	16	$0.633\,{\scriptstyle(0.287)}$	$1.125\,{\scriptstyle(0.186)}$
Add. CE	32	$0.613\,{\scriptstyle(0.282)}$	$1.117\,{\scriptstyle(0.181)}$
Decoupled	1	$0.704\,{\scriptstyle(0.222)}$	$0.368\,{\scriptstyle(0.111)}$

Appendix

Table 8: Per-sample curvature as overlap multiplicity

|\mathcal{J}|

varies (5 seeds,

{\sim}900

samples each). The Add. CE top eigenvalue grows from

0.82

|\mathcal{J}|{=}1

1.12

|\mathcal{J}|{=}32

, consistent with the multiplicative

(1{+}|\mathcal{J}|)

factor in Proposition 4 , while also reflecting the dependence of the softmax covariance on

q

. The decoupled surrogate remains flat at

{\approx}0.36

for all

|\mathcal{J}|

, consistent with the constant

\leq 1/2

bound from ( 12 ).

Method	Mean gradient $\partial\Phi/\partial a_{K+k}$	Positive-gradient rate
Decoupled	$-0.245\,{\scriptstyle(0.103)}$	$0.000$
PiCCE	$+0.240\,{\scriptstyle(0.076)}$	$1.000$

Appendix

Table 9: Gradient on suppressed correct expert logits (5 seeds; 784 decoupled-surrogate and 392 PiCCE measurements). PiCCE produces a positive gradient on

100\%

of suppressed experts — pushing them away from deferral, exactly as Proposition 5 predicts (

2q_{K+k}>0

). The decoupled surrogate always produces a negative gradient, pulling every correct expert toward deferral.

Method	Experts $J$	Mixed-block norm $\\|H_{\mathrm{mix}}\\|_{\mathrm{op}}$
A-SM	1	$0.122\,{\scriptstyle(0.004)}$
A-SM	3	$0.207\,{\scriptstyle(0.011)}$
A-SM	5	$0.263\,{\scriptstyle(0.009)}$
A-SM	9	$0.351\,{\scriptstyle(0.009)}$
A-SM	17	$0.475\,{\scriptstyle(0.012)}$
Decoupled	1	$0.000\,{\scriptstyle(0.000)}$

Appendix

Table 10: Mixed class–expert Hessian block operator norm (5 seeds, 128 samples each). A-SM coupling grows monotonically from

0.12

J{=}1

0.48

J{=}17

, consistent with the

O(\sqrt{J})

scaling from Proposition 7 . The decoupled surrogate is identically zero for all

J

, confirming the block-diagonal structure of ( 12 ).

		Validation Metrics
Method	$J$	Sys. Acc. $\uparrow$	Defer Loss $\downarrow$	Cls. Acc.	Cov.
Decoupled	8	$\mathbf{0.918}_{\pm.001}$	$\mathbf{0.082}_{\pm.001}$	$\underline{0.909}_{\pm.001}$	$0.961_{\pm.004}$
Classifier only	8	$\underline{0.910}_{\pm.002}$	$\underline{0.090}_{\pm.002}$	$\mathbf{0.910}_{\pm.002}$	$1.000_{\pm.000}$
OvA	8	$0.901_{\pm.001}$	$0.099_{\pm.001}$	$0.875_{\pm.001}$	$0.857_{\pm.015}$
PiCCE	8	$0.858_{\pm.007}$	$0.142_{\pm.007}$	$0.888_{\pm.008}$	$0.616_{\pm.032}$
A-SM	8	$0.860_{\pm.002}$	$0.140_{\pm.002}$	$0.831_{\pm.002}$	$0.774_{\pm.014}$

Appendix

Table 11: CIFAR-10 redundant expert suite (mean

\pm

std over

4

seeds). Best values per

J

-block in bold ; second-best underlined . “Cls. Acc.”

=

classifier accuracy (ignoring the routing decision). “Cov.”

=

coverage (fraction classified by the model, not deferred).

		Test Metrics
Method	$J$	Sys. Acc. $\uparrow$	Defer Loss $\downarrow$	Cls. Acc.	Cov.
Decoupled	5	$\mathbf{0.960}\,{\scriptstyle(0.005)}$	$\mathbf{0.039}\,{\scriptstyle(0.005)}$	$\mathbf{0.892}\,{\scriptstyle(0.001)}$	$0.517\,{\scriptstyle(0.036)}$
OvA	5	$\underline{0.959}\,{\scriptstyle(0.005)}$	$\underline{0.039}\,{\scriptstyle(0.004)}$	$0.889\,{\scriptstyle(0.001)}$	$0.375\,{\scriptstyle(0.016)}$
A-SM	5	$0.959\,{\scriptstyle(0.004)}$	$0.040\,{\scriptstyle(0.004)}$	$\underline{0.890}\,{\scriptstyle(0.002)}$	$0.341\,{\scriptstyle(0.035)}$
PiCCE	5	$0.954\,{\scriptstyle(0.003)}$	$0.045\,{\scriptstyle(0.003)}$	$0.889\,{\scriptstyle(0.001)}$	$0.390\,{\scriptstyle(0.020)}$
Add. CE	5	$0.953\,{\scriptstyle(0.004)}$	$0.045\,{\scriptstyle(0.004)}$	$0.889\,{\scriptstyle(0.002)}$	$0.248\,{\scriptstyle(0.037)}$

Appendix

Table 12: CIFAR-10H with real human annotators (test metrics, mean with std over

5

annotator draws). Best values per

J

-block in bold ; second-best underlined . “Cls. Acc.”

=

classifier accuracy (ignoring the routing decision). “Cov.”

=

coverage (fraction classified by the model, not deferred).

Method	Sys. Acc. $\uparrow$	Defer Loss $\downarrow$	Cls. Acc.	Cov.
Decoupled	$\mathbf{0.934}_{\pm.000}$	$\mathbf{0.066}_{\pm.000}$	$\mathbf{0.941}_{\pm.001}$	$0.712_{\pm.040}$
Classifier only	$\underline{0.929}_{\pm.001}$	$\underline{0.071}_{\pm.001}$	$\underline{0.929}_{\pm.001}$	$1.000_{\pm.000}$
PiCCE	$0.908_{\pm.003}$	$0.092_{\pm.003}$	$0.911_{\pm.002}$	$0.449_{\pm.007}$
OvA	$0.907_{\pm.001}$	$0.093_{\pm.001}$	$0.906_{\pm.000}$	$0.437_{\pm.011}$
A-SM	$0.899_{\pm.001}$	$0.101_{\pm.001}$	$0.892_{\pm.001}$	$0.439_{\pm.007}$
Add. CE	$0.899_{\pm.001}$	$0.101_{\pm.001}$	$0.907_{\pm.001}$	$0.098_{\pm.002}$

Appendix

Table 13: Covertype with model experts (

J{=}5

, mean

\pm

std over

4

seeds). Best values in bold ; second-best underlined . “Cls. Acc.”

=

classifier accuracy (ignoring routing). “Cov.”

=

coverage (fraction classified, not deferred).

Learning to Defer

Learning to defer trains AI systems to decide when to predict autonomously and when to route cases to human or machine experts.

See all 26 papers

Feb 19, 2026

When More Experts Hurt: Underfitting in Multi-Expert Learning to Defer

Learning to Defer (L2D) enables a classifier to abstain from predictions and defer to an expert, and has recently been extended to multi-expert settings. In this work, we show that multi-expert L2D is fundamentally more challenging than the single-expert case. With multiple experts, the classifier's underfitting becomes inherent, which seriously degrades prediction performance, whereas in the single-expert setting it arises only under specific conditions. We theoretically reveal that this stems from an intrinsic expert identifiability issue: learning which expert to trust from a diverse pool, a problem absent in the single-expert case and renders existing underfitting remedies failed. To tackle this issue, we propose PiCCE (Pick the Confident and Correct Expert), a surrogate-based method that adaptively identifies a reliable expert based on empirical evidence. PiCCE effectively reduces multi-expert L2D to a single-expert-like learning problem, thereby resolving multi expert underfitting. We further prove its statistical consistency and ability to recover class probabilities and expert accuracies. Extensive experiments across diverse settings, including real-world expert scenarios, validate our theoretical results and demonstrate improved performance.

Shuqi Liu, Yuzhou Cao, Lei Feng +2

Nanyang Technological University · Southeast University

Learning to Defer

Ask PDF

May 12, 2026

Online Learning-to-Defer with Varying Experts

Learning-to-Defer (L2D) methods route each query either to a predictive model or to external experts. While existing work studies this problem in batch settings, real-world deployments require handling streaming data, changing expert availability, and shifting expert distribution. We introduce the first online L2D algorithm for multiclass classification with bandit feedback and a dynamically varying pool of experts. Our method achieves regret guarantees of

O((n+n_e)T^{2/3})

in general and

O((n+n_e)\sqrt{T})

under a low-noise condition, where

T

is the time horizon,

n

is the number of labels, and

n_e

is the number of distinct experts observed across rounds. The analysis builds on novel

\mathcal{H}

-consistency bounds for the online framework, combined with first-order methods for online convex optimization. Experiments on synthetic and real-world datasets demonstrate that our approach effectively extends standard Learning-to-Defer to settings with varying expert availability and reliability.

Dang Hoang Duy, Yannis Montreuil, Maxime Meyer +3

School of Computing, National University of Singapore, Singapore · IPAL, IRL 2955, Singapore · Department of Mathematics, National University of Singapore, Singapore, 117543 +2

Learning to Defer Bandit Learning Multiclass Classification Online Learning

Ask PDF

Oct 10, 2026

To Defer or Not to Defer? Online Learning with LLMs and Congested Experts

Modern AI systems increasingly combine large language models (LLMs) with human experts, using the LLM to handle routine queries while deferring uncertain cases for expert review. When expert capacity is limited, however, deferral decisions must account not only for the reliability of the LLM but also for congestion in the expert queue. We formulate this as an online sequential decision-making problem in which, for each query, the system observes the confidence signal of the LLM's answer and must decide whether to output the answer or defer the query to a capacity-constrained expert. The reliability of the confidence signal is initially unknown and must be learned from expert feedback, which is both selectively observed (only deferred queries are labeled) and delayed by queuing. We propose KL-PBD, a learning algorithm that uses Kullback--Leibler lower confidence bounds on the unknown reliability parameters to guide deferral decisions. For

K

confidence levels and horizon

T

, KL-PBD achieves

O(K\log T)

gap-dependent regret when the accuracy at every confidence level is bounded away from the deferral boundary, and

\widetilde{O}(\sqrt{KT})

worst-case regret. We establish matching lower bounds, showing that these rates are optimal up to logarithmic factors. Finally, we validate our approach on synthetic instances and real-world question-answering datasets, namely GSM8K, NuminaMath, and MedMCQA, using Qwen LLMs.

Vasileios Saketos, William Reveillard, Alexandre Proutiere +2

KTH Royal Institute of Technology, Stockholm, Sweden

Learning to Defer Online Learning

Ask PDF

Abstract

Learning to Defer

Explore similar work

When More Experts Hurt: Underfitting in Multi-Expert Learning to Defer

Online Learning-to-Defer with Varying Experts

To Defer or Not to Defer? Online Learning with LLMs and Congested Experts