Beyond Augmented-Action Surrogates for Multi-Expert Learning-to-Defer

Yannis Montreuil; Axel Carlier; Lai Xing Ng; Wei Tsang Ooi

stat.MLApr 10, 2026

Beyond Augmented-Action Surrogates for Multi-Expert Learning-to-Defer

Authors: Yannis Montreuil, Axel Carlier, Lai Xing Ng, Wei Tsang Ooi

Organizations: School of Computing, National University of Singapore, Singapore, 117418 · Fédération ENAC ISAE-SUPAERO ONERA, Université de Toulouse, Toulouse, 31555, France · Agency for Science, Technology and Research, Institute for Infocomm Research, Singapore, 138634

Abstract

A learning-to-defer (L2D) system decides, for each input, whether to predict on its own or to hand it to one of several available experts. The very well established recipe trains classifier and router jointly by treating the $K$ classes and $J$ experts as competing actions in one shared $(K{+}J)$ -action geometry. Subsequent work has proposed a series of incremental fixes within this geometry; we show that each still suffers, to varying severity, from an optimization-level pathology (target distortion, gradient amplification, winner-take-all starvation, set-mass collapse, or class-expert coupling) even under statistical consistency. We step outside the augmented-action family entirely and propose a decoupled surrogate: a softmax classifier head and an independent sigmoid head per expert, mirroring the two natural objects of the problem. We show that per-sample updates are then coordinatewise and the class-expert Hessian block is identically zero, and prove an excess-risk bound with calibration constant $\max\{2\sqrt{2},\sqrt{2J/λ}\}$ -- to our knowledge the first multi-expert L2D guarantee whose constant does not grow with the expert pool when the per-expert weight is held fixed. On controlled synthetic studies and on CIFAR-10, CIFAR-10H, and Covertype, it is the only method in our comparison that remains stable as the expert pool grows, preserves rare specialists, and improves over a standalone classifier on every real-data benchmark.

Figures & tables

		Add. CE	PiCCE	Mao25	A-SM	OvA	Decoupled
Conditional minimizer recovers the Bayes pair $(\eta,\alpha)$	(i)	✗ ( 14 )	✗ ( E.3 )	✗ ( 6 )	✓ ( 17 )	✓ ( 19 )	✓ ( 10 )
Per-sample gradient and Hessian free of $O(\|\mathcal{J}\|)$ prefactor	(ii)	✗ ( 4 )	✓ ( E.4 )	✓ ( 6 )	✓ ( 7 )	✓ ( H.3 )	✓ ( 9 )
No winner-take-all suppression of correct experts	(ii)	✓ ( 4 )	✗ ( 5 )	✓ ( 6 )	✓ ( 7 )	✓ ( 19 )	✓ ( 9 )
No class–expert coupling in the per-sample gradient	(ii)	✗ ( 4 )	✗ ( 5 )	✗ ( 6 )	✗ ( 7 )	✓ ( H.3 )	✓ ( 12 )
Class head is a categorical posterior throughout training	(i)	✗ ( 3 )	✗ ( 3 )	✗ ( 3 )	✓ ( 17 )	✗ ( 19 )	✓ ( 8 )
Quantitative surrogate-to-target excess-risk bound	(th)	✓ †	✗	✓ ‡	✗	✗	✓ ( 10 )

Table 1: Structural properties of multi-expert L2D surrogates. ✓ = satisfied; ✗ = violated. Columns (i) / (ii) / (th) : statistical target / local geometry / theoretical guarantee. Each entry cites the proposition, definition, equation, or appendix subsection that establishes it.

	Sys. Acc. $\uparrow$			Cls. Acc. $\uparrow$
Method	$J{=}8$	$J{=}16$	$J{=}32$	$J{=}8$	$J{=}16$	$J{=}32$
Decoupled	$\mathbf{.918}_{\pm.001}$	$\mathbf{.919}_{\pm.003}$	$\mathbf{.919}_{\pm.002}$	$\underline{.909}_{\pm.001}$	$\underline{.909}_{\pm.002}$	$\mathbf{.908}_{\pm.003}$
Classif. only	$\underline{.910}_{\pm.002}$	$\underline{.911}_{\pm.001}$	$\underline{.908}_{\pm.002}$	$\mathbf{.910}_{\pm.002}$	$\mathbf{.911}_{\pm.001}$	$\mathbf{.908}_{\pm.002}$
OvA	$.901_{\pm.001}$	$.890_{\pm.002}$	$.880_{\pm.003}$	$.875_{\pm.001}$	$.859_{\pm.004}$	$.844_{\pm.003}$
PiCCE	$.858_{\pm.007}$	$.860_{\pm.011}$	$.859_{\pm.008}$	$.888_{\pm.008}$	$.895_{\pm.006}$	$.880_{\pm.014}$
A-SM	$.860_{\pm.002}$	$.827_{\pm.003}$	$.783_{\pm.006}$	$.831_{\pm.002}$	$.772_{\pm.003}$	$.680_{\pm.009}$

Table 2: CIFAR-10 with redundant synthetic experts (mean

\pm

std,

4

seeds). Best in bold ; second-best underlined .

	Sys. Acc. $\uparrow$			Cls. Acc. $\uparrow$
Method	$J{=}5$	$J{=}10$	$J{=}20$	$J{=}5$	$J{=}10$	$J{=}20$
Decoupled	$\mathbf{.960}_{\pm.005}$	$\mathbf{.958}_{\pm.003}$	$\mathbf{.961}_{\pm.004}$	$\mathbf{.892}_{\pm.001}$	$\mathbf{.892}_{\pm.003}$	$.889_{\pm.001}$
OvA	$\underline{.959}_{\pm.005}$	$.954_{\pm.008}$	$\underline{.956}_{\pm.006}$	$.889_{\pm.001}$	$.799_{\pm.203}$	$\mathbf{.890}_{\pm.002}$
A-SM	$.959_{\pm.004}$	$.952_{\pm.007}$	$.949_{\pm.005}$	$\underline{.890}_{\pm.002}$	$.747_{\pm.313}$	$.470_{\pm.435}$
PiCCE	$.954_{\pm.003}$	$\underline{.957}_{\pm.004}$	$.953_{\pm.002}$	$.889_{\pm.001}$	$\underline{.890}_{\pm.002}$	$\underline{.890}_{\pm.001}$
Add. CE	$.953_{\pm.004}$	$.952_{\pm.004}$	$.951_{\pm.003}$	$.889_{\pm.002}$	$.888_{\pm.001}$	$.742_{\pm.331}$

Table 3: CIFAR-10H with real human annotators (mean

\pm

std,

5

draws). Best in bold ; second-best underlined .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Exact defer regret	System accuracy
Decoupled	$\mathbf{0.0002\,{\scriptstyle(0.0001)}}$	$\mathbf{0.993\,{\scriptstyle(0.001)}}$
PiCCE	$0.238\,{\scriptstyle(0.116)}$	$0.755\,{\scriptstyle(0.118)}$
OvA	$0.294\,{\scriptstyle(0.020)}$	$0.699\,{\scriptstyle(0.020)}$
A-SM	$0.304\,{\scriptstyle(0.035)}$	$0.688\,{\scriptstyle(0.035)}$
Mao25	$0.323\,{\scriptstyle(0.035)}$	$0.669\,{\scriptstyle(0.036)}$
Add. CE	$0.348\,{\scriptstyle(0.038)}$	$0.646\,{\scriptstyle(0.036)}$

Appendix

Table 4: Nested redundant suite at

J{=}24

. The decoupled surrogate remains essentially Bayes-optimal; all baselines degrade severely. Means with std in parentheses over 3 seeds.

	$J=16$	$J=24$	$J=32$
Decoupled defer regret	$0.013$	$0.0002$	$0.007$
Best non-decoupled defer regret	$0.226$	$0.238$	$0.293$
Best non-decoupled method	PiCCE	PiCCE	OvA

Appendix

Table 5:

J

-sweep summary. The decoupled surrogate stays near Bayes-optimal as the expert pool grows, while the best non-decoupled regret increases monotonically.

Method	Defer regret	System acc.	Coverage	Spec. sel. on $R$	Shared-correct routing
Decoupled	$\mathbf{0.003\,{\scriptstyle(0.002)}}$	$0.875\,{\scriptstyle(0.002)}$	$0.849$	$\mathbf{0.993\,{\scriptstyle(0.007)}}$	$\mathbf{0.994\,{\scriptstyle(0.007)}}$
OvA	$0.003\,{\scriptstyle(0.001)}$	$0.875\,{\scriptstyle(0.003)}$	$0.850$	$0.988\,{\scriptstyle(0.023)}$	$0.987\,{\scriptstyle(0.026)}$
A-SM	$0.003\,{\scriptstyle(0.002)}$	$0.875\,{\scriptstyle(0.003)}$	$0.850$	$0.988\,{\scriptstyle(0.017)}$	$0.988\,{\scriptstyle(0.017)}$
Mao25	$0.026\,{\scriptstyle(0.002)}$	$0.852\,{\scriptstyle(0.004)}$	$1.000$	$0.000\,{\scriptstyle(0.000)}$	$0.000\,{\scriptstyle(0.000)}$
PiCCE	$0.035\,{\scriptstyle(0.001)}$	$0.843\,{\scriptstyle(0.002)}$	$0.821$	$0.000\,{\scriptstyle(0.000)}$	$0.000\,{\scriptstyle(0.000)}$
Add. CE	$0.324\,{\scriptstyle(0.067)}$	$0.554\,{\scriptstyle(0.069)}$	$0.128$	$1.000\,{\scriptstyle(0.000)}$	$1.000\,{\scriptstyle(0.000)}$

Appendix

Table 6: Rare-specialist suite. Coverage is

\Pr(r(X){=}\mathrm{classify})

; Bayes-optimal coverage is

0.85

. PiCCE defers but never to the specialist; Mao25 never defers at all; The decoupled surrogate, OvA, and A-SM recover the specialist almost perfectly.

Method	Defer regret	System acc.	Coverage	Best expert sel. on $D$
Decoupled	$\mathbf{0.0008\,{\scriptstyle(0.001)}}$	$\mathbf{0.957\,{\scriptstyle(0.003)}}$	$0.397$	$1.000$
A-SM	$0.064\,{\scriptstyle(0.020)}$	$0.894\,{\scriptstyle(0.018)}$	$0.330$	$1.000$
OvA	$0.082\,{\scriptstyle(0.076)}$	$0.876\,{\scriptstyle(0.078)}$	$0.349$	$1.000$
PiCCE	$0.129\,{\scriptstyle(0.045)}$	$0.829\,{\scriptstyle(0.043)}$	$0.383$	$0.661$
Add. CE	$0.151\,{\scriptstyle(0.033)}$	$0.808\,{\scriptstyle(0.031)}$	$0.241$	$1.000$
Mao25	$0.154\,{\scriptstyle(0.043)}$	$0.804\,{\scriptstyle(0.045)}$	$0.324$	$0.750$

Appendix

Table 7: Pair-shared ranking suite. Chance-level best-expert selection is

0.25

; uniform-among-correct is

0.48

. Mao25 reaches

0.750

, well above chance but far below the

1.000

achieved by the decoupled surrogate and three other baselines, confirming that the loss provides partial but degraded ranking signal.

Method	$\|\mathcal{J}\|$	Gradient norm $\\|\nabla_{a}\Phi\\|_{2}$	Top eigenvalue $\lambda_{\max}(\nabla_{a}^{2}\Phi)$
Add. CE	1	$0.586\,{\scriptstyle(0.305)}$	$0.821\,{\scriptstyle(0.084)}$
Add. CE	4	$0.609\,{\scriptstyle(0.266)}$	$1.093\,{\scriptstyle(0.067)}$
Add. CE	8	$0.616\,{\scriptstyle(0.292)}$	$1.101\,{\scriptstyle(0.116)}$
Add. CE	16	$0.633\,{\scriptstyle(0.287)}$	$1.125\,{\scriptstyle(0.186)}$
Add. CE	32	$0.613\,{\scriptstyle(0.282)}$	$1.117\,{\scriptstyle(0.181)}$
Decoupled	1	$0.704\,{\scriptstyle(0.222)}$	$0.368\,{\scriptstyle(0.111)}$

Appendix

Table 8: Per-sample curvature as overlap multiplicity

|\mathcal{J}|

varies (5 seeds,

{\sim}900

samples each). The Add. CE top eigenvalue grows from

0.82

|\mathcal{J}|{=}1

1.12

|\mathcal{J}|{=}32

, consistent with the multiplicative

(1{+}|\mathcal{J}|)

factor in Proposition 4 , while also reflecting the dependence of the softmax covariance on

q

. The decoupled surrogate remains flat at

{\approx}0.36

for all

|\mathcal{J}|

, consistent with the constant

\leq 1/2

bound from ( 12 ).

Method	Mean gradient $\partial\Phi/\partial a_{K+k}$	Positive-gradient rate
Decoupled	$-0.245\,{\scriptstyle(0.103)}$	$0.000$
PiCCE	$+0.240\,{\scriptstyle(0.076)}$	$1.000$

Appendix

Table 9: Gradient on suppressed correct expert logits (5 seeds; 784 decoupled-surrogate and 392 PiCCE measurements). PiCCE produces a positive gradient on

100\%

of suppressed experts — pushing them away from deferral, exactly as Proposition 5 predicts (

2q_{K+k}>0

). The decoupled surrogate always produces a negative gradient, pulling every correct expert toward deferral.

Method	Experts $J$	Mixed-block norm $\\|H_{\mathrm{mix}}\\|_{\mathrm{op}}$
A-SM	1	$0.122\,{\scriptstyle(0.004)}$
A-SM	3	$0.207\,{\scriptstyle(0.011)}$
A-SM	5	$0.263\,{\scriptstyle(0.009)}$
A-SM	9	$0.351\,{\scriptstyle(0.009)}$
A-SM	17	$0.475\,{\scriptstyle(0.012)}$
Decoupled	1	$0.000\,{\scriptstyle(0.000)}$

Appendix

Table 10: Mixed class–expert Hessian block operator norm (5 seeds, 128 samples each). A-SM coupling grows monotonically from

0.12

J{=}1

0.48

J{=}17

, consistent with the

O(\sqrt{J})

scaling from Proposition 7 . The decoupled surrogate is identically zero for all

J

, confirming the block-diagonal structure of ( 12 ).

		Validation Metrics
Method	$J$	Sys. Acc. $\uparrow$	Defer Loss $\downarrow$	Cls. Acc.	Cov.
Decoupled	8	$\mathbf{0.918}_{\pm.001}$	$\mathbf{0.082}_{\pm.001}$	$\underline{0.909}_{\pm.001}$	$0.961_{\pm.004}$
Classifier only	8	$\underline{0.910}_{\pm.002}$	$\underline{0.090}_{\pm.002}$	$\mathbf{0.910}_{\pm.002}$	$1.000_{\pm.000}$
OvA	8	$0.901_{\pm.001}$	$0.099_{\pm.001}$	$0.875_{\pm.001}$	$0.857_{\pm.015}$
PiCCE	8	$0.858_{\pm.007}$	$0.142_{\pm.007}$	$0.888_{\pm.008}$	$0.616_{\pm.032}$
A-SM	8	$0.860_{\pm.002}$	$0.140_{\pm.002}$	$0.831_{\pm.002}$	$0.774_{\pm.014}$

Appendix

Table 11: CIFAR-10 redundant expert suite (mean

\pm

std over

4

seeds). Best values per

J

-block in bold ; second-best underlined . “Cls. Acc.”

=

classifier accuracy (ignoring the routing decision). “Cov.”

=

coverage (fraction classified by the model, not deferred).

		Test Metrics
Method	$J$	Sys. Acc. $\uparrow$	Defer Loss $\downarrow$	Cls. Acc.	Cov.
Decoupled	5	$\mathbf{0.960}\,{\scriptstyle(0.005)}$	$\mathbf{0.039}\,{\scriptstyle(0.005)}$	$\mathbf{0.892}\,{\scriptstyle(0.001)}$	$0.517\,{\scriptstyle(0.036)}$
OvA	5	$\underline{0.959}\,{\scriptstyle(0.005)}$	$\underline{0.039}\,{\scriptstyle(0.004)}$	$0.889\,{\scriptstyle(0.001)}$	$0.375\,{\scriptstyle(0.016)}$
A-SM	5	$0.959\,{\scriptstyle(0.004)}$	$0.040\,{\scriptstyle(0.004)}$	$\underline{0.890}\,{\scriptstyle(0.002)}$	$0.341\,{\scriptstyle(0.035)}$
PiCCE	5	$0.954\,{\scriptstyle(0.003)}$	$0.045\,{\scriptstyle(0.003)}$	$0.889\,{\scriptstyle(0.001)}$	$0.390\,{\scriptstyle(0.020)}$
Add. CE	5	$0.953\,{\scriptstyle(0.004)}$	$0.045\,{\scriptstyle(0.004)}$	$0.889\,{\scriptstyle(0.002)}$	$0.248\,{\scriptstyle(0.037)}$

Appendix

Table 12: CIFAR-10H with real human annotators (test metrics, mean with std over

5

annotator draws). Best values per

J

-block in bold ; second-best underlined . “Cls. Acc.”

=

classifier accuracy (ignoring the routing decision). “Cov.”

=

coverage (fraction classified by the model, not deferred).

Method	Sys. Acc. $\uparrow$	Defer Loss $\downarrow$	Cls. Acc.	Cov.
Decoupled	$\mathbf{0.934}_{\pm.000}$	$\mathbf{0.066}_{\pm.000}$	$\mathbf{0.941}_{\pm.001}$	$0.712_{\pm.040}$
Classifier only	$\underline{0.929}_{\pm.001}$	$\underline{0.071}_{\pm.001}$	$\underline{0.929}_{\pm.001}$	$1.000_{\pm.000}$
PiCCE	$0.908_{\pm.003}$	$0.092_{\pm.003}$	$0.911_{\pm.002}$	$0.449_{\pm.007}$
OvA	$0.907_{\pm.001}$	$0.093_{\pm.001}$	$0.906_{\pm.000}$	$0.437_{\pm.011}$
A-SM	$0.899_{\pm.001}$	$0.101_{\pm.001}$	$0.892_{\pm.001}$	$0.439_{\pm.007}$
Add. CE	$0.899_{\pm.001}$	$0.101_{\pm.001}$	$0.907_{\pm.001}$	$0.098_{\pm.002}$

Appendix

Table 13: Covertype with model experts (

J{=}5

, mean

\pm

std over

4

seeds). Best values in bold ; second-best underlined . “Cls. Acc.”

=

classifier accuracy (ignoring routing). “Cov.”

=

coverage (fraction classified, not deferred).

Learning to Defer

Learning to defer trains AI systems to decide when to predict autonomously and when to route cases to human or machine experts.

See all 26 papers

When More Experts Hurt: Underfitting in Multi-Expert Learning to Defer
Feb 19, 2026Shuqi Liu, Yuzhou Cao, Lei Feng +2Learning to Defer
Nanyang Technological University · Southeast University
Online Learning-to-Defer with Varying Experts
May 12, 2026Dang Hoang Duy, Yannis Montreuil, Maxime Meyer +3Learning to Defer Bandit Learning
School of Computing, National University of Singapore, Singapore · IPAL, IRL 2955, Singapore · Department of Mathematics, National University of Singapore, Singapore, 117543 +2
To Defer or Not to Defer? Online Learning with LLMs and Congested Experts
Oct 10, 2026Vasileios Saketos, William Reveillard, Alexandre Proutiere +2Learning to Defer Online Learning
KTH Royal Institute of Technology, Stockholm, Sweden

Feb 19, 2026J/K move · Enter open · S save

When More Experts Hurt: Underfitting in Multi-Expert Learning to Defer

Shuqi Liu, Yuzhou Cao, Lei Feng, Bo An+1

Nanyang Technological University · Southeast University

Learning to Defer

Open paper AskPDF arXiv

Learning to Defer (L2D) enables a classifier to abstain from predictions and defer to an expert, and has recently been extended to multi-expert settings. In this work, we show that multi-expert L2D is fundamentally more challenging than the single-expert case. With multiple experts, the classifier's underfitting becomes inherent, which seriously degrades prediction performance, whereas in the single-expert setting it arises only under specific conditions. We theoretically reveal that this stems from an intrinsic expert identifiability issue: learning which expert to trust from a diverse pool, a problem absent in the single-expert case and renders existing underfitting remedies failed. To tackle this issue, we propose PiCCE (Pick the Confident and Correct Expert), a surrogate-based method that adaptively identifies a reliable expert based on empirical evidence. PiCCE effectively reduces multi-expert L2D to a single-expert-like learning problem, thereby resolving multi expert underfitting. We further prove its statistical consistency and ability to recover class probabilities and expert accuracies. Extensive experiments across diverse settings, including real-world expert scenarios, validate our theoretical results and demonstrate improved performance.

Abstract

Learning to Defer

Explore similar work

When More Experts Hurt: Underfitting in Multi-Expert Learning to Defer

Online Learning-to-Defer with Varying Experts

To Defer or Not to Defer? Online Learning with LLMs and Congested Experts