When More Experts Hurt: Underfitting in Multi-Expert Learning to Defer

Shuqi Liu; Yuzhou Cao; Lei Feng; Bo An; Luke Ong

cs.LGFeb 19, 2026

When More Experts Hurt: Underfitting in Multi-Expert Learning to Defer

Authors: Shuqi Liu, Yuzhou Cao, Lei Feng, Bo An, Luke Ong

Organizations: Nanyang Technological University · Southeast University

Abstract

Learning to Defer (L2D) enables a classifier to abstain from predictions and defer to an expert, and has recently been extended to multi-expert settings. In this work, we show that multi-expert L2D is fundamentally more challenging than the single-expert case. With multiple experts, the classifier's underfitting becomes inherent, which seriously degrades prediction performance, whereas in the single-expert setting it arises only under specific conditions. We theoretically reveal that this stems from an intrinsic expert identifiability issue: learning which expert to trust from a diverse pool, a problem absent in the single-expert case and renders existing underfitting remedies failed. To tackle this issue, we propose PiCCE (Pick the Confident and Correct Expert), a surrogate-based method that adaptively identifies a reliable expert based on empirical evidence. PiCCE effectively reduces multi-expert L2D to a single-expert-like learning problem, thereby resolving multi expert underfitting. We further prove its statistical consistency and ability to recover class probabilities and expert accuracies. Extensive experiments across diverse settings, including real-world expert scenarios, validate our theoretical results and demonstrate improved performance.

Figures & tables

Panel of Figure 1 — Figure 1: Left: Illustration of underfitting when using multi-expert CE surrogate loss proposed by Verma et al. [39] on ImageNet. We consider a MobileNet-v2 model and progressively introduce “dog experts”, where each expert covers a domain consisting of 5 dog species, attaining $85\%$ accuracy on its domain, $75\%$ on the other dog species, and random guessing on remaining classes. Since the experts have non-overlapping domains, adding more experts strictly increases the aggregate accuracy of the expert set. We report the test accuracy of both the system and the classifier. Right: An illustration of the degraded distribution for a 5-class classification task. We present the predicted class-posterior probabilities for an instance in descending order before and after the introduction of three experts.

Dataset		CIFAR-100								ImageNet
Expert Pattern		Animal Expert				Overlapped Animal Expert				Dog Expert				Overlapped Dog Expert
Loss Formulation		Vanilla		PiCCE		Vanilla		PiCCE		Vanilla		PiCCE		Vanilla		PiCCE
Method	$\#$ Exp	Err	Cov	Err	Cov	Err	Cov	Err	Cov	Err	Cov	Err	Cov	Err	Cov	Err	Cov
CE	4	18.48	74.58	18.32	77.50	16.10	64.50	15.10	67.80	42.74	89.08	42.10	89.85	41.56	88.37	41.23	89.16
	8	18.91	74.35	18.21	76.35	16.24	64.59	16.10	71.35	44.16	88.71	41.96	89.91	42.65	87.99	41.72	89.23
	12	19.08	75.18	18.19	77.43	16.99	66.09	15.84	69.86	46.36	88.50	41.50	89.78	44.37	87.85	41.77	89.14

Table 1: The mean of the system error (Err, rescaled to 0-100) and coverage (Cov) for 3 trails on the CIFAR-100 and ImageNet datasets.

Dataset	MiceBone								Chaoyang
Method	CE		PiCCE-CE		OvA		PiCCE-OvA		CE		PiCCE-CE		OvA		PiCCE-OvA
$\#$ Exp	Err	Cov	Err	Cov	Err	Cov	Err	Cov	Err	Cov	Err	Cov	Err	Cov	Err	Cov
2	15.17	60.92	15.23	69.28	14.91	72.26	14.06	83.34	1.32	53.86	1.22	63.05	1.51	32.33	1.46	37.76
4	15.88	55.09	15.17	68.11	15.02	68.24	13.80	70.33	1.75	51.68	1.60	55.86	1.90	20.37	1.56	27.76
6	15.99	51.11	14.97	62.22	15.89	63.84	13.09	66.10	2.48	45.99	1.75	52.02	2.04	14.39	1.31	23.58
8	16.72	43.62	13.35	60.72	16.72	57.62	13.03	59.96	3.35	41.23	2.04	50.90	2.48	13.17	1.02	20.38

Table 2: The mean of the system error (Err, rescaled to 0-100) and coverage (Cov) for 3 trails on the MiceBone and Chaoyang dataset.

Panel of Figure 2 — Figure 2: Classifier accuracy vs. number of experts on synthetic (CIFAR-100, ImageNet) and real-world expert datasets (MiceBone, Chaoyang). Solid lines denote methods derived from ( 4 ), while dashed lines correspond to those derived from ( 9 ). Across all datasets, existing methods exhibit a performance drop as the number of experts increases, whereas PiCCE remains stable.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Expert ID	047	290	533	534	580	581	966	745
Train Acc	84.64	85.01	87.43	88.13	81.73	85.96	87.05	85.45
Test Acc	84.64	84.71	86.33	85.68	79.59	84.64	87.88	84.90

Appendix

Table 3: Prediction accuracy (Acc,

\%

) of the eight selected experts on the MiceBone dataset.

Expert Pattern			Varying-Acc Animal Expert
Loss Formulation			Vanilla		PiCCE
Dataset	Method	$\#$ Exp	Err	Cov	Err	Cov
CIFAR-100	CE	4	18.96	75.34	18.40	77.16
		8	19.56	76.59	18.38	77.76
		12	20.18	75.31	18.21	78.09
		16	22.46	76.10	18.13	78.65

Appendix

Table 4: The mean of the system error (Err, rescaled to 0-100) and coverage (Cov) for 3 trails on the CIFAR-100 and ImageNet datasets. The best performance is highlighted in boldface.

Panel of Figure 3 — Figure 3: We report the classifier’s accuracy on CIFAR-100 and ImageNet datasets. Solid lines are the methods from existing formulation ( 4 ) while dashed lines are the methods derived from our PiCCE formulation ( 9 ).

Learning to Defer

Learning to defer trains AI systems to decide when to predict autonomously and when to route cases to human or machine experts.

See all 26 papers

Beyond Augmented-Action Surrogates for Multi-Expert Learning-to-Defer
Apr 10, 2026Yannis Montreuil, Axel Carlier, Lai Xing Ng +1Learning to Defer
School of Computing, National University of Singapore, Singapore, 117418 · Fédération ENAC ISAE-SUPAERO ONERA, Université de Toulouse, Toulouse, 31555, France · Agency for Science, Technology and Research, Institute for Infocomm Research, Singapore, 138634
Online Learning-to-Defer with Varying Experts
May 12, 2026Dang Hoang Duy, Yannis Montreuil, Maxime Meyer +3Learning to Defer Bandit Learning
School of Computing, National University of Singapore, Singapore · IPAL, IRL 2955, Singapore · Department of Mathematics, National University of Singapore, Singapore, 117543 +2
Optimized Deferral for Imbalanced Settings
Apr 30, 2026Corinna Cortes, Anqi Mao, Mehryar Mohri +1Cost-Sensitive Learning Learning to Defer
Google Research, New York, NY · Courant Institute of Mathematical Sciences, New York, NY

Apr 10, 2026J/K move · Enter open · S save

Beyond Augmented-Action Surrogates for Multi-Expert Learning-to-Defer

Yannis Montreuil, Axel Carlier, Lai Xing Ng, Wei Tsang Ooi

School of Computing, National University of Singapore, Singapore, 117418 · Fédération ENAC ISAE-SUPAERO ONERA, Université de Toulouse, Toulouse, 31555, France · Agency for Science, Technology and Research, Institute for Infocomm Research, Singapore, 138634

Learning to Defer

Open paper AskPDF arXiv

A learning-to-defer (L2D) system decides, for each input, whether to predict on its own or to hand it to one of several available experts. The very well established recipe trains classifier and router jointly by treating the

K

classes and

J

experts as competing actions in one shared

(K{+}J)

-action geometry. Subsequent work has proposed a series of incremental fixes within this geometry; we show that each still suffers, to varying severity, from an optimization-level pathology (target distortion, gradient amplification, winner-take-all starvation, set-mass collapse, or class-expert coupling) even under statistical consistency. We step outside the augmented-action family entirely and propose a decoupled surrogate: a softmax classifier head and an independent sigmoid head per expert, mirroring the two natural objects of the problem. We show that per-sample updates are then coordinatewise and the class-expert Hessian block is identically zero, and prove an excess-risk bound with calibration constant

\max\{2\sqrt{2},\sqrt{2J/λ}\}

-- to our knowledge the first multi-expert L2D guarantee whose constant does not grow with the expert pool when the per-expert weight is held fixed. On controlled synthetic studies and on CIFAR-10, CIFAR-10H, and Covertype, it is the only method in our comparison that remains stable as the expert pool grows, preserves rare specialists, and improves over a standalone classifier on every real-data benchmark.

Abstract

Learning to Defer

Explore similar work

Beyond Augmented-Action Surrogates for Multi-Expert Learning-to-Defer

Online Learning-to-Defer with Varying Experts

Optimized Deferral for Imbalanced Settings