Fatigue-Aware Learning to Defer via Constrained Optimisation

Zheng Zhang; Cuong C. Nguyen; David Rosewarne; Kevin Wells; Gustavo Carneiro

cs.LGApr 1, 2026

Fatigue-Aware Learning to Defer via Constrained Optimisation

Authors: Zheng Zhang, Cuong C. Nguyen, David Rosewarne, Kevin Wells, Gustavo Carneiro

Organizations: Centre for Vision, Speech and Signal Processing University of Surrey Guildford, Surrey, UK

Abstract

Learning to defer (L2D) enables human-AI cooperation by deciding when an AI system should act autonomously or defer to a human expert. Existing L2D methods, however, assume static human performance, contradicting well-established findings on fatigue-induced degradation. We propose Fatigue-Aware Learning to Defer via Constrained Optimisation (FALCON), which explicitly models workload-varying human performance using psychologically grounded fatigue curves. FALCON formulates L2D as a Constrained Markov Decision Process (CMDP) whose state includes both task features and cumulative human workload, and optimises accuracy under human-AI cooperation budgets via PPO-Lagrangian training. We further introduce FA-L2D, a benchmark that systematically varies fatigue dynamics from near-static to rapidly degrading regimes. Experiments across multiple datasets show that FALCON consistently outperforms state-of-the-art L2D methods across coverage levels, generalises zero-shot to unseen experts with different fatigue patterns, and demonstrates the advantage of adaptive human-AI collaboration over AI-only or human-only decision-making when coverage lies strictly between 0 and 1.

Figures & tables

Figure 1: Example of an L2D scenario illustrating workload-variant human performance in human–AI task allocation within a single episode. FALCON adapts deferral decisions based on both task difficulty and accumulated human fatigue. At $t=1$ , an easy task is handled by the AI while the human expert remains fresh. At $t=2$ , a challenging case is deferred to the human expert who has sufficient cognitive capacity. By $t=3$ , another hard task is still assigned to the human despite mild fatigue accumulation. At the final time step $t=T$ , severe human fatigue leads to AI handling the task to prevent performance degradation.

Panel of Figure 2 — Figure 2: (a): Examples of $\mathsf{w}(\rho)$ . The values of parameters $(w_{0},w_{\text{peak}},w_{\text{base}},k,\bar{\rho},\hat{\rho})$ in Example 1,2 and 3 are $(0.9,1,0.7,0.1,0.375,0.05)$ , $(0.8,0.95,0.5,0.09,0.5,0.025)$ and $(0.8,0.9,0.6,0.2,0.6,0.1)$ . (b): The architecture of FALCON with workload-variant human performance. A backbone model extracts visual features from the input $\mathbf{x}_{t}$ , while the cumulative human workload $\rho_{t}$ is passed through an embedding layer. The visual and workload features are concatenated and processed by a Resettable S5 layers ( Lu et al., 2023 ) to capture temporal dependencies and output the policy $\pi(\mathbf{a}_{t}|\mathbf{s}_{t})$ alongside value estimates.

Panel of Figure 3 — Algorithm 1 FALCON training procedure

Panel of Figure 4 — Algorithm 2 FALCON testing procedure

Params	Value
Activation	Relu
Clipping_Coefficient $\epsilon$	0.2
Entropy_Coefficient	0.001
Lagrangian_LR	0.035
Lagrangian_INIT $\lambda$	0.001
GAE_LAMBDA	0.95

Table 1: PPO parameters

Params	Range	Description
$w_{0}$	$\mathcal{U}(0.7,0.9)$	initial performance
$w_{\text{base}}$	$\mathcal{U}(0.4,0.5)$	minimum performance
$w_{\text{peak}}$	$\mathcal{U}(0.8,1.0)$	maximum performance
$\hat{\rho}$	$\mathcal{U}(0.025,0.1)$	relative workload at the peak performance
$\bar{\rho}$	$\mathcal{U}(0.25,0.5)$	relative workload at the inflection point of the decay phase
$k$	$\mathcal{U}(0.05,0.1)$	steepness of performance decline

Table 2: The range of parameters of human performance variation in Eq. 2 on Cifar100 dataset.

Params	Range	Description
$w_{0}$	$\mathcal{U}(0.8,0.9)$	initial performance
$w_{\text{base}}$	$\mathcal{U}(0.6,0.7)$	minimum performance
$w_{\text{peak}}$	$\mathcal{U}(0.9,1.0)$	maximum performance
$\hat{\rho}$	$\mathcal{U}(0.025,0.1)$	relative workload at the peak performance
$\bar{\rho}$	$\mathcal{U}(0.25,0.5)$	relative workload at the inflection point of the decay phase
$k$	$\mathcal{U}(0.05,0.1)$	steepness of performance decline

Table 3: The range of parameters of human performance variation in Eq. 2 on Chaoyang dataset.

Params	Range	Description
$w_{0}$	$\mathcal{U}(0.65,0.9)$	initial performance
$w_{\text{base}}$	$\mathcal{U}(0.3,0.4)$	minimum performance
$w_{\text{peak}}$	$\mathcal{U}(0.8,1.0)$	maximum performance
$\hat{\rho}$	$\mathcal{U}(0.025,0.1)$	relative workload at the peak performance
$\bar{\rho}$	$\mathcal{U}(0.25,0.5)$	relative workload at the inflection point of the decay phase
$k$	$\mathcal{U}(0.05,0.1)$	steepness of performance decline

Table 4: The range of parameters of human performance variation in Eq. 2 on FLickr10K dataset.

Params	Range	Description
$w_{0}$	$\mathcal{U}(0.8,0.9)$	initial performance
$w_{\text{base}}$	$\mathcal{U}(0.6,0.7)$	minimum performance
$w_{\text{peak}}$	$\mathcal{U}(0.9,1.0)$	maximum performance
$\hat{\rho}$	$\mathcal{U}(0.025,0.1)$	relative workload at the peak performance
$\bar{\rho}$	$\mathcal{U}(0.25,0.5)$	relative workload at the inflection point of the decay phase
$k$	$\mathcal{U}(0.05,0.1)$	steepness of performance decline

Table 5: The range of parameters of human performance variation in Eq. 2 on Micebone dataset.

Panel of Figure 5 — Figure 5: Inference time of FALCON and competing methods on Cifar100 (50 episodes).

Panel of Figure 6 — Figure 6: Accuracy-Coverage curves of several L2D strategies and FALCON on various datasets.

	Cifar100	Chaoyang	FLickr10K	MiceBone
OneStage L2D	70.87 $\pm$ 0.13	83.24 $\pm$ 0.14	63.06 $\pm$ 0.12	84.61 $\pm$ 0.12
TwoStage L2D	70.50 $\pm$ 0.15	83.15 $\pm$ 0.08	61.77 $\pm$ 0.13	84.58 $\pm$ 0.15
L2D-Pop	71.01 $\pm$ 0.11	82.65 $\pm$ 0.17	62.72 $\pm$ 0.18	83.96 $\pm$ 0.14
EA-L2D	66.26 $\pm$ 0.39	82.39 $\pm$ 0.08	63.26 $\pm$ 0.23	84.59 $\pm$ 0.12
Ours	74.01 $\pm$ 0.09	84.13 $\pm$ 0.11	64.40 $\pm$ 0.08	86.08 $\pm$ 0.13

Table 6: Quantitative comparison in terms of AUACC

(\times 100)

( Nadeem et al., 2009 ) of the SOTA L2D ( Mozannar and Sontag, 2020a ; Madras et al., 2018 ; Tailor et al., 2024 ; Strong et al., 2025 ) on the L2D datasets. The results consist of the mean and standard deviations obtained from three experiments using models trained with different random seeds. The best result per benchmark is marked in bold.

Panel of Figure 7 — Figure 7: Different human performance during testing (left column) and corresponding results with fine-tuning (middle column) and zero-shot (right column) testing on Cifar100 across all methods.

	Sustained High Performance		Normal Fatigue		Rapid Fatigue
	Fine-tuning	Zero-shot	Fine-tuning	Zero-shot	Fine-tuning	Zero-shot
OneStage L2D	78.23 $\pm$ 0.17	77.25 $\pm$ 0.13	72.67 $\pm$ 0.10	70.83 $\pm$ 0.16	66.75 $\pm$ 0.09	63.85 $\pm$ 0.11
TwoStage L2D	78.78 $\pm$ 0.14	75.22 $\pm$ 0.16	73.10 $\pm$ 0.09	71.56 $\pm$ 0.07	67.36 $\pm$ 0.14	67.49 $\pm$ 0.13
L2D-Pop	77.63 $\pm$ 0.08	76.38 $\pm$ 0.12	72.07 $\pm$ 0.11	70.35 $\pm$ 0.13	64.82 $\pm$ 0.12	64.40 $\pm$ 0.08
EA-L2D	73.46 $\pm$ 0.10	72.23 $\pm$ 0.14	67.87 $\pm$ 0.15	66.51 $\pm$ 0.12	63.97 $\pm$ 0.17	62.31 $\pm$ 0.14
Ours	79.58 $\pm$ 0.10	79.70 $\pm$ 0.12	76.93 $\pm$ 0.07	76.20 $\pm$ 0.09	72.36 $\pm$ 0.15	71.68 $\pm$ 0.07

Table 7: Quantitative comparison in terms of the Area Under Accuracy-Coverage Curve (AUACC)

(\times 100)

( Nadeem et al., 2009 ) of the SOTA L2D ( Mozannar and Sontag, 2020a ; Madras et al., 2018 ; Tailor et al., 2024 ; Strong et al., 2025 ) with three different human performance curves on the Cifar100 dataset. The results consist of the mean value obtained from three experiments using models trained with different random seeds. The best result per benchmark is marked in bold.

Figure 8: Accuracy-Coverage curves of CMDP ablation on Cifar100 dataset.

Panel of Figure 9 — Figure 9: Validation against clinical human performance data. (a) Comparison between simulated human performance in FALCON and real-world human data ( Cowley and Gale, 1997 ) . The orange line represents the radiologist’s correct recall rate, which shows a significant decline from 78% to 66% after the vigilance threshold. The green shaded area indicates the range of our simulated experts on the Chaoyang dataset, demonstrating high ecological validity. (b) Zero-shot testing results on the Chaoyang dataset.

Learning to Defer

Learning to defer trains AI systems to decide when to predict autonomously and when to route cases to human or machine experts.

See all 26 papers

To Defer or Not to Defer? Online Learning with LLMs and Congested Experts
Oct 10, 2026Vasileios Saketos, William Reveillard, Alexandre Proutiere +2Learning to Defer Online Learning
KTH Royal Institute of Technology, Stockholm, Sweden
FatigueFusion: Latent Space Fusion for Fatigue-Driven Motion Synthesis
Apr 11, 2026Iliana Loi, Konstantinos MoustakasHuman Motion Generation
Department of Electrical and Computer Engineering, University of Patras, Patras, 26504 Greece
Beyond Augmented-Action Surrogates for Multi-Expert Learning-to-Defer
Apr 10, 2026Yannis Montreuil, Axel Carlier, Lai Xing Ng +1Learning to Defer
School of Computing, National University of Singapore, Singapore, 117418 · Fédération ENAC ISAE-SUPAERO ONERA, Université de Toulouse, Toulouse, 31555, France · Agency for Science, Technology and Research, Institute for Infocomm Research, Singapore, 138634

Oct 10, 2026J/K move · Enter open · S save

To Defer or Not to Defer? Online Learning with LLMs and Congested Experts

Vasileios Saketos, William Reveillard, Alexandre Proutiere, Sebastian Dalleiger+1

KTH Royal Institute of Technology, Stockholm, Sweden

Learning to Defer Online Learning

Open paper AskPDF arXiv

Modern AI systems increasingly combine large language models (LLMs) with human experts, using the LLM to handle routine queries while deferring uncertain cases for expert review. When expert capacity is limited, however, deferral decisions must account not only for the reliability of the LLM but also for congestion in the expert queue. We formulate this as an online sequential decision-making problem in which, for each query, the system observes the confidence signal of the LLM's answer and must decide whether to output the answer or defer the query to a capacity-constrained expert. The reliability of the confidence signal is initially unknown and must be learned from expert feedback, which is both selectively observed (only deferred queries are labeled) and delayed by queuing. We propose KL-PBD, a learning algorithm that uses Kullback--Leibler lower confidence bounds on the unknown reliability parameters to guide deferral decisions. For

K

confidence levels and horizon

T

, KL-PBD achieves

O(K\log T)

gap-dependent regret when the accuracy at every confidence level is bounded away from the deferral boundary, and

\widetilde{O}(\sqrt{KT})

worst-case regret. We establish matching lower bounds, showing that these rates are optimal up to logarithmic factors. Finally, we validate our approach on synthetic instances and real-world question-answering datasets, namely GSM8K, NuminaMath, and MedMCQA, using Qwen LLMs.

Abstract

Learning to Defer

Explore similar work

To Defer or Not to Defer? Online Learning with LLMs and Congested Experts

FatigueFusion: Latent Space Fusion for Fatigue-Driven Motion Synthesis

Beyond Augmented-Action Surrogates for Multi-Expert Learning-to-Defer