Safe reinforcement learning with online filtering for fatigue-predictive human-robot task planning and allocation in production

Jintao Xue; Xiao Li; Nianmin Zhang

cs.AIApr 14, 2026

Safe reinforcement learning with online filtering for fatigue-predictive human-robot task planning and allocation in production

Authors: Jintao Xue, Xiao Li, Nianmin Zhang

Organizations: CI3 Lab, Department of Civil Engineering, The University of Hong Kong, Hong Kong SAR

Abstract

Human-robot collaborative manufacturing, a core aspect of Industry 5.0, emphasizes ergonomics to enhance worker well-being. This paper addresses the dynamic human-robot task planning and allocation (HRTPA) problem, which involves determining when to perform tasks and who should execute them to maximize efficiency while ensuring workers' physical fatigue remains within safe limits. The inclusion of fatigue constraints, combined with production dynamics, significantly increases the complexity of the HRTPA problem. Traditional fatigue-recovery models in HRTPA often rely on static, predefined hyperparameters. However, in practice, human fatigue sensitivity varies daily due to factors such as changed work conditions and insufficient sleep. To better capture this uncertainty, we treat fatigue-related parameters as inaccurate and estimate them online based on observed fatigue progression during production. To address these challenges, we propose PF-CD3Q, a safe reinforcement learning (safe RL) approach that integrates the particle filter with constrained dueling double deep Q-learning for real-time fatigue-predictive HRTPA. Specifically, we first develop PF-based estimators to track human fatigue and update fatigue model parameters in real-time. These estimators are then integrated into CD3Q by making task-level fatigue predictions during decision-making and excluding tasks that exceed fatigue limits, thereby constraining the action space and formulating the problem as a constrained Markov decision process (CMDP).

Figures & tables

Figure 1: Illustration of the real-time production process flow and state information.

Figure 2: Illustration of our method PF-CD3Q: fatigue-predictive HRTPA algorithm with CD3Q and online filtering.

Figure 3: Network architecture of our method PF-CD3Q.

Figure 4: Experimental environment, entities, human-robot-machine task descriptions, and task dependency graph

1. (Static) $\lambda$ –Fatigue of subtask	Value
Put flange into cage	0.12
Put bend duct into cage	0.18
Put flange on side storage	0.12
Put bend duct on side storage	0.18
Loading flange on welding station 1	0.36
Loading flange on welding station 2	0.36

Table 1: Static and dynamic values of human fatigue model parameters, as detailed in Sec. 3.3 and Sec. 4.1 for parameter definitions.

Panel of Table 2 — Table 2: Comparison summary of RL or safe RL algorithms. Our proposed method is indicated by a “ † ”.

Figure 5: Case study: Human episodic fatigue variation, fatigue model parameter estimation, and task-level fatigue prediction.

Figure 6: Comparison of prediction loss and parameter accuracy for PF, KF, and EKF.

Figure 7: Measurement noise analysis: filter accuracy and PF-CD3Q performance on test stage.

Figure 8: Algorithms performance in training stage.

Figure 9: Algorithm performance in the test stage, evaluated through makespan and overwork metrics.

Algorithm	H1,R1	H1,R2	H1,R3	H2,R1	H2,R2	H2,R3	H3,R1	H3,R2	H3,R3	Mean
Makespan
DQN	1764.60	1490.50	1587.98	1240.20	1172.34	1163.80	1180.70	1121.18	1126.90	1316.47
PPO	1615.76	1734.76	1738.04	1217.74	1215.42	1286.34	1149.90	1143.76	1141.80	1360.39
D3QN	1635.94	1506.26	1511.24	1193.52	1168.44	1174.56	1084.46	1129.10	1133.18	1281.86
PPO-Lag	1656.24	1451.18	1469.60	1401.06	1174.70	1170.62	1393.86	1118.38	1125.54	1329.02
PF-DQN	1767.10	1644.94	1641.90	1161.74	1173.22	1194.26	1174.22	1113.86	1124.62	1332.87

Table 3: Algorithm performance in the test stage, measured by makespan and overwork metrics. Note that † denotes the performance of our proposed algorithm. ’Hn’ indicates n humans, and ’Rn’ signifies n robots during testing.

Figure 10: Algorithm performance in various human-robot settings.

Figure 11: Sensitivity analysis of fatigue constraints: comparison of PF-CD3Q and PPO-Lag.

Figure 12: Multi-metric algorithm ability: radar chart visualization.

Figure 13: PF-CD3Q vs D3QN: Case study of real-time HRTPA in the production process.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Settings	DQN PF-DQN	D3QN PF-CD3Q	PPO PF-PPO	PPO-Lag PF-PPO-Lag
Network capacity	8.9M	9.4M	12.8M	16.1M
Replay setting	buffer size = 5e5	buffer size = 5e5	buffer size = 1e4	buffer size = 1e4
Optimizer	Adam lr $1\!\times\!10^{-4}$	Adam lr $1\!\times\!10^{-4}$	Adam lr $3\!\times\!10^{-4}$	Adam lr $3\!\times\!10^{-4}$
Warmup steps	5e4	5e4	1e3	1e3
Batch size	512	512	512	512
Q-func update	per 400 steps	per 400 steps	nan	nan

Appendix

Table 4: Training hyper-parameters across each algorithm family.

Figure 14: Filter latency and algorithm performance in the test stage, varying number particles.

Figure 15: Ablation Study: training stage metrics.

Figure 16: Ablation Study: test stage metrics.

Figure 17: The training curves of CPO exhibit clear divergence trends.

Explore similar work

Estimating Human Muscular Fatigue in Dynamic Collaborative Robotic Tasks with Learning-Based Models
Feb 17, 2026Feras Kiki, Pouya P. Niaz, Alireza Madani +1Human-Robot Collaboration
F. Kiki, P. P. Niaz, A. Madani, and C. Basdogan (corresponding author) are with the Robotics and Mechatronics Laboratory (RML) and the KUIS AI Center, Koc University, Sariyer, Istanbul 34450, Turkey.
A hierarchical spatial-aware algorithm with efficient reinforcement learning for human-robot task planning and allocation in production
Apr 14, 2026Jintao Xue, Xiao Li, Nianmin ZhangRobot Task Planning Human-Robot Collaboration
Department of Civil Engineering, The University of Hong Kong, Hong Kong SAR
Real-Time Human-Adaptive Task Allocation for Multi-Human Multi-Robot Supervision
Oct 1, 2026Seabin Lee, Sujeong Park, Nayoung Kim +3Human-Robot Collaboration Multi-Robot Task Allocation
Sogang University, Seoul, Korea. 1Dept. of Electronic Engineering. · Sogang University, Seoul, Korea. 2Dept. of Artificial Intelligence.

Feb 17, 2026J/K move · Enter open · S save

Estimating Human Muscular Fatigue in Dynamic Collaborative Robotic Tasks with Learning-Based Models

Feras Kiki, Pouya P. Niaz, Alireza Madani, Cagatay Basdogan

F. Kiki, P. P. Niaz, A. Madani, and C. Basdogan (corresponding author) are with the Robotics and Mechatronics Laboratory (RML) and the KUIS AI Center, Koc University, Sariyer, Istanbul 34450, Turkey.

Human-Robot Collaboration

Open paper AskPDF arXiv

Assessing human muscle fatigue is critical for optimizing performance and safety in physical human-robot interaction(pHRI). This work presents a data-driven framework to estimate fatigue in dynamic, cyclic pHRI using arm-mounted surface electromyography(sEMG). Subject-specific machine-learning regression models(Random Forest, XGBoost, and Linear Regression predict the fraction of cycles to fatigue(FCF) from three frequency-domain and one time-domain EMG features, and are benchmarked against a convolutional neural network(CNN) that ingests spectrograms of filtered EMG. Framing fatigue estimation as regression (rather than classification) captures continuous progression toward fatigue, supporting earlier detection, timely intervention, and adaptive robot control. In experiments with ten participants, a collaborative robot under admittance control guided repetitive lateral (left-right) end-effector motions until muscular fatigue. Average FCF RMSE across participants was 20.8+/-4.3% for the CNN, 23.3+/-3.8% for Random Forest, 24.8+/-4.5% for XGBoost, and 26.9+/-6.1% for Linear Regression. To probe cross-task generalization, one participant additionally performed unseen vertical (up-down) and circular repetitions; models trained only on lateral data were tested directly and largely retained accuracy, indicating robustness to changes in movement direction, arm kinematics, and muscle recruitment, while Linear Regression deteriorated. Overall, the study shows that both feature-based ML and spectrogram-based DL can estimate remaining work capacity during repetitive pHRI, with the CNN delivering the lowest error and the tree-based models close behind. The reported transfer to new motion patterns suggests potential for practical fatigue monitoring without retraining for every task, improving operator protection and enabling fatigue-aware shared autonomy, for safer fatigue-adaptive pHRI control.

Abstract

Explore similar work

Estimating Human Muscular Fatigue in Dynamic Collaborative Robotic Tasks with Learning-Based Models

A hierarchical spatial-aware algorithm with efficient reinforcement learning for human-robot task planning and allocation in production

Real-Time Human-Adaptive Task Allocation for Multi-Human Multi-Robot Supervision