In advanced manufacturing systems, humans and robots collaborate to conduct the production process. Effective task planning and allocation (TPA) is crucial for achieving high production efficiency, yet it remains challenging in complex and dynamic manufacturing environments. The dynamic nature of humans and robots, particularly the need to consider spatial information (e.g., humans' real-time position and the distance they need to move to complete a task), substantially complicates TPA. To address the above challenges, we decompose production tasks into manageable subtasks. We then implement a real-time hierarchical human-robot TPA algorithm, including a high-level agent for task planning and a low-level agent for task allocation. For the high-level agent, we propose an efficient buffer-based deep Q-learning method (EBQ), which reduces training time and enhances performance in production problems with long-term and sparse reward challenges. For the low-level agent, a path planning-based spatially aware method (SAP) is designed to allocate tasks to the appropriate human-robot resources, thereby achieving the corresponding sequential subtasks. We conducted experiments on a complex real-time production process in a 3D simulator. The results demonstrate that our proposed EBQ&SAP method effectively addresses human-robot TPA problems in complex and dynamic production processes.
Figures & tables
Figure 1: Illustration of the real-time production process flow and state information.
Figure 2: Our proposed hierarchical human-robot TPA algorithm with the high-level agent for real-time task planning, and a low-level agent for task allocation. The details of the high-level agent and low-level agent are expressed in Sec. 3.3 and Sec. 3.5 .
Notation
Description
T
Task set
O
Subtask set
E
Entity set of production participants
st
State of environment at time step t
R
Reward function
Table 1: Notations table.
Figure 3: Efficient buffer mechanism: episode-end processing. We modify the rewards of each episode’s transitions and duplicate the positive data before transferring it to the training buffer, increasing and prioritizing positive data, and shortening training time.
Figure 4: Network architecture for high-level agent, including feature embedding, transformer architecture, dueling network and Noisy Net.
Figure 5: SAP: Spatial aware by path planning for low-level agent.
Figure 6: SAP: offline node graph generation and real-time usage for task allocation.
Figure 7: Production workspace including humans, robots, machines, and materials.
Figure 8: Details of the task dependency graph, machine tasks, and human-robot tasks including subtasks.
Table 2: Comparison for efficient buffer mechanism proposed in Sec. 3.3.3 . Our proposed method is indicated by a “ † ”.
Table 3: Comparison for high-level algorithms. Our proposed method is indicated by a “ † ”.
Explanation
NoSpatial
Randomly allocates tasks to entities
LongestPath
allocates tasks to the farthest entities
EBQ-G †
allocates tasks to the closest entities
Table 4: Comparisons for low-level algorithms. See Sec. 3.5 for SAP algorithm details.
Training time
D3QN
≥ 40 hours
NoModify
10 ∼ 11 hours
EBQ-G †
10 ∼ 11 hours
Table 5: Training time comparison. See Sec. 3.3.3 for efficient buffer mechanism.
Figure 9: Evaluation curves during training intervals for efficient buffer mechanism.
Algorithm
Metric: Makespan
H1,R1
H1,R2
H1,R3
H2,R1
H2,R2
H2,R3
H3, R1
H3,R2
H3,R3
Mean
D3QN
1013.93
983.03
961.05
837.25
855.96
746.61
871.47
865.49
737.82
874.73
NoModify
990.35
1018.21
1004.94
758.72
789.63
735.42
825.92
782.96
702.07
845.36
EDQN1
1032.35
1069.95
1019.08
755.75
751.14
752.90
750.01
692.63
690.46
834.92
Table 6: Makespan performance in the testing phase. Note that a ” † ” indicates the performance of our proposed algorithm. ”Hn” denotes the number of humans is set to n, while ”Rn” indicates n robots during testing.
Figure 10: Training curves of high-level algorithms, all use SAP as the low-level algorithm.
Figure 11: Evaluation of high-level algorithms during training, all use SAP as the low-level algorithm.
Figure 12: Visualized results of Table 6 . Makespan under varying numbers of humans and robots.
Figure 13: Visualization of Table 7 . Makespan in unseen product quantities.
Table 7: The table presents zero-shot results, evaluating algorithms on unseen product quantities post-training. ”NumN” means producing N products. The performance of our proposed model is marked with a “ † ”.
Figure 14: The improvement degree of EBQ- ∗ Vs. D3QN and others, D3QN is the baseline.
Figure 15: Showcase of path planning for offline node graph generation. For both humans and robots, we present two cases each, using diverse start nodes from task-related working area nodes. Path results are provided from each start node to all other task-related working area nodes.
Figure 16: Evaluation curves during training intervals for EBQ-N combined with various low-level algorithms.
Table 8: Makespan and distance performance in the testing phase for EBQ-N combined with various low-level algorithms. Note that a ” † ” indicates the performance of our proposed algorithm. ”Hn” denotes the number of humans is set to n, while ”Rn” indicates n robots during testing.
Figure 17: Case study: Gantt chart and visualization results. Tasks’ details in Fig. 8 , and environment’s details in Fig. 7 .