Adaptive Interaction Network for Human Motion Prediction During Human-Robot Collaboration
Authors: Mengyuan Liu, Yangting Lin, Qiongjie Cui
Organizations: State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School, Shenzhen, Guangdong 518055, China · Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology, Peking University, Shenzhen Graduate School, Shenzhen, Guangdong 518055, China · National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China
Human motion prediction during human-robot collaboration is critical for achieving safe and efficient interactions in shared environments. Unlike previous studies that have primarily focused on humans and objects, we focus on the robot-aware human motion prediction task, which explicitly models the influence of robots on human motion. This task presents new challenges, such as capturing human-robot heterogeneity and modeling complex spatial interactions. To address these issues, we develop an adaptive interaction network (AINet) model that consists of two branches: a primary branch that predicts future human motion and an auxiliary branch that estimates robot trajectories. The two branches are jointly optimized and coupled via a Local-Global Spatial Interaction (LGSI) Module, which effectively captures fine-grained and global contextual dependencies between human and robot motion sequences. Then, we introduce an Adaptive Weighted Aggregation (AWA) Module to dynamically fuse motion features using context-dependent weights, thereby increasing adaptability across diverse scenarios. Furthermore, we adopt a coarse-to-fine prediction strategy, in which a coarse pseudo-trajectory is first predicted, followed by the refinement of detailed human poses based on that trajectory. Extensive experiments on three challenging datasets reveal that our proposed method achieves superior performance, validating its effectiveness in human-robot collaboration scenarios. Our code is available at: https://github.com/LyTingHub/AINet.
Figures & tables
Fig. 1: Comparison of HMP paradigms. The solid and dashed human skeletons represent past observations and future motions, respectively: (a) single-person motion prediction [ 15 ] , (b) multi-person motion prediction [ 16 ] , (c) HMP during human-object interactions [ 17 ] , (d) scene-aware HMP [ 18 ] , and (e) our proposed robot-aware HMP. The proposed setting introduces unique challenges: (1) inherent skeleton disparity between humans and robots, (2) interaction complexity due to the robot’s structure and mobility, and (3) dynamic context variations caused by the robot’s autonomous mobility.
Fig. 2: An overview of our proposed AINet. The input motion data is first transformed into the frequency domain using the Motion Encoder. Human and robot features then interact through the LGSI module, which generates local and global interaction features via a three-step process: Align, Interact, and Reconstruct. The AWA module dynamically aggregates these interaction features using context-based weights. Next, the Traj-Pose Module extracts global trajectories and refines local poses. Finally, the Motion Decoder reconstructs the motion sequence from its frequency components. For training, we utilize the combined loss function Lhuman+Lrobot+λLreg , where Lhuman and Lrobot supervise the human and robot branches, respectively, while Lreg regularizes the weights generated by AWA.
Fig. 3: Joint-wise motion patterns on HARPER, CHICO, and CoMaD. Red circles indicate the average motion magnitude at each joint. (a) HARPER: hip joints exhibit large displacements; (b–c) CHICO and CoMaD: motion is primarily concentrated in the upper limbs, while hips remain nearly static. These patterns reveal the limitation of using a fixed-joint trajectory as a proxy for global motion.
Fig. 4: Architecture of the proposed Traj-Pose Module, composed of ST Encoders, a Traj Module for pseudo-trajectory generation, and a Pose Module for refinement.
Dataset
CoMaD
CHICO
HARPER
Scenario
HRC
HRC
HRI
Duration(s)
3,578
6,773
2,451
Robot
Arm
Arm
Quadruped
action #
5
7
15
subject #
9
20
17
TABLE I: Comparisons of the Three Datasets Used in the Experiments.
Fig. 5: Illustrative examples from the CoMaD dataset showcase its varied tasks and scenes for HRC in the kitchen. To visualize motion, we stack the corresponding RGB images.
Fig. 6: Illustrative examples from the CHICO dataset show HRC scenarios in real-world factory assembly lines.
Fig. 7: Illustrative examples from the HARPER dataset showcase its diverse poses and rich human-robot interactions.
Method
Input
100
200
300
400
500
600
700
800
900
1000
Avg.
SeS-GCN [ 20 ]
H
21.79
28.00
33.68
39.15
44.06
48.70
53.05
57.06
60.78
64.10
45.04
(ECCV’22)
HR
17.34
23.18
29.08
35.07
40.84
46.15
50.99
55.55
59.75
63.48
42.14
siMLPe [ 24 ]
H
6.01
11.33
16.63
21.94
27.09
31.91
36.43
40.58
44.34
47.93
28.42
(WACV’23)
HR
5.91
11.64
17.51
23.29
28.63
33.50
38.09
42.29
46.18
49.91
29.69
PoseTron [ 38 ]
HR
10.71
18.87
26.95
32.66
38.69
43.08
47.48
50.91
54.50
57.34
38.12
GCNext [ 25 ]
H
5.89
11.33
16.75
22.10
27.18
31.81
36.04
39.99
43.62
47.00
28.17
TABLE II: Quantitative comparisons on CoMaD. “H” denotes human motion input, and “HR” denotes human and robot motion input. Our approach achieves SOTA performance at all time steps.
Method
input
80
160
320
400
560
720
880
1000
Avg.
SeS-GCN [ 20 ]
H
16.72
28.11
46.75
53.58
64.30
72.71
80.40
86.04
56.08
(ECCV’22)
HR
18.03
30.34
50.65
57.91
68.92
77.54
85.51
91.30
60.03
siMLPe [ 24 ]
H
13.51
24.04
40.95
47.58
58.62
68.40
77.10
83.03
51.65
(WACV’23)
HR
13.23
23.41
40.11
46.78
57.33
66.86
75.05
80.18
50.37
PoseTron [ 38 ]
HR
19.31
33.35
54.94
63.06
75.24
84.18
91.76
96.77
64.83
GCNext [ 25 ]
H
13.94
24.84
42.61
49.16
59.95
70.00
78.54
84.16
52.90
TABLE III: Quantitative comparisons on CHICO. Our approach achieves SOTA performance at all time steps.
Method
Input
100
200
300
400
500
600
700
800
900
1000
Avg.
SeS-GCN [ 20 ]
H
42.48
49.92
59.80
68.30
79.79
93.41
105.13
119.13
133.21
148.80
90.00
(ECCV’22)
HR
48.10
55.76
64.10
76.66
86.56
98.59
113.31
125.96
139.10
154.16
96.23
siMLPe [ 24 ]
H
5.81
13.47
22.31
32.38
43.17
54.46
66.49
79.23
92.51
106.16
51.60
(WACV’23)
HR
5.93
13.61
22.65
32.85
43.58
54.81
66.78
79.32
92.27
105.49
51.73
PoseTron [ 38 ]
HR
19.87
36.91
53.12
69.42
84.82
100.50
115.96
131.06
145.87
160.16
91.77
GCNext [ 25 ]
H
5.70
13.77
23.28
33.53
44.04
55.08
66.95
79.45
92.40
105.65
51.98
TABLE IV: Quantitative comparisons on HARPER. Ours-HR demonstrates a marginal advantage over the baselines.
Action
Walk + Crash Sideways
Walk + Crash Backwards
Walk + Avoid Right
milliseconds
0.1s
0.4s
0.7s
1s
Avg.
0.1s
0.4s
0.7s
1s
Avg.
0.1s
0.4s
0.7s
1s
Avg.
SeS-GCN-H [ 20 ]
20.14
73.76
139.67
216.66
98.93
92.74
132.34
182.45
243.55
160.56
83.76
95.76
119.93
154.39
111.45
SeS-GCN-HR [ 20 ]
56.98
105.36
167.75
239.79
139.78
53.63
101.88
162.11
231.36
134.81
46.31
67.08
98.34
133.86
84.59
siMLPe-H [ 24 ]
9.87
58.68
126.81
212.97
98.87
11.49
66.49
144.94
240.29
112.53
5.18
29.08
59.62
94.83
46.15
siMLPe-HR [ 24 ]
9.76
55.19
116.77
191.98
90.95
10.93
61.65
130.57
204.26
99.95
4.97
27.40
57.83
96.07
45.20
PoseTron-HR [ 38 ]
31.34
112.05
184.62
263.74
148.09
34.68
130.66
220.78
307.96
174.37
23.17
78.89
134.38
188.42
106.23
TABLE V: Action-wise results on HARPER. Our method significantly outperforms the baselines across various actions.
Model
FLOPs (M) ↓
Para. (M) ↓
MPJPE (mm) ↓
Inference FPS ↑
RTX 3060
Jetson Nano
siMLPe-HR
24.90
0.21
106.16
278.89
37.08
GCNext-HR
24.90
0.21
105.96
56.23
8.52
Ours-HR
25.58
0.41
100.10
148.59
30.69
TABLE VI: Efficiency comparison across hardware platforms. FLOPs/Params are theoretical; FPS is measured on an RTX 3060 and Jetson Nano.
Fig. 8: Qualitative comparison on the tabletop collaboration setting of CoMaD [ 19 ] between our method and prior approaches, including siMLPe [ 24 ] and GCNext [ 25 ] . Robot motions are shown as ground truth to illustrate the human-robot interaction context. MPJPE values (lower is better) are reported above each prediction, and dashed circles highlight notable differences.
Fig. 9: Qualitative comparison on the HRC scene of CHICO [ 20 ] between our method and prior approaches, including siMLPe [ 24 ] and GCNext [ 25 ] .
Fig. 10: Qualitative comparison on the “Walk+Avoid right” action of HARPER [ 21 ] between our method and prior approaches, including siMLPe [ 24 ] and GCNext [ 25 ] .
Method
MPJPE(mm)
100
1000
Avg.
Avg. △
ours-HR
4.94
100.11
47.71
-
w/o LSI
4.97
101.06
48.12
0.40
w/o GSI
5.14
101.08
48.69
0.97
w/o LGSI
5.01
101.82
48.15
0.44
w/o AWA
5.11
102.62
49.28
1.57
TABLE VII: Ablation studies on Core Components of Model Structures. Experiments are Conducted With the HARPER Dataset.
Method
MPJPE(mm)
100
400
700
1000
Avg.
Ours-HR
4.94
29.25
61.48
100.11
47.71
w/o LN
5.08
29.99
63.27
103.58
49.16
w/o truncation
4.99
29.68
61.83
100.13
47.99
w/o sumNorm.
5.37
30.20
63.14
102.53
49.11
w/o context
5.17
30.15
62.54
101.53
48.63
TABLE VIII: Ablation study on the implementation of the AWA. Comparison of the performance of the full model (Ours-HR) with variations in which specific AWA components (LN, truncation, and sumNorm) and context are excluded. “w/o context” refers to using learnable parameters instead.
Method
MPJPE (mm)
100
400
700
1000
Avg.
hip-based
4.95
29.50
61.96
101.01
48.11
mean-joint-based
5.02
29.44
61.55
100.35
47.85
ours
4.94
29.25
61.48
100.11
47.71
TABLE IX: Ablation on the Traj-Pose Module. Our Learned Pseudo-trajectory Outperforms Handcrafted Baselines Across all Time Steps on the HARPER Dataset.
Dept. of Information Engineering and Computer Science (DISI), University of Trento, Trento, Italy · Department of Industrial Engineering (DII), University of Trento, Trento, Italy
Department of Information Engineering and Computer Science, Universit`a di Trento, Trento, Italy. · Interdepartmental Robotics Labs (IDRA), University of Trento, Trento, Italy · Department of Industrial Engineering, Universit`a di Trento, Trento, Italy.