Organizations: School of Computer Science, Sun Yat-sen University, Guangzhou, China. · the Guangdong Province Key Laboratory of Information Security Technology and the Key Laboratory of Machine Intelligence and Advanced Computing (Sun Yat-sen University), Ministry of Education, China.
Human Motion Prediction (HMP) aims to predict future human poses at different moments according to observed past motion sequences. Previous approaches mainly treated the prediction of different temporal moments as a single prediction task and learned the predictions of varied moments simultaneously, which would encounter a main limitation: the learning of short-term predictions (referring to near-future'' prediction) could be hindered by the predictions of long-term (referring to far-future'' prediction) motions. In this paper, we develop a novel temporal continual learning framework called Continual Prior Compensation (CPC) to progressively train HMP models, in which we divide the prediction task of motions corresponding to varied temporal moments into several subtasks and train the model in a multi-stage manner. To mitigate the prior information forgetting in the progressive training, we further introduce a learnable random variable Prior Compensation Factor (PCF) to explicitly measure the prior knowledge loss. We theoretically show that the PCF can be efficiently learned together with the model parameters by minimizing a reasonable upper bound of the objective function. The proposed CPC is further enhanced to estimate the prior information loss for each subtask and a new framework called Continual Prior Compensation++ (CPC++) with Fine-Grained Prior Compensation Factor (FGPCF) is finally developed. Our CPC and CPC++ frameworks are quite flexible and can be easily integrated with different HMP backbone models and adapted to various datasets and applications. Extensive experiments on three HMP benchmark datasets using multiple SOTA HMP backbones (PGBIG, siMLPe, MotionMixer, and LTD) demonstrate the effectiveness and flexibility of our frameworks.
Figures & tables
Fig. 1 : Preliminary experiment results of three different prediction settings (lower values indicate better performance). (a) represents the experimental settings. (b) shows the short-term prediction results (predicting the 2-nd frame), while (c) illustrates the long-term prediction results (predicting the 25-th frame).
Fig. 2 : Illustration of Prior Loss Estimation Module. InterFASS is the Inter-Task Forgetting Assessment module, and IntraFAGG represents the Intra-Task Forgetting Aggregation module. MP denotes the max-pooling layer.
Dataset
Method
80ms
160ms
320ms
400ms
560ms
1000ms
Human3.6M
PGBIG
10.3
22.7
47.4
58.5
76.9
110.3
PGBIG&CPC
9.4 (-0.9)
21.3 (-1.4)
45.7 (-1.7)
56.8 (-1.7)
75.4 (-1.5)
108.8 (-1.5)
PGBIG&CPC++
9.1 (-1.2)
20.9 (-1.8)
45.5 (-1.9)
56.6 (-1.9)
75.4 (-1.5)
109.2 (-1.1)
CMU-MoCap
PGBIG
7.6
14.3
29.0
36.6
50.9
80.1
PGBIG&CPC
7.5 (-0.1)
14.3 (-0.0)
28.3 (-0.7)
35.4 (-1.2)
48.6 (-2.3)
78.4 (-1.7)
PGBIG&CPC++
7.6 (-0.0)
14.2 (-0.1)
27.9 (-1.1)
34.9 (-1.7)
48.0 (-2.9)
76.8 (-3.3)
TABLE I : Results on Human3.6M, CMU-MoCap and 3DPW using PGBIG as baseline. A lower value means better performance.
Method
Framework
80ms
160ms
320ms
400ms
560ms
1000ms
LTD
baseline
12.7
26.1
52.3
63.5
81.6
114.3
CPC
10.8 (-1.9)
23.0 (-3.1)
48.2 (-4.1)
59.3 (-4.2)
77.5 (-4.1)
111.2 (-3.1)
CPC++
10.2 (-2.5)
22.3 (-3.8)
47.3 (-5.0)
58.5 (-5.0)
77.2 (-4.4)
111.0 (-3.3)
MM
baseline
12.7
26.4
53.4
65.0
83.6
117.6
CPC
11.5 (-1.2)
25.0 (-1.4)
52.0 (-1.4)
63.7 (-1.3)
82.8 (-0.8)
117.1 (-0.5)
CPC++
10.8 (-1.9)
24.3 (-2.1)
51.4 (-2.0)
63.1 (-1.9)
82.4 (-1.2)
117.0 (-0.6)
TABLE II : Results of other baseline models on Human3.6M. A lower value means better performance.
Fig. 3 : Some visualized results from the Human3.6M dataset are presented. We display four action categories: “Directions”, “Takingphoto”, “Purchases”, and “Posing”. The observed frames are shown in black, while the colorful motion sequences represent the prediction results. “GT” denotes the ground truth results and “PGBIG” is the baseline. “+Ours” represents the results of using the proposed CPC framework. As demonstrated, our method produces the best future motion sequences. And we highlight the different parts in PGBIG’s results with red dashed circles.
Fig. 4 : Visualized results of PCF and FGPCF. Each subplot is represented in polar coordinates, where the angular coordinate represents different action categories, and the radial distance represents values. The drawn circular area represents the results of a certain stage.
Fig. 5 : The average error of different tasks at the end of each stage. A lower value means better performance.
Method
InterFASS
IntraFAGG
Avg ↓
w/o both
65.7
with InterFASS
✓
65.0
with IntraFAGG
✓
65.5
with both
✓
✓
64.9
TABLE III : Results of our method with/without InterFASS and IntraFAGG modules.
Fig. 6 : Comparison of different approaches. PGBIG is a baseline model that is trained without multi-stage. “w/o α ” represents training multi-stage process without PCF. “HC” means using a manually designed coefficient. “Ours” is a multi-stage training process with PCF. A lower value means better performance.
number of tasks
1
2
3
5
8
avg error ↓
66.95
66.02
65.00
65.05
65.03
TABLE IV : The average error of different numbers of tasks.
Method
80ms
160ms
320ms
400ms
560ms
1000ms
PGBIG (Baseline)
10.3
22.7
47.4
58.5
76.9
110.3
6-overlap
9.9
21.9
46.5
57.6
76.5
110.4
3-overlap
9.8
21.8
46.5
57.6
76.3
110.3
no-overlap
9.1
20.9
45.5
56.6
75.4
109.2
TABLE V : Comparison results (MPJPE) of our method with different overlaps. X-overlap denotes the X-frame overlap between adjacent segments.
Method
80ms
160ms
320ms
400ms
560ms
1000ms
PGBIG
10.3
22.7
47.4
58.5
76.9
110.3
+multi-head
10.0
22.0
47.0
58.0
77.1
111.1
+single shared-head
9.4
21.3
45.7
56.8
75.4
108.8
TABLE VI : Comparison results (MPJPE) of implementing multi-stage training with different strategies (single head vs. multi-head).
Method
Avg. err.
stage 1
stage 2
stage 3
total
PGBIG
67.0
16.3 h
-
-
16.3 h
PGBIG+Ours
64.9
8.1 h
6.0 h
5.4 h
19.5 h
LTD
70.2
8.4 h
-
-
8.4 h
LTD+Ours
67.3
5.3 h
3.0 h
3.1 h
11.4 h
TABLE VII : Total training time comparison. The training procedure is stopped when the loss variation remains below 1e-5 within 5 epochs.
Fig. 7 : The ratio of samples and the upper bound of maximun gap between UB and target objective when the confidence value is larger than a certain level.
Method
80ms
160ms
320ms
400ms
560ms
1000ms
Transformer
0.235
0.581
0.990
1.143
1.362
1.826
Ours
0.230
0.553
0.921
1.051
1.266
1.729
Gain
0.005
0.028
0.069
0.092
0.096
0.097
TABLE VIII : Experimental results on Transformer-based model POTR with different training strategies.
Human motion prediction combines the tasks of trajectory forecasting and human pose prediction. For each of the two tasks, specialized models have been developed. Combining these models for holistic human motion prediction is non-trivial, and recent methods have struggled to compete on established benchmarks for individual tasks. To address this, we propose a simple yet effective transformer-based model for human motion prediction. The model employs a stack of self-attention modules to effectively capture both spatial dependencies within a pose and temporal relationships across a motion sequence. This simple, streamlined, end-to-end model is sufficiently versatile to handle pose-only, trajectory-only, and combined prediction tasks without task-specific modifications. We demonstrate that this approach achieves state-of-the-art results across all tasks through extensive experiments on a wide range of benchmark datasets, including Human3.6M, AMASS, ETH-UCY, and 3DPW.
Human motion prediction is a crucial capability for advanced robotic systems that interact with humans. In facilities with dynamic human-robot collaboration settings, robots must anticipate human movements to ensure safety, prevent collisions, and optimize cooperative tasks. Traditionally, motion forecasting is treated as a sequential modeling problem using historical pose data, but achieving long-term accuracy and physical realism remains challenging. We present Adversarial Motion Transformer (AdvMT), a novel approach that integrates a Transformer-based motion encoder with a temporal continuity discriminator to address these challenges. The Transformer captures rich spatio-temporal dependencies across human joints, while adversarial training with a continuity discriminator enforces smooth, natural motion trajectories that adhere to biomechanical constraints. Our training scheme includes a bone-length consistency term and adversarial loss to reduce common artifacts like pose freezing or unnatural transitions. In experiments on the Human3.6M motion dataset, AdvMT achieves state-of-the-art long-horizon prediction accuracy while also delivering robust short-term predictions. These improvements strengthen the prediction foundation for physical AI in manufacturing and human-robot collaboration, where anticipating human motion is a prerequisite for safe and efficient robot coordination.
Sarmad Idrees, Seokman Sohn, Jongeun Choi
School of Mechanical Engineering, Yonsei University, Seoul 03722, Korea · Power Generation Lab, Korea Power Research Institute, 105, Munji-Ro, Yuseong-Gu, Daejeon, 34056, South Korea
Human motion describes the three-dimensional full-body movement of a person. Anticipating such motion holds significant relevance across a wide range of application domains such as human-robot interaction, autonomous driving, animation, and healthcare. In recent research, spatial and temporal dependencies are modeled by bidirectional attention mechanisms. These typically anticipate human motion in an autoregressive manner which could cause an accumulation of errors over time. As a consequence, they solely focus on local pose forecasting. To address these limitations, we propose a non-autoregressive transformer based on spatio-temporal attention, and train it not only for local pose anticipation, but also for global motion prediction in space. Furthermore, to enhance its applicability in real-world scenarios, our model is also trained to recover missing joints due to occlusions, and is capable of processing varying lengths of history observations. Our code is publicly available at https://github.com/Q-Y-Yang/Prediction-of-Local-and-Global-Human-Motion.
Qiaoyue Yang, Sven Heutger, Christopher Niemann +3