Organizations: School of Computer Science, Sun Yat-sen University, Guangzhou, China. · the Guangdong Province Key Laboratory of Information Security Technology and the Key Laboratory of Machine Intelligence and Advanced Computing (Sun Yat-sen University), Ministry of Education, China.
Human Motion Prediction (HMP) aims to predict future human poses at different moments according to observed past motion sequences. Previous approaches mainly treated the prediction of different temporal moments as a single prediction task and learned the predictions of varied moments simultaneously, which would encounter a main limitation: the learning of short-term predictions (referring to near-future'' prediction) could be hindered by the predictions of long-term (referring to far-future'' prediction) motions. In this paper, we develop a novel temporal continual learning framework called Continual Prior Compensation (CPC) to progressively train HMP models, in which we divide the prediction task of motions corresponding to varied temporal moments into several subtasks and train the model in a multi-stage manner. To mitigate the prior information forgetting in the progressive training, we further introduce a learnable random variable Prior Compensation Factor (PCF) to explicitly measure the prior knowledge loss. We theoretically show that the PCF can be efficiently learned together with the model parameters by minimizing a reasonable upper bound of the objective function. The proposed CPC is further enhanced to estimate the prior information loss for each subtask and a new framework called Continual Prior Compensation++ (CPC++) with Fine-Grained Prior Compensation Factor (FGPCF) is finally developed. Our CPC and CPC++ frameworks are quite flexible and can be easily integrated with different HMP backbone models and adapted to various datasets and applications. Extensive experiments on three HMP benchmark datasets using multiple SOTA HMP backbones (PGBIG, siMLPe, MotionMixer, and LTD) demonstrate the effectiveness and flexibility of our frameworks.
Figures & tables
Fig. 1 : Preliminary experiment results of three different prediction settings (lower values indicate better performance). (a) represents the experimental settings. (b) shows the short-term prediction results (predicting the 2-nd frame), while (c) illustrates the long-term prediction results (predicting the 25-th frame).
Fig. 2 : Illustration of Prior Loss Estimation Module. InterFASS is the Inter-Task Forgetting Assessment module, and IntraFAGG represents the Intra-Task Forgetting Aggregation module. MP denotes the max-pooling layer.
Dataset
Method
80ms
160ms
320ms
400ms
560ms
1000ms
Human3.6M
PGBIG
10.3
22.7
47.4
58.5
76.9
110.3
PGBIG&CPC
9.4 (-0.9)
21.3 (-1.4)
45.7 (-1.7)
56.8 (-1.7)
75.4 (-1.5)
108.8 (-1.5)
PGBIG&CPC++
9.1 (-1.2)
20.9 (-1.8)
45.5 (-1.9)
56.6 (-1.9)
75.4 (-1.5)
109.2 (-1.1)
CMU-MoCap
PGBIG
7.6
14.3
29.0
36.6
50.9
80.1
PGBIG&CPC
7.5 (-0.1)
14.3 (-0.0)
28.3 (-0.7)
35.4 (-1.2)
48.6 (-2.3)
78.4 (-1.7)
PGBIG&CPC++
7.6 (-0.0)
14.2 (-0.1)
27.9 (-1.1)
34.9 (-1.7)
48.0 (-2.9)
76.8 (-3.3)
TABLE I : Results on Human3.6M, CMU-MoCap and 3DPW using PGBIG as baseline. A lower value means better performance.
Method
Framework
80ms
160ms
320ms
400ms
560ms
1000ms
LTD
baseline
12.7
26.1
52.3
63.5
81.6
114.3
CPC
10.8 (-1.9)
23.0 (-3.1)
48.2 (-4.1)
59.3 (-4.2)
77.5 (-4.1)
111.2 (-3.1)
CPC++
10.2 (-2.5)
22.3 (-3.8)
47.3 (-5.0)
58.5 (-5.0)
77.2 (-4.4)
111.0 (-3.3)
MM
baseline
12.7
26.4
53.4
65.0
83.6
117.6
CPC
11.5 (-1.2)
25.0 (-1.4)
52.0 (-1.4)
63.7 (-1.3)
82.8 (-0.8)
117.1 (-0.5)
CPC++
10.8 (-1.9)
24.3 (-2.1)
51.4 (-2.0)
63.1 (-1.9)
82.4 (-1.2)
117.0 (-0.6)
TABLE II : Results of other baseline models on Human3.6M. A lower value means better performance.
Fig. 3 : Some visualized results from the Human3.6M dataset are presented. We display four action categories: “Directions”, “Takingphoto”, “Purchases”, and “Posing”. The observed frames are shown in black, while the colorful motion sequences represent the prediction results. “GT” denotes the ground truth results and “PGBIG” is the baseline. “+Ours” represents the results of using the proposed CPC framework. As demonstrated, our method produces the best future motion sequences. And we highlight the different parts in PGBIG’s results with red dashed circles.
Fig. 4 : Visualized results of PCF and FGPCF. Each subplot is represented in polar coordinates, where the angular coordinate represents different action categories, and the radial distance represents values. The drawn circular area represents the results of a certain stage.
Fig. 5 : The average error of different tasks at the end of each stage. A lower value means better performance.
Method
InterFASS
IntraFAGG
Avg ↓
w/o both
65.7
with InterFASS
✓
65.0
with IntraFAGG
✓
65.5
with both
✓
✓
64.9
TABLE III : Results of our method with/without InterFASS and IntraFAGG modules.
Fig. 6 : Comparison of different approaches. PGBIG is a baseline model that is trained without multi-stage. “w/o α ” represents training multi-stage process without PCF. “HC” means using a manually designed coefficient. “Ours” is a multi-stage training process with PCF. A lower value means better performance.
number of tasks
1
2
3
5
8
avg error ↓
66.95
66.02
65.00
65.05
65.03
TABLE IV : The average error of different numbers of tasks.
Method
80ms
160ms
320ms
400ms
560ms
1000ms
PGBIG (Baseline)
10.3
22.7
47.4
58.5
76.9
110.3
6-overlap
9.9
21.9
46.5
57.6
76.5
110.4
3-overlap
9.8
21.8
46.5
57.6
76.3
110.3
no-overlap
9.1
20.9
45.5
56.6
75.4
109.2
TABLE V : Comparison results (MPJPE) of our method with different overlaps. X-overlap denotes the X-frame overlap between adjacent segments.
Method
80ms
160ms
320ms
400ms
560ms
1000ms
PGBIG
10.3
22.7
47.4
58.5
76.9
110.3
+multi-head
10.0
22.0
47.0
58.0
77.1
111.1
+single shared-head
9.4
21.3
45.7
56.8
75.4
108.8
TABLE VI : Comparison results (MPJPE) of implementing multi-stage training with different strategies (single head vs. multi-head).
Method
Avg. err.
stage 1
stage 2
stage 3
total
PGBIG
67.0
16.3 h
-
-
16.3 h
PGBIG+Ours
64.9
8.1 h
6.0 h
5.4 h
19.5 h
LTD
70.2
8.4 h
-
-
8.4 h
LTD+Ours
67.3
5.3 h
3.0 h
3.1 h
11.4 h
TABLE VII : Total training time comparison. The training procedure is stopped when the loss variation remains below 1e-5 within 5 epochs.
Fig. 7 : The ratio of samples and the upper bound of maximun gap between UB and target objective when the confidence value is larger than a certain level.
Method
80ms
160ms
320ms
400ms
560ms
1000ms
Transformer
0.235
0.581
0.990
1.143
1.362
1.826
Ours
0.230
0.553
0.921
1.051
1.266
1.729
Gain
0.005
0.028
0.069
0.092
0.096
0.097
TABLE VIII : Experimental results on Transformer-based model POTR with different training strategies.
School of Mechanical Engineering, Yonsei University, Seoul 03722, Korea · Power Generation Lab, Korea Power Research Institute, 105, Munji-Ro, Yuseong-Gu, Daejeon, 34056, South Korea