Human trajectory prediction plays a crucial role in applications such as autonomous navigation and video surveillance. While recent works have explored the integration of human skeleton sequences to complement trajectory information, skeleton data in real-world environments often suffer from missing joints caused by occlusions. These disturbances significantly degrade prediction accuracy, indicating the need for more robust skeleton representations. We propose a robust trajectory prediction method that incorporates a self-supervised skeleton representation model pretrained with masked autoencoding. Experimental results in occlusion-prone scenarios show that our method improves robustness to missing skeletal data without sacrificing prediction accuracy, and consistently outperforms baseline models in clean-to-moderate missingness regimes.
Figures & tables
Figure 1: Impact of occlusion-induced missing skeletons on trajectory prediction accuracy. The top row shows a clean (non-occluded) skeleton observation, while the bottom row shows an occluded observation with missing joints. Using an existing human trajectory prediction (HTP) model, prediction accuracy is high under clean inputs (FDE = 0.17 m) but degrades substantially under occluded inputs (FDE = 0.87 m), where FDE denotes the final displacement error.
Figure 2: Overview of the self-supervised skeleton learning framework. Random masks are applied to the input skeleton sequence, and a spatio-temporal encoder extracts latent representations from the remaining visible joints. A decoder then reconstructs the original skeleton sequence. Through this process, the model learns structural representations invariant to missing data and noise.
Figure 3: Illustration of three mask patterns used for self-supervised skeleton pretraining: (a) Temporally Consistent masks the same joints across all frames; (b) Random masks joints independently at each frame; (c) Body-Part masks multiple joints in the same body part together.
Figure 4: Overall architecture of the proposed framework. (a) Self-supervised pretraining : a skeleton encoder is pretrained by reconstructing masked joints from partially observed skeleton sequences. (b) Individual feature extraction : observed trajectories are embedded, and skeleton sequences are encoded by the pretrained (frozen) encoder. Positional encoding (PE) is added to each stream, the resulting tokens are concatenated, and a Cross-Modality Transformer produces an agent-wise representation. (c) Interaction modeling : the agent-wise representations are processed by a Social Transformer to model inter-agent interactions.
Train
Test
MPJPE
Cos Sim
Temporally-Consistent
Temporally-Consistent
0.058
0.841
Random
0.201
0.833
Body-Part
0.215
0.821
Random
Temporally-Consistent
0.062
0.912
Random
0.036
0.935
Body-Part
0.047
0.928
Table 1: Cross-evaluation of three masking strategies (Fig. 3 ). Each model is trained with one strategy and evaluated under all three test-time masks (Temporally-Consistent, Random, Body-Part) with a fixed test mask ratio of 0.5. Bold indicates the best result within each training strategy (min MPJPE / max cosine similarity).
Figure 5: Reconstruction MPJPE under varying training and test-time mask ratios. (a) MPJPE versus the training mask ratio rtrain , with one curve for each test-time mask ratio rtest . (b) Heatmap over (rtrain,rtest) , where darker colors indicate lower MPJPE.
Mask ratio in train
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Mask-rate-avg
0.070
0.058
0.051
0.046
0.047
0.051
0.066
Table 2: Reconstruction performance under different training mask ratios. Each model is evaluated across test-time mask ratios rtest∈{0.1,0.2,…,0.9} . We report the mask-rate-averaged MPJPE (mean over rtest ).
Figure 6: Reconstruction example at a representative test-time mask ratio ( rtest=0.5 ). The model reconstructs a coherent body structure and preserves the overall motion trend across frames.
Method
Clean (0.0)
Low (0.2)
Moderate (0.4)
High (0.6)
Baseline (Standard)
0.920 / 1.884 (+0.0 / +0.0%)
0.940 / 1.916 (+2.2 / +1.7%)
0.969 / 1.957 (+5.3 / +3.9%)
1.050 / 2.090 (+14.0 / +10.9%)
Baseline (Reconstruction)
0.920 / 1.884 (+0.0 / +0.0%)
0.927 / 1.892 (+0.8 / +0.4%)
0.940 / 1.907 (+2.1 / +1.2%)
0.973 / 1.939 (+5.7 / +2.9%)
Baseline (Corruption-trained)
0.956 / 1.948 (+0.0 / +0.0%)
0.953 / 1.942 (-0.3 / -0.3%)
0.955 / 1.946 (-0.1 / -0.1%)
0.960 / 1.955 (+0.4 / +0.4%)
Ours
0.898 / 1.832 (+0.0 / +0.0%)
0.913 / 1.853 (+1.7 / +1.1%)
0.934 / 1.884 (+4.0 / +2.8%)
1.017 / 2.058 (+13.3 / +12.3%)
Table 3: Comparison under random joint masking. Missingness levels are grouped as Clean ( 0.0 ), Low ( 0.2 ), Moderate ( 0.4 ), and High ( 0.6 ), where the value in parentheses denotes the mask rate. Each cell reports ADE / FDE, followed by the degradation rate (%) relative to the clean condition of the same method. Bold indicates the best ADE / FDE at each missingness level.
Method
Clean
Skeleton disabled
Δ (ADE / FDE)
Baseline (Standard / Reconstruction)
0.920 / 1.884
1.563 / 3.186
+0.643 / +1.301
Baseline (Corruption-trained)
0.956 / 1.948
1.239 / 2.355
+0.283 / +0.407
Ours
0.898 / 1.832
2.389 / 4.789
+1.491 / +2.957
Table 4: Analysis of skeleton contribution. We quantify the contribution of skeleton cues by zeroing the 3D pose part in the Cross-Modality Transformer input and measuring the resulting performance drop. A larger drop ( Δ ) indicates stronger reliance on skeleton information.
Figure 7: Straight-trajectory case under clean and masked inputs. This example is consistent with the quantitative trends: Ours remains stable, while the baselines show either masking-induced degradation or lower clean accuracy.
Figure 8: Curved-trajectory case under clean and masked inputs. This is a skeleton-sensitive case: the baselines often fail to capture path curvature even under clean input, whereas Ours preserves the turning tendency under both clean and masked inputs.
Depth
1
2
3
4
Mask-rate-avg MPJPE
0.058
0.047
0.046
0.051
Table 5: Effect of encoder depth L in the ST-GCN-based reconstruction encoder. We report mask-rate-averaged MPJPE (lower is better), averaged over test-time mask ratios rtest∈{0.1,0.2,…,0.9} .
Loss design
Masked
Unmasked
Average
Cos sim
All-joint loss
0.046
0.026
0.036
0.935
Masked-only loss
0.043
0.218
0.131
0.899
Table 6: Effect of reconstruction loss design in self-supervised pretraining ( rtrain=0.5 , rval=0.5 ). Masked-only loss slightly improves masked-joint reconstruction, but severely degrades unmasked-joint reconstruction and representation consistency.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Reconstruction example 1 (successful case) under two validation masking ratios ( rval=0.4 and 0.6 ).
Figure 10: Reconstruction example 2 (successful case) under two validation masking ratios ( rval=0.4 and 0.6 ).
Figure 11: Reconstruction example 3 (challenging case) under two validation masking ratios ( rval=0.4 and 0.6 ).
Figure 12: Reconstruction example 3 (challenging case) under two validation masking ratios ( rval=0.4 and 0.6 ).
Model
Clean
Skeleton disabled
Δ ADE
Δ FDE
Baseline (Standard)
0.920 / 1.884
1.563 / 3.186
+0.643
+1.301
Baseline (ST-GCN)
0.895 / 1.810
2.657 / 4.868
+1.762
+3.058
Ours
0.898 / 1.832
2.389 / 4.789
+1.491
+2.957
Appendix
Table 7: Controlled skeleton ablation comparison (clean input; mask rate 0.0 ). We compare our pretrained encoder against a same-parameter ST-GCN replacement trained without pretraining. This provides a reference with similar skeleton contribution ( Δ ), isolating the effect of pretraining.
Method
Clean (0.0)
Low (0.2)
Moderate (0.4)
High (0.6)
Baseline (Standard)
0.920 / 1.884
0.940 / 1.916
0.969 / 1.957
1.050 / 2.090
Baseline (ST-GCN)
0.895/1.810
1.060 / 2.094
1.330 / 2.646
1.543 / 3.106
Ours
0.898 / 1.832
0.913 / 1.853
0.934 / 1.884
1.017 / 2.058
Appendix
Table 8: Performance under random joint masking. Each entry reports ADE / FDE. Bold indicates the best result at each mask rate.
Method
Clean (0.0)
Low (0.2)
Moderate (0.4)
High (0.6)
Baseline (ST-GCN)
0.895/1.810
1.060 / 2.094
1.330 / 2.646
1.543 / 3.106
Baseline (ST-GCN) + Recon
0.895 / 1.810
0.902 / 1.820
0.969 / 1.952
1.357 / 2.781
Ours
0.898 / 1.832
0.913 / 1.853
0.934/1.884
1.017 / 2.058
Appendix
Table 9: Performance under random joint masking. Each entry reports ADE / FDE. Bold indicates the best result at each mask rate. The coordinate-completion baseline reconstructs missing joints and feeds completed coordinates to the predictor.
Method
Clean (0.0)
Low (0.2)
Moderate (0.4)
High (0.6)
Baseline (Reconstruction)
0.920 / 1.884
0.927 / 1.892
0.940 / 1.907
0.973 / 1.939
Ours
0.898 / 1.832
0.913 / 1.853
0.934 / 1.884
1.017 / 2.058
Ours + Reconstruction
0.898 / 1.832
0.903 / 1.838
0.910 / 1.849
0.933 / 1.878
Appendix
Table 10: Compatibility of reconstruction-based coordinate completion with the proposed method under random joint masking. Each entry reports ADE / FDE. Lower is better.
Figure 13: Qualitative case study on prediction updates under clean skeleton inputs. The same scene is shown at two observation steps separated by two additional observed frames. In each panel, the top row shows the input skeleton sequence and the bottom row shows the predicted trajectories by Baseline (Standard) and Ours with the ground truth. In the right panel, the two newly observed frames are highlighted with red boxes. The later observation reveals a clearer body-orientation cue for the upcoming turn. While both methods are inaccurate at the earlier step, Ours updates its prediction toward the curved ground-truth trajectory after the additional observations, whereas Baseline (Standard) remains less responsive.
Figure 14: Qualitative comparison on curved-trajectory scenes under clean and partially missing skeleton inputs. Methods that effectively exploit skeletal cues better capture turning motion under clean inputs, whereas methods with low skeleton reliance tend to produce overly straight predictions. Under random joint masking ( r=0.4 ), the proposed method better preserves the turning tendency than the baselines, supporting improved robustness through more stable skeleton-based motion cues.
Method
Params
Inference Time
Baseline (Standard/Corruption-trained)
3,188,546
1.942
Ours/Baseline (ST-GCN)
3,703,198
2.099
Baseline (Reconstruction)
3,720,097
2.132
Ours + Recon
4,234,749
2.273
Appendix
Table 11: Comparison of the number of parameters and inference time (ms/sample).
Current skeleton representation learning paradigms face distinct limitations: Contrastive Learning (CL) often overlooks fine-grained motion details, while Masked Auto-Encoders (MAE) rely on coordinate-level reconstruction. This reconstruction inherently demands dense token sequences and heavy decoders, wasting pre-training computation on discarded components and forcing downstream inference to process dense token grids. To resolve these bottlenecks, we propose SLiM (Skeleton Less is More), a compact-token framework that unifies masked feature prediction and contrastive learning via a shared encoder. By shifting the objective from raw coordinate reconstruction to decoder-free, teacher-guided feature prediction, SLiM breaks the reliance on dense tokenization and enables effective learning with a highly compact token grid. Crucially, to prevent trivial shortcut learning arising from strong inter-joint dependencies of human, we introduce Semantic Tube Masking together with Skeleton-Aware Augmentations to enforce deep skeletal-temporal reasoning and anatomical consistency. Extensive experiments across multiple downstream protocols demonstrate that SLiM achieves state-of-the-art performance while structurally reducing inference computation by 7.89× compared to dense-token MAE baselines.
Jeonghyeok Do, Yun Chen, Geunhyuk Youk +1
Information & Electronics Research Institute, KAIST · School of Electrical Engineering, KAIST
This paper presents a self-supervised representation learning framework for understanding 3D skeleton-based human motion in soccer, using future motion prediction as the learning objective. Since human motion is inherently uncertain, accounting for multiple plausible futures is essential for capturing the underlying motion dynamics and learning effective representations. To this end, we introduce a conditioning module for motion prediction that models a probabilistic distribution over discretized future motions in 3D Euclidean space, learning multimodality with explicit supervision from future trajectories. Experiments on large-scale soccer player tracking data show that our approach substantially improves motion prediction accuracy. Moreover, the learned representations effectively transfer to multiple soccer downstream applications, demonstrating strong cross-task generalization.
Yizhou Xu, Lars Bretzner, Tiesheng Wang +1
KTH Royal Institute of Technology · EA Sports TRACAB
Human motion describes the three-dimensional full-body movement of a person. Anticipating such motion holds significant relevance across a wide range of application domains such as human-robot interaction, autonomous driving, animation, and healthcare. In recent research, spatial and temporal dependencies are modeled by bidirectional attention mechanisms. These typically anticipate human motion in an autoregressive manner which could cause an accumulation of errors over time. As a consequence, they solely focus on local pose forecasting. To address these limitations, we propose a non-autoregressive transformer based on spatio-temporal attention, and train it not only for local pose anticipation, but also for global motion prediction in space. Furthermore, to enhance its applicability in real-world scenarios, our model is also trained to recover missing joints due to occlusions, and is capable of processing varying lengths of history observations. Our code is publicly available at https://github.com/Q-Y-Yang/Prediction-of-Local-and-Global-Human-Motion.
Qiaoyue Yang, Sven Heutger, Christopher Niemann +3