Robust Human Trajectory Prediction via Self-Supervised Skeleton Representation Learning

Taishu Arashima; Hiroshi Kera; Kazuhiko Kawamoto

cs.CVFeb 26, 2026

Robust Human Trajectory Prediction via Self-Supervised Skeleton Representation Learning

Authors: Taishu Arashima, Hiroshi Kera, Kazuhiko Kawamoto

Organizations: Chiba University

Abstract

Human trajectory prediction plays a crucial role in applications such as autonomous navigation and video surveillance. While recent works have explored the integration of human skeleton sequences to complement trajectory information, skeleton data in real-world environments often suffer from missing joints caused by occlusions. These disturbances significantly degrade prediction accuracy, indicating the need for more robust skeleton representations. We propose a robust trajectory prediction method that incorporates a self-supervised skeleton representation model pretrained with masked autoencoding. Experimental results in occlusion-prone scenarios show that our method improves robustness to missing skeletal data without sacrificing prediction accuracy, and consistently outperforms baseline models in clean-to-moderate missingness regimes.

Figures & tables

Figure 1: Impact of occlusion-induced missing skeletons on trajectory prediction accuracy. The top row shows a clean (non-occluded) skeleton observation, while the bottom row shows an occluded observation with missing joints. Using an existing human trajectory prediction (HTP) model, prediction accuracy is high under clean inputs (FDE = 0.17 m) but degrades substantially under occluded inputs (FDE = 0.87 m), where FDE denotes the final displacement error.

Figure 2: Overview of the self-supervised skeleton learning framework. Random masks are applied to the input skeleton sequence, and a spatio-temporal encoder extracts latent representations from the remaining visible joints. A decoder then reconstructs the original skeleton sequence. Through this process, the model learns structural representations invariant to missing data and noise.

Panel of Figure 3 — Figure 3: Illustration of three mask patterns used for self-supervised skeleton pretraining: (a) Temporally Consistent masks the same joints across all frames; (b) Random masks joints independently at each frame; (c) Body-Part masks multiple joints in the same body part together.

Figure 4: Overall architecture of the proposed framework. (a) Self-supervised pretraining : a skeleton encoder is pretrained by reconstructing masked joints from partially observed skeleton sequences. (b) Individual feature extraction : observed trajectories are embedded, and skeleton sequences are encoded by the pretrained (frozen) encoder. Positional encoding (PE) is added to each stream, the resulting tokens are concatenated, and a Cross-Modality Transformer produces an agent-wise representation. (c) Interaction modeling : the agent-wise representations are processed by a Social Transformer to model inter-agent interactions.

Train	Test	MPJPE	Cos Sim
Temporally-Consistent	Temporally-Consistent	0.058	0.841
	Random	0.201	0.833
	Body-Part	0.215	0.821
Random	Temporally-Consistent	0.062	0.912
	Random	0.036	0.935
	Body-Part	0.047	0.928

Table 1: Cross-evaluation of three masking strategies (Fig. 3 ). Each model is trained with one strategy and evaluated under all three test-time masks (Temporally-Consistent, Random, Body-Part) with a fixed test mask ratio of 0.5. Bold indicates the best result within each training strategy (min MPJPE / max cosine similarity).

Panel of Figure 5 — Figure 5: Reconstruction MPJPE under varying training and test-time mask ratios. (a) MPJPE versus the training mask ratio $r_{\mathrm{train}}$ , with one curve for each test-time mask ratio $r_{\mathrm{test}}$ . (b) Heatmap over $(r_{\mathrm{train}},r_{\mathrm{test}})$ , where darker colors indicate lower MPJPE.

Mask ratio in train	0.2	0.3	0.4	0.5	0.6	0.7	0.8
Mask-rate-avg	0.070	0.058	0.051	0.046	0.047	0.051	0.066

Table 2: Reconstruction performance under different training mask ratios. Each model is evaluated across test-time mask ratios

r_{\mathrm{test}}\in\{0.1,0.2,\dots,0.9\}

. We report the mask-rate-averaged MPJPE (mean over

r_{\mathrm{test}}

Figure 6: Reconstruction example at a representative test-time mask ratio ( $r_{\mathrm{test}}=0.5$ ). The model reconstructs a coherent body structure and preserves the overall motion trend across frames.

Method	Clean (0.0)	Low (0.2)	Moderate (0.4)	High (0.6)
Baseline (Standard)	0.920 / 1.884 (+0.0 / +0.0%)	0.940 / 1.916 (+2.2 / +1.7%)	0.969 / 1.957 (+5.3 / +3.9%)	1.050 / 2.090 (+14.0 / +10.9%)
Baseline (Reconstruction)	0.920 / 1.884 (+0.0 / +0.0%)	0.927 / 1.892 (+0.8 / +0.4%)	0.940 / 1.907 (+2.1 / +1.2%)	0.973 / 1.939 (+5.7 / +2.9%)
Baseline (Corruption-trained)	0.956 / 1.948 (+0.0 / +0.0%)	0.953 / 1.942 (-0.3 / -0.3%)	0.955 / 1.946 (-0.1 / -0.1%)	0.960 / 1.955 (+0.4 / +0.4%)
Ours	0.898 / 1.832 (+0.0 / +0.0%)	0.913 / 1.853 (+1.7 / +1.1%)	0.934 / 1.884 (+4.0 / +2.8%)	1.017 / 2.058 (+13.3 / +12.3%)

Table 3: Comparison under random joint masking. Missingness levels are grouped as Clean (

0.0

), Low (

0.2

), Moderate (

0.4

), and High (

0.6

), where the value in parentheses denotes the mask rate. Each cell reports ADE / FDE, followed by the degradation rate (%) relative to the clean condition of the same method. Bold indicates the best ADE / FDE at each missingness level.

Method	Clean	Skeleton disabled	$\mathbf{\Delta}$ (ADE / FDE)
Baseline (Standard / Reconstruction)	0.920 / 1.884	1.563 / 3.186	+0.643 / +1.301
Baseline (Corruption-trained)	0.956 / 1.948	1.239 / 2.355	+0.283 / +0.407
Ours	0.898 / 1.832	2.389 / 4.789	+1.491 / +2.957

Table 4: Analysis of skeleton contribution. We quantify the contribution of skeleton cues by zeroing the 3D pose part in the Cross-Modality Transformer input and measuring the resulting performance drop. A larger drop (

\Delta

) indicates stronger reliance on skeleton information.

Panel of Figure 7 — Figure 7: Straight-trajectory case under clean and masked inputs. This example is consistent with the quantitative trends: Ours remains stable, while the baselines show either masking-induced degradation or lower clean accuracy.

Panel of Figure 8 — Figure 8: Curved-trajectory case under clean and masked inputs. This is a skeleton-sensitive case: the baselines often fail to capture path curvature even under clean input, whereas Ours preserves the turning tendency under both clean and masked inputs.

Depth	1	2	3	4
Mask-rate-avg MPJPE	0.058	0.047	0.046	0.051

Table 5: Effect of encoder depth

L

in the ST-GCN-based reconstruction encoder. We report mask-rate-averaged MPJPE (lower is better), averaged over test-time mask ratios

r_{\mathrm{test}}\in\{0.1,0.2,\dots,0.9\}

Loss design	Masked	Unmasked	Average	Cos sim
All-joint loss	0.046	0.026	0.036	0.935
Masked-only loss	0.043	0.218	0.131	0.899

Table 6: Effect of reconstruction loss design in self-supervised pretraining (

r_{\mathrm{train}}=0.5

r_{\mathrm{val}}=0.5

). Masked-only loss slightly improves masked-joint reconstruction, but severely degrades unmasked-joint reconstruction and representation consistency.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: Reconstruction example 1 (successful case) under two validation masking ratios ( $r_{\mathrm{val}}=0.4$ and $0.6$ ).

Panel of Figure 10 — Figure 10: Reconstruction example 2 (successful case) under two validation masking ratios ( $r_{\mathrm{val}}=0.4$ and $0.6$ ).

Panel of Figure 11 — Figure 11: Reconstruction example 3 (challenging case) under two validation masking ratios ( $r_{\mathrm{val}}=0.4$ and $0.6$ ).

Panel of Figure 12 — Figure 12: Reconstruction example 3 (challenging case) under two validation masking ratios ( $r_{\mathrm{val}}=0.4$ and $0.6$ ).

Model	Clean	Skeleton disabled	$\mathbf{\Delta}$ ADE	$\mathbf{\Delta}$ FDE
Baseline (Standard)	0.920 / 1.884	1.563 / 3.186	+0.643	+1.301
Baseline (ST-GCN)	0.895 / 1.810	2.657 / 4.868	+1.762	+3.058
Ours	0.898 / 1.832	2.389 / 4.789	+1.491	+2.957

Appendix

Table 7: Controlled skeleton ablation comparison (clean input; mask rate

0.0

). We compare our pretrained encoder against a same-parameter ST-GCN replacement trained without pretraining. This provides a reference with similar skeleton contribution (

\Delta

), isolating the effect of pretraining.

Method	Clean (0.0)	Low (0.2)	Moderate (0.4)	High (0.6)
Baseline (Standard)	0.920 / 1.884	0.940 / 1.916	0.969 / 1.957	1.050 / 2.090
Baseline (ST-GCN)	0.895/1.810	1.060 / 2.094	1.330 / 2.646	1.543 / 3.106
Ours	0.898 / 1.832	0.913 / 1.853	0.934 / 1.884	1.017 / 2.058

Appendix

Table 8: Performance under random joint masking. Each entry reports ADE / FDE. Bold indicates the best result at each mask rate.

Method	Clean (0.0)	Low (0.2)	Moderate (0.4)	High (0.6)
Baseline (ST-GCN)	0.895/1.810	1.060 / 2.094	1.330 / 2.646	1.543 / 3.106
Baseline (ST-GCN) + Recon	0.895 / 1.810	0.902 / 1.820	0.969 / 1.952	1.357 / 2.781
Ours	0.898 / 1.832	0.913 / 1.853	0.934/1.884	1.017 / 2.058

Appendix

Table 9: Performance under random joint masking. Each entry reports ADE / FDE. Bold indicates the best result at each mask rate. The coordinate-completion baseline reconstructs missing joints and feeds completed coordinates to the predictor.

Method	Clean (0.0)	Low (0.2)	Moderate (0.4)	High (0.6)
Baseline (Reconstruction)	0.920 / 1.884	0.927 / 1.892	0.940 / 1.907	0.973 / 1.939
Ours	0.898 / 1.832	0.913 / 1.853	0.934 / 1.884	1.017 / 2.058
Ours + Reconstruction	0.898 / 1.832	0.903 / 1.838	0.910 / 1.849	0.933 / 1.878

Appendix

Table 10: Compatibility of reconstruction-based coordinate completion with the proposed method under random joint masking. Each entry reports ADE / FDE. Lower is better.

Panel of Figure 13 — Figure 13: Qualitative case study on prediction updates under clean skeleton inputs. The same scene is shown at two observation steps separated by two additional observed frames. In each panel, the top row shows the input skeleton sequence and the bottom row shows the predicted trajectories by Baseline (Standard) and Ours with the ground truth. In the right panel, the two newly observed frames are highlighted with red boxes. The later observation reveals a clearer body-orientation cue for the upcoming turn. While both methods are inaccurate at the earlier step, Ours updates its prediction toward the curved ground-truth trajectory after the additional observations, whereas Baseline (Standard) remains less responsive.

Panel of Figure 14 — Figure 14: Qualitative comparison on curved-trajectory scenes under clean and partially missing skeleton inputs. Methods that effectively exploit skeletal cues better capture turning motion under clean inputs, whereas methods with low skeleton reliance tend to produce overly straight predictions. Under random joint masking ( $r=0.4$ ), the proposed method better preserves the turning tendency than the baselines, supporting improved robustness through more stable skeleton-based motion cues.

Method	Params	Inference Time
Baseline (Standard/Corruption-trained)	3,188,546	1.942
Ours/Baseline (ST-GCN)	3,703,198	2.099
Baseline (Reconstruction)	3,720,097	2.132
Ours + Recon	4,234,749	2.273

Appendix

Table 11: Comparison of the number of parameters and inference time (ms/sample).

Human Trajectory Prediction

See all 14 papers

Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning
Mar 11, 2026Jeonghyeok Do, Yun Chen, Geunhyuk Youk +1Representation Learning Motion Representation Learning
Information & Electronics Research Institute, KAIST · School of Electrical Engineering, KAIST
Capturing Uncertainty in Human Motion for Representation Learning in Soccer
Aug 11, 2026Yizhou Xu, Lars Bretzner, Tiesheng Wang +1Self-Supervised Time Series Representation Learning Sports Video Analysis
KTH Royal Institute of Technology · EA Sports TRACAB
Long-Term Prediction of Local and Global Human Motion with Occlusion Recovery
Jun 26, 2026Qiaoyue Yang, Sven Heutger, Christopher Niemann +33D ViTs Human Motion Prediction
Neuro-Information Technology, Otto-von-Guericke-University Magdeburg, Magdeburg, 39106, Germany

Mar 11, 2026J/K move · Enter open · S save

Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning

Jeonghyeok Do, Yun Chen, Geunhyuk Youk, Munchurl Kim

Information & Electronics Research Institute, KAIST · School of Electrical Engineering, KAIST

Representation Learning Motion Representation Learning Masked Autoencoders Contrastive Learning

Open paper AskPDF arXiv

Current skeleton representation learning paradigms face distinct limitations: Contrastive Learning (CL) often overlooks fine-grained motion details, while Masked Auto-Encoders (MAE) rely on coordinate-level reconstruction. This reconstruction inherently demands dense token sequences and heavy decoders, wasting pre-training computation on discarded components and forcing downstream inference to process dense token grids. To resolve these bottlenecks, we propose SLiM (Skeleton Less is More), a compact-token framework that unifies masked feature prediction and contrastive learning via a shared encoder. By shifting the objective from raw coordinate reconstruction to decoder-free, teacher-guided feature prediction, SLiM breaks the reliance on dense tokenization and enables effective learning with a highly compact token grid. Crucially, to prevent trivial shortcut learning arising from strong inter-joint dependencies of human, we introduce Semantic Tube Masking together with Skeleton-Aware Augmentations to enforce deep skeletal-temporal reasoning and anatomical consistency. Extensive experiments across multiple downstream protocols demonstrate that SLiM achieves state-of-the-art performance while structurally reducing inference computation by 7.89

\times

compared to dense-token MAE baselines.

Abstract

Human Trajectory Prediction

Explore similar work

Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning

Capturing Uncertainty in Human Motion for Representation Learning in Soccer

Long-Term Prediction of Local and Global Human Motion with Occlusion Recovery