Current skeleton representation learning paradigms face distinct limitations: Contrastive Learning (CL) often overlooks fine-grained motion details, while Masked Auto-Encoders (MAE) rely on coordinate-level reconstruction. This reconstruction inherently demands dense token sequences and heavy decoders, wasting pre-training computation on discarded components and forcing downstream inference to process dense token grids. To resolve these bottlenecks, we propose SLiM (Skeleton Less is More), a compact-token framework that unifies masked feature prediction and contrastive learning via a shared encoder. By shifting the objective from raw coordinate reconstruction to decoder-free, teacher-guided feature prediction, SLiM breaks the reliance on dense tokenization and enables effective learning with a highly compact token grid. Crucially, to prevent trivial shortcut learning arising from strong inter-joint dependencies of human, we introduce Semantic Tube Masking together with Skeleton-Aware Augmentations to enforce deep skeletal-temporal reasoning and anatomical consistency. Extensive experiments across multiple downstream protocols demonstrate that SLiM achieves state-of-the-art performance while structurally reducing inference computation by 7.89× compared to dense-token MAE baselines.
Figures & tables
Figure 1: Accuracy–efficiency trade-off on NTU-60. Accuracy is averaged over X-Sub and X-View, and inference cost is measured by downstream GFLOPs. The suffix denotes the temporal token budget ( NT ). SLiM maintains SOTA accuracy with much lower inference cost than dense-token MAE-based methods.
Figure 2: Conceptual comparison between coordinate-reconstruction MAE and SLiM. (a) Standard skeleton MAE relies on dense skeletal-temporal tokens ( 30×J ) and a full-sequence decoder to reconstruct raw coordinates. Although the decoder is discarded after pre-training, downstream inference still requires dense encoder tokens. (b) SLiM replaces coordinate reconstruction with decoder-free masked feature prediction under a compact token grid ( 8×J ), using stochastic mask-token corruption and contrastive learning during pre-training.
Figure 3: Overview of SLiM. SLiM combines decoder-free masked feature prediction (MFP) and global-local contrastive learning (GLCL) in a teacher–student framework. The student predicts teacher feature distributions from masked skeleton views and aligns global-local views for contrastive learning, using a shared compact-token encoder.
Figure 4: Comparison of masking and augmentation strategies. Top: independent joint masking and conventional geometric augmentations may allow local interpolation shortcuts or distort skeleton structure. Bottom: Semantic Tube Masking masks connected anatomical regions over time, while Skeleton-Aware Augmentations preserve basic anatomical consistency during view generation.
Method
Publication
NTU-60
NTU-120
PKU-MMD II
X-Sub
X-View
X-Sub
X-Set
X-Sub
Other skeleton SSL baselines:
GL-Transformer [ 13 ]
ECCV’22
76.3
83.8
66.0
68.7
–
MacDiff [ 37 ]
ECCV’24
86.4
91.0
79.4
80.2
–
IGM [ 16 ]
ECCV’24
86.2
91.2
80.0
81.4
–
USDRL [ 36 ]
AAAI’25
85.2
91.7
76.6
78.1
54.4
Table 1: Linear evaluation on NTU-60, NTU-120, and PKU-MMD II. We report Top-1 accuracy (%) using the joint modality. Bold and underline indicate the best and second-best results, respectively.
Table 6
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Section
Descriptions for Analysis and Discussion
Sec. A
Implementation Details :
Architecture, experiment settings, and hyper-parameters
Sec. B
Hierarchical Temporal Sampling :
Global/local view generation
Sec. C
Semantic Tube Masking (STM) (Algorithm 1 )
Sec. D
Skeleton-Aware Augmentations (SAA) :
Appendix
Table 6: Overview of appendix contents.
Method
Clip T
Patch PT
Tokens NT×NJ
Inference GFLOPs
Avg. Acc. (%)
SkeletonMAE-8 [ 38 ]
64
8
8×25
3.59
70.50
SkeletonMAE-15 [ 38 ]
120
8
15×25
8.36
74.00
SkeletonMAE-30 [ 38 ]
120
4
30×25
28.32
76.25
MAMP-8 [ 22 ]
64
8
8×25
3.59
80.90
MAMP-15 [ 22 ]
120
8
15×25
8.36
85.50
MAMP-30 [ 22 ]
120
4
30×25
28.32
87.00
Appendix
Table 7: Token-budget configurations and average accuracy on NTU-60. T denotes the sampled clip length, PT denotes the temporal patch size, and NT=T/PT is the number of temporal tokens. All settings use J=25 joints and PJ=1 . Accuracy is averaged over the X-Sub and X-View protocols under linear evaluation. Inference GFLOPs are measured for the downstream encoder. SLiM-8 uses the default total batch size of 768 unless otherwise specified.
Method
Tokens
Inference GFLOPs
Single-view module GFLOPs
Reported Training Resource
NT×NJ
Enc.
Enc.
Dec.
Target/Aux.
GPU Type
Batch Size
SkeletonMAE [ 38 ]
30×25
28.32
1.97
17.70
–
8 × A6000
64
MAMP [ 22 ]
30×25
28.32
1.97
17.70
–
4 × RTX3090
128
S-JEPA [ 1 ]
30×25
28.32
1.97
17.70
28.32
8 × A100
256
GFP [ 31 ]
30×25
28.32
1.97
1.57
0.64
–
–
SLiM (Ours)
8×25
3.59
3.59
–
3.59
4 × A6000
768
Appendix
Table 8: Detailed cost comparison on NTU-60. NT×NJ denotes the encoder token grid after patchification. Inference GFLOPs are measured for the downstream encoder. Module-level GFLOPs are separated into student encoder, decoder, and target/auxiliary branches for one forward view. The target/auxiliary branch includes target generation networks for prior methods when used, and the EMA teacher forward for SLiM. These values do not aggregate SLiM’s multi-view student/teacher forwards and therefore are not a total per-iteration pre-training comparison. Training resources are reported as implementation context and are not intended as a controlled hardware comparison.
Method
To PKU-II
NTU-60
NTU-120
LongT-GAN [ 41 ]
44.8
–
ISC [ 34 ]
51.1
52.3
CMD [ 23 ]
56.0
57.0
MacDiff [ 37 ]
72.2
73.4
IGM [ 16 ]
59.8
–
Appendix
Table 9: Transfer learning performance on PKU-MMD II. We pre-train on the X-Sub splits of NTU-60 or NTU-120 and evaluate frozen representations on PKU-MMD II X-Sub with a linear classifier. Bold and underline indicate the best and second-best results, respectively.
MFP target
Loss
Target-specific module
X-Sub
X-View
Avg.
Raw joint coordinates
MSE
5-block decoder
84.30
89.62
86.96
Teacher-projected features
MSE
None
84.53
89.32
86.92
Teacher-projected features
MSE
Predictor head
85.07
90.12
87.59
Teacher prototype distribution
Cross-entropy
None
87.66±0.11
92.68±0.09
90.17
Appendix
Table 10: Matched comparison of masked-prediction targets on NTU-60. The uncertainty shown for the proposed variant is the sample standard deviation over three independently trained linear probes on one fixed pre-trained checkpoint; it is not pre-training-run variability.
Test condition
Clean pre-training
Degraded pre-training
Δ
Clean test
86.12
83.86
−2.26
Declared missing, 3/25
85.65±0.07
82.48±0.06
−3.17
Declared missing, 8/25
85.03±0.12
80.95±0.12
−4.08
Parent-joint fill, 3/25
80.40±0.18
77.73±0.10
−2.67
Parent-joint fill, 8/25
67.39±0.37
67.01±0.37
−0.38
Zero fill, 3/25
74.18±0.12
79.75±0.08
+5.57
Appendix
Table 12: Matched 50-epoch clean and degraded pre-training. Degraded pre-training uses persistent unflagged zero-fill of five non-reference joints plus Gaussian noise with σ=0.03 times mean bone length.