Current skeleton representation learning paradigms face distinct limitations: Contrastive Learning (CL) often overlooks fine-grained motion details, while Masked Auto-Encoders (MAE) rely on coordinate-level reconstruction. This reconstruction inherently demands dense token sequences and heavy decoders, wasting pre-training computation on discarded components and forcing downstream inference to process dense token grids. To resolve these bottlenecks, we propose SLiM (Skeleton Less is More), a compact-token framework that unifies masked feature prediction and contrastive learning via a shared encoder. By shifting the objective from raw coordinate reconstruction to decoder-free, teacher-guided feature prediction, SLiM breaks the reliance on dense tokenization and enables effective learning with a highly compact token grid. Crucially, to prevent trivial shortcut learning arising from strong inter-joint dependencies of human, we introduce Semantic Tube Masking together with Skeleton-Aware Augmentations to enforce deep skeletal-temporal reasoning and anatomical consistency. Extensive experiments across multiple downstream protocols demonstrate that SLiM achieves state-of-the-art performance while structurally reducing inference computation by 7.89× compared to dense-token MAE baselines.
Figures & tables
Figure 1: Accuracy–efficiency trade-off on NTU-60. Accuracy is averaged over X-Sub and X-View, and inference cost is measured by downstream GFLOPs. The suffix denotes the temporal token budget ( NT ). SLiM maintains SOTA accuracy with much lower inference cost than dense-token MAE-based methods.
Figure 2: Conceptual comparison between coordinate-reconstruction MAE and SLiM. (a) Standard skeleton MAE relies on dense skeletal-temporal tokens ( 30×J ) and a full-sequence decoder to reconstruct raw coordinates. Although the decoder is discarded after pre-training, downstream inference still requires dense encoder tokens. (b) SLiM replaces coordinate reconstruction with decoder-free masked feature prediction under a compact token grid ( 8×J ), using stochastic mask-token corruption and contrastive learning during pre-training.
Figure 3: Overview of SLiM. SLiM combines decoder-free masked feature prediction (MFP) and global-local contrastive learning (GLCL) in a teacher–student framework. The student predicts teacher feature distributions from masked skeleton views and aligns global-local views for contrastive learning, using a shared compact-token encoder.
Figure 4: Comparison of masking and augmentation strategies. Top: independent joint masking and conventional geometric augmentations may allow local interpolation shortcuts or distort skeleton structure. Bottom: Semantic Tube Masking masks connected anatomical regions over time, while Skeleton-Aware Augmentations preserve basic anatomical consistency during view generation.
Method
Publication
NTU-60
NTU-120
PKU-MMD II
X-Sub
X-View
X-Sub
X-Set
X-Sub
Other skeleton SSL baselines:
GL-Transformer [ 13 ]
ECCV’22
76.3
83.8
66.0
68.7
–
MacDiff [ 37 ]
ECCV’24
86.4
91.0
79.4
80.2
–
IGM [ 16 ]
ECCV’24
86.2
91.2
80.0
81.4
–
USDRL [ 36 ]
AAAI’25
85.2
91.7
76.6
78.1
54.4
Table 1: Linear evaluation on NTU-60, NTU-120, and PKU-MMD II. We report Top-1 accuracy (%) using the joint modality. Bold and underline indicate the best and second-best results, respectively.
Table 6
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Section
Descriptions for Analysis and Discussion
Sec. A
Implementation Details :
Architecture, experiment settings, and hyper-parameters
Sec. B
Hierarchical Temporal Sampling :
Global/local view generation
Sec. C
Semantic Tube Masking (STM) (Algorithm 1 )
Sec. D
Skeleton-Aware Augmentations (SAA) :
Appendix
Table 6: Overview of appendix contents.
Method
Clip T
Patch PT
Tokens NT×NJ
Inference GFLOPs
Avg. Acc. (%)
SkeletonMAE-8 [ 38 ]
64
8
8×25
3.59
70.50
SkeletonMAE-15 [ 38 ]
120
8
15×25
8.36
74.00
SkeletonMAE-30 [ 38 ]
120
4
30×25
28.32
76.25
MAMP-8 [ 22 ]
64
8
8×25
3.59
80.90
MAMP-15 [ 22 ]
120
8
15×25
8.36
85.50
MAMP-30 [ 22 ]
120
4
30×25
28.32
87.00
Appendix
Table 7: Token-budget configurations and average accuracy on NTU-60. T denotes the sampled clip length, PT denotes the temporal patch size, and NT=T/PT is the number of temporal tokens. All settings use J=25 joints and PJ=1 . Accuracy is averaged over the X-Sub and X-View protocols under linear evaluation. Inference GFLOPs are measured for the downstream encoder. SLiM-8 uses the default total batch size of 768 unless otherwise specified.
Method
Tokens
Inference GFLOPs
Single-view module GFLOPs
Reported Training Resource
NT×NJ
Enc.
Enc.
Dec.
Target/Aux.
GPU Type
Batch Size
SkeletonMAE [ 38 ]
30×25
28.32
1.97
17.70
–
8 × A6000
64
MAMP [ 22 ]
30×25
28.32
1.97
17.70
–
4 × RTX3090
128
S-JEPA [ 1 ]
30×25
28.32
1.97
17.70
28.32
8 × A100
256
GFP [ 31 ]
30×25
28.32
1.97
1.57
0.64
–
–
SLiM (Ours)
8×25
3.59
3.59
–
3.59
4 × A6000
768
Appendix
Table 8: Detailed cost comparison on NTU-60. NT×NJ denotes the encoder token grid after patchification. Inference GFLOPs are measured for the downstream encoder. Module-level GFLOPs are separated into student encoder, decoder, and target/auxiliary branches for one forward view. The target/auxiliary branch includes target generation networks for prior methods when used, and the EMA teacher forward for SLiM. These values do not aggregate SLiM’s multi-view student/teacher forwards and therefore are not a total per-iteration pre-training comparison. Training resources are reported as implementation context and are not intended as a controlled hardware comparison.
Method
To PKU-II
NTU-60
NTU-120
LongT-GAN [ 41 ]
44.8
–
ISC [ 34 ]
51.1
52.3
CMD [ 23 ]
56.0
57.0
MacDiff [ 37 ]
72.2
73.4
IGM [ 16 ]
59.8
–
Appendix
Table 9: Transfer learning performance on PKU-MMD II. We pre-train on the X-Sub splits of NTU-60 or NTU-120 and evaluate frozen representations on PKU-MMD II X-Sub with a linear classifier. Bold and underline indicate the best and second-best results, respectively.
MFP target
Loss
Target-specific module
X-Sub
X-View
Avg.
Raw joint coordinates
MSE
5-block decoder
84.30
89.62
86.96
Teacher-projected features
MSE
None
84.53
89.32
86.92
Teacher-projected features
MSE
Predictor head
85.07
90.12
87.59
Teacher prototype distribution
Cross-entropy
None
87.66±0.11
92.68±0.09
90.17
Appendix
Table 10: Matched comparison of masked-prediction targets on NTU-60. The uncertainty shown for the proposed variant is the sample standard deviation over three independently trained linear probes on one fixed pre-trained checkpoint; it is not pre-training-run variability.
Test condition
Clean pre-training
Degraded pre-training
Δ
Clean test
86.12
83.86
−2.26
Declared missing, 3/25
85.65±0.07
82.48±0.06
−3.17
Declared missing, 8/25
85.03±0.12
80.95±0.12
−4.08
Parent-joint fill, 3/25
80.40±0.18
77.73±0.10
−2.67
Parent-joint fill, 8/25
67.39±0.37
67.01±0.37
−0.38
Zero fill, 3/25
74.18±0.12
79.75±0.08
+5.57
Appendix
Table 12: Matched 50-epoch clean and degraded pre-training. Degraded pre-training uses persistent unflagged zero-fill of five non-reference joints plus Gaussian noise with σ=0.03 times mean bone length.
For learning generalizable motion representations from large-scale unlabeled data, Self-supervised learning (SSL) has become a widely adopted methodology. However, existing approaches are primarily limited by the inherent heterogeneity of skeleton data---characterized by varying joint counts, indexing protocols, and topological structures across different sensors---which typically necessitates training separate, sensor-specific, or even entirely dataset-specific models. To overcome this, we introduce SOfA (Skeleton One for All), the first generalist foundation model designed to achieve sensor-unified skeleton representation learning across diverse sensors. To accommodate the dimensional gap caused by varying joint counts, we introduce a fixed-size set of learnable Canonical Joint Slots, acting as a universal vessel that seamlessly accommodates arbitrary skeletal topologies. SOfA fills these slots via an attention mechanism that dynamically aggregates skeletal information from sensor-specific inputs. Furthermore, we resolve joint index misalignment between various sensors by introducing a Semantic Joint Embedding derived from a pre-trained text encoder, rather than relying on absolute positional embeddings. To validate our approach, we standardized ten 3D skeleton datasets for unified training. Extensive experiments demonstrate that SOfA can serve as a truly universal encoder, achieving state-of-the-art (SOTA) performance across a wide range of downstream tasks and sensor types, often outperforming dataset-specific specialist models with a single foundation model.
Jeonghyeok Do, Yun Chen, Munchurl Kim
Korea Advanced Institute of Science and Technology (KAIST)
Automatic skeleton generation involves predicting both joint positions and skeletal connectivity. However, existing approaches struggle to encode branch structures into token sequences and do not use test-time computation effectively. We study these choices within a unified autoregressive framework. First, we introduce branch-centric tokenization, a branch-aware representation that places structurally related elements next to each other and encodes connectivity directly in the sequence. Compared with standard BFS-style serialization, this representation yields more compact sequences. Second, we introduce view-augmented generation, a test-time augmentation procedure that applies axis-aligned rotations to the input mesh, maps all predictions back to a common frame, and selects the final skeleton based on mesh coverage and consistency among predictions from different views. Experiments show that our method achieves better skeleton prediction accuracy than state-of-the-art methods. In particular, our method reduces the CD-J2B error by 16.9% on the Articulation-XL2.0 dataset compared to the strongest directly comparable baseline, Auto-Connect. Qualitative results on in-the-wild meshes further demonstrate generalization across diverse inputs.
Recently, masked skeleton reconstruction models have emerged as strong action representation learners, driving significant progress in self-supervised skeleton-based action recognition. However, existing state-of-the-art methods must predict an exceedingly large number of spatiotemporal patches, significantly prolonging training time. Besides, by treating all spatiotemporal regions equally during reconstruction, these models are distracted from learning the critical motion patterns that underlie action semantics. To address these challenges, we propose Adaptive Masked Reconstruction (AMR), a faster and stronger pre-training framework. We first decouple the decoder from the encoder, enabling flexible prediction of larger spatiotemporal patches and dramatically reducing reconstruction complexity. Given that larger patches contain more complex information, which is challenging to predict and consequently degrades performance, we accordingly introduce an adaptive guidance module. This module identifies regions of high motion informativeness, guiding the model to focus on the most discriminative parts of each patch and alleviating reconstruction difficulty. Experiments on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD datasets demonstrate that AMR not only accelerates pre-training substantially but also improves downstream recognition accuracy, surpassing current state-of-the-art approaches.
Shengkai Sun, Zhiyong Cheng, Zefan Zhang +3
1Hefei University of Technology · 2Jilin University · 3Zhejiang Gongshang University +1