Recent advances in large-scale pretrained vision models have demonstrated impressive capabilities across a wide range of downstream tasks, including cross-modal and multi-modal scenarios. However, their direct application to 3D human skeleton data remains challenging due to fundamental differences in data format. Moreover, the scarcity of large-scale skeleton datasets and the need to incorporate skeleton data into multi-modal action recognition without introducing additional model branches present significant research opportunities. To address these challenges, we introduce Skeleton-to-Image Encoding (S2I), a novel representation that transforms skeleton sequences into image-like data by partitioning and arranging joints based on body-part semantics and resizing to standardized image dimensions. This encoding enables, for the first time, the use of powerful vision-pretrained models for self-supervised skeleton representation learning, effectively transferring rich visual-domain knowledge to skeleton analysis. While existing skeleton methods often design models tailored to specific, homogeneous skeleton formats, they overlook the structural heterogeneity that naturally arises from diverse data sources. In contrast, our S2I representation offers a unified image-like format that naturally accommodates heterogeneous skeleton data. Extensive experiments on NTU-60, NTU-120, and PKU-MMD demonstrate the effectiveness and generalizability of our method for self-supervised skeleton representation learning, including under challenging cross-format evaluation settings.
Figures & tables
Figure 1: Overview of the existing methods and our proposed method. The existing methods train the skeleton models directly, while the proposed method converts skeleton data into image-like data and then train with the vision models, which can be initialized with pre-trained weights.
Figure 2: Illustration of the Skeleton-to-Image Encoding (S2I) process, which transforms skeleton sequences into image-like representations via joint partitioning, temporal stacking, and interpolation.
Figure 3Table 4
Method
NTU-60
NTU-120
PKU-MMD
C-sub
C-view
C-sub
C-set
Phase I
Phase II
LongT GAN Zheng et al. [2018]
39.1
48.1
-
-
67.7
26.0
P&C Su et al. [2020]
50.7
76.3
42.7
41.7
59.9
25.5
MS 2 L Lin et al. [2020]
52.6
-
-
-
64.9
27.6
SkeletonMAE Wu et al. [2023]
74.8
77.7
72.5
73.5
82.8
36.1
3s-SkeletonCLR Li et al. [2021]
75.0
79.8
60.7
62.6
85.3
-
Table 4: Comparison of Linear Evaluation results on NTU 60, NTU and PKU datasets. 3s- represents the ensemble results of joint(J), bone(B) and motion(M) streams. Bold and underlined indicate the best and second best results, respectively. The same notation applies throughout.
Table 6Table 7
Method
NTU120-Csub
NTU60-Csub
PKU-I-CS
PKU-II-CS
Toyota-CS
NW-UCLA
Self-pretrain
86.6
91.0
94.0
65.8
64.5
91.6
Universal-pretrain
87.0
91.6
95.2
71.1
68.0
93.5
Table 9: Universal pretraining evaluation on multiple skeleton datasets.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Config
Value
Optimizer
AdamW
Base learning rate
1.5×10−4
Weight decay
0.05
Optimizer momentum
β1,β2=0.9,0.95
Batch size
512
Learning rate schedule
cosine decay
Appendix
Table S1: Pretraining settings for MAE on S2I representation.
Config
Value
Optimizer
AdamW
Weight decay
0.05
Base learning rate
1.5×10−4
Optimizer momentum
β1,β2=0.9,0.95
Batch size
512
Learning rate schedule
cosine decay
Appendix
Table S2: Pretraining settings for DiffMAE on S2I representation.
Config
Value
Optimizer
AdamW
Base learning rate
1×10−3
Weight decay
0.05
Layer-wise lr decay
0.75
Batch size
128 (NTU/PKU/Toyota), 32 (N-UCLA)
Learning rate schedule
cosine decay
Appendix
Table S3: Fine-tuning settings for S2I representation.
Config
Value
Optimizer
SGD
Base learning rate
2×10−1
Weight decay
0
Optimizer momentum
0.9
Batch size
128
Learning rate schedule
cosine decay
Appendix
Table S4: Linear probing settings for S2I representation.
Figure S1: Comparison between the existing skeleton-specific pipeline and our proposed S2I-based pipeline for transfer learning across datasets. (a) Existing methods first align joint formats before pretraining and fine-tuning on target datasets, which may lead to information loss. (b) Our approach bypasses manual joint selection by encoding the raw skeleton sequence into an image via S2I, enabling unified processing across datasets using a vision model.
Figure S2: Comparison of skeleton-specific and universal representation learning pipelines. (a) Conventional methods require format-specific models for each skeleton format. (b) Our method encodes arbitrary skeleton formats into image representations via S2I, enabling unified pretraining with a single backbone model.
Figure S3: Visualization of three commonly used skeleton formats: (a) 25-joint (NTU/PKU), (b) 20-joint (NW-UCLA), and (c) 13-joint (Toyota). Each skeleton is partitioned into five semantic body parts—Spine, Left Arm, Right Arm, Left Leg, and Right Leg—highlighted in different colors to ensure consistent representation across formats.
Body Part
NTU/PKU (25 joints)
NW-UCLA (20 joints)
Toyota (13 joints)
Spine
head, neck, spine, middle of spine, base of spine
head, spine, middle of spine, base of spine
head
Left Arm
left shoulder, left elbow, left wrist, left hand, left thumb, tip of left hand
left shoulder, left elbow, left wrist, left hand
left shoulder, left elbow, left wrist
Right Arm
right shoulder, right elbow, right wrist, right hand, right thumb, tip of right hand
right shoulder, right elbow, right wrist, right hand
right shoulder, right elbow, right wrist
Left Leg
left hip, left knee, left ankle, left foot
left hip, left knee, left ankle, left foot
left hip, left knee, left ankle
Right Leg
right hip, right knee, right ankle, right foot
right hip, right knee, right ankle, right foot
right hip, right knee, right ankle
Appendix
Table S5: Semantic body part partitioning across different skeleton formats, using joint names.
For learning generalizable motion representations from large-scale unlabeled data, Self-supervised learning (SSL) has become a widely adopted methodology. However, existing approaches are primarily limited by the inherent heterogeneity of skeleton data---characterized by varying joint counts, indexing protocols, and topological structures across different sensors---which typically necessitates training separate, sensor-specific, or even entirely dataset-specific models. To overcome this, we introduce SOfA (Skeleton One for All), the first generalist foundation model designed to achieve sensor-unified skeleton representation learning across diverse sensors. To accommodate the dimensional gap caused by varying joint counts, we introduce a fixed-size set of learnable Canonical Joint Slots, acting as a universal vessel that seamlessly accommodates arbitrary skeletal topologies. SOfA fills these slots via an attention mechanism that dynamically aggregates skeletal information from sensor-specific inputs. Furthermore, we resolve joint index misalignment between various sensors by introducing a Semantic Joint Embedding derived from a pre-trained text encoder, rather than relying on absolute positional embeddings. To validate our approach, we standardized ten 3D skeleton datasets for unified training. Extensive experiments demonstrate that SOfA can serve as a truly universal encoder, achieving state-of-the-art (SOTA) performance across a wide range of downstream tasks and sensor types, often outperforming dataset-specific specialist models with a single foundation model.
Jeonghyeok Do, Yun Chen, Munchurl Kim
Korea Advanced Institute of Science and Technology (KAIST)
Skeleton-based human action recognition is a powerful approach for understanding human behaviour from pose data, but collecting large-scale, diverse, and well-annotated 3D skeleton datasets is both expensive and labor-intensive. To address this challenge, we propose a conditional generative pipeline for data augmentation in skeleton action recognition. Our method learns the distribution of real skeleton sequences under the constraint of action labels, enabling the synthesis of diverse and high-fidelity data. Even with limited training samples, it can effectively generate skeleton sequences and achieve competitive recognition performance in low-data scenarios, demonstrating strong generalisation in downstream tasks. Specifically, we introduce a Transformer-based encoder-decoder architecture, combined with a generative refinement module and a dropout mechanism, to balance fidelity and diversity during sampling. Experiments on HumanAct12 and the refined NTU-RGBD (NTU-VIBE) dataset show that our approach consistently improves the accuracy of multiple skeleton-based action recognition models, validating its effectiveness in both few-shot and full-data settings. The source code can be found at here.
Xu Dong, Wanqing Li, Anthony Adeyemi-Ejeye +1
Innovative Media Lab, University of Surrey, Guildford, UK · Advanced Multimedia Research Lab, University of Wollongong, Wollongong, Australia
With the development of robotics, skeleton-based action recognition has become increasingly important, as human-robot interaction requires understanding the actions of humans and humanoid robots. Due to different sources of human skeletons and structures of humanoid robots, skeleton data naturally exhibit heterogeneity. However, previous works overlook the data heterogeneity of skeletons and solely construct models using homogeneous skeletons. Moreover, open-vocabulary action recognition is also essential for real-world applications. To this end, this work studies the challenging problem of heterogeneous skeleton-based action recognition with open vocabularies. We construct a large-scale Heterogeneous Open-Vocabulary (HOV) Skeleton dataset by integrating and refining multiple representative large-scale skeleton-based action datasets. To address universal skeleton-based action recognition, we propose a Transformer-based model that comprises three key components: unified skeleton representation, motion encoder for skeletons, and multi-grained motion-text alignment. The motion encoder feeds multi-modal skeleton embeddings into a two-stream Transformer-based encoder to learn spatio-temporal action representations, which are then mapped to a semantic space to align with text embeddings. Multi-grained motion-text alignment incorporates contrastive learning at three levels: global instance alignment, stream-specific alignment, and fine-grained alignment. Extensive experiments on popular benchmarks with heterogeneous skeleton data demonstrate both the effectiveness and the generalization ability of the proposed method. Code is available at https://github.com/jidongkuang/Universal-Skeleton.
Jidong Kuang, Hongsong Wang, Jie Gui
School of Cyber Science and Engineering, Southeast University, Nanjing, China · School of Computer Science and Engineering, Southeast University, Nanjing, China