Recent advances in large-scale pretrained vision models have demonstrated impressive capabilities across a wide range of downstream tasks, including cross-modal and multi-modal scenarios. However, their direct application to 3D human skeleton data remains challenging due to fundamental differences in data format. Moreover, the scarcity of large-scale skeleton datasets and the need to incorporate skeleton data into multi-modal action recognition without introducing additional model branches present significant research opportunities. To address these challenges, we introduce Skeleton-to-Image Encoding (S2I), a novel representation that transforms skeleton sequences into image-like data by partitioning and arranging joints based on body-part semantics and resizing to standardized image dimensions. This encoding enables, for the first time, the use of powerful vision-pretrained models for self-supervised skeleton representation learning, effectively transferring rich visual-domain knowledge to skeleton analysis. While existing skeleton methods often design models tailored to specific, homogeneous skeleton formats, they overlook the structural heterogeneity that naturally arises from diverse data sources. In contrast, our S2I representation offers a unified image-like format that naturally accommodates heterogeneous skeleton data. Extensive experiments on NTU-60, NTU-120, and PKU-MMD demonstrate the effectiveness and generalizability of our method for self-supervised skeleton representation learning, including under challenging cross-format evaluation settings.
Figures & tables
Figure 1: Overview of the existing methods and our proposed method. The existing methods train the skeleton models directly, while the proposed method converts skeleton data into image-like data and then train with the vision models, which can be initialized with pre-trained weights.
Figure 2: Illustration of the Skeleton-to-Image Encoding (S2I) process, which transforms skeleton sequences into image-like representations via joint partitioning, temporal stacking, and interpolation.
Figure 3Table 4
Method
NTU-60
NTU-120
PKU-MMD
C-sub
C-view
C-sub
C-set
Phase I
Phase II
LongT GAN Zheng et al. [2018]
39.1
48.1
-
-
67.7
26.0
P&C Su et al. [2020]
50.7
76.3
42.7
41.7
59.9
25.5
MS 2 L Lin et al. [2020]
52.6
-
-
-
64.9
27.6
SkeletonMAE Wu et al. [2023]
74.8
77.7
72.5
73.5
82.8
36.1
3s-SkeletonCLR Li et al. [2021]
75.0
79.8
60.7
62.6
85.3
-
Table 4: Comparison of Linear Evaluation results on NTU 60, NTU and PKU datasets. 3s- represents the ensemble results of joint(J), bone(B) and motion(M) streams. Bold and underlined indicate the best and second best results, respectively. The same notation applies throughout.
Table 6Table 7
Method
NTU120-Csub
NTU60-Csub
PKU-I-CS
PKU-II-CS
Toyota-CS
NW-UCLA
Self-pretrain
86.6
91.0
94.0
65.8
64.5
91.6
Universal-pretrain
87.0
91.6
95.2
71.1
68.0
93.5
Table 9: Universal pretraining evaluation on multiple skeleton datasets.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Config
Value
Optimizer
AdamW
Base learning rate
1.5×10−4
Weight decay
0.05
Optimizer momentum
β1,β2=0.9,0.95
Batch size
512
Learning rate schedule
cosine decay
Appendix
Table S1: Pretraining settings for MAE on S2I representation.
Config
Value
Optimizer
AdamW
Weight decay
0.05
Base learning rate
1.5×10−4
Optimizer momentum
β1,β2=0.9,0.95
Batch size
512
Learning rate schedule
cosine decay
Appendix
Table S2: Pretraining settings for DiffMAE on S2I representation.
Config
Value
Optimizer
AdamW
Base learning rate
1×10−3
Weight decay
0.05
Layer-wise lr decay
0.75
Batch size
128 (NTU/PKU/Toyota), 32 (N-UCLA)
Learning rate schedule
cosine decay
Appendix
Table S3: Fine-tuning settings for S2I representation.
Config
Value
Optimizer
SGD
Base learning rate
2×10−1
Weight decay
0
Optimizer momentum
0.9
Batch size
128
Learning rate schedule
cosine decay
Appendix
Table S4: Linear probing settings for S2I representation.
Figure S1: Comparison between the existing skeleton-specific pipeline and our proposed S2I-based pipeline for transfer learning across datasets. (a) Existing methods first align joint formats before pretraining and fine-tuning on target datasets, which may lead to information loss. (b) Our approach bypasses manual joint selection by encoding the raw skeleton sequence into an image via S2I, enabling unified processing across datasets using a vision model.
Figure S2: Comparison of skeleton-specific and universal representation learning pipelines. (a) Conventional methods require format-specific models for each skeleton format. (b) Our method encodes arbitrary skeleton formats into image representations via S2I, enabling unified pretraining with a single backbone model.
Figure S3: Visualization of three commonly used skeleton formats: (a) 25-joint (NTU/PKU), (b) 20-joint (NW-UCLA), and (c) 13-joint (Toyota). Each skeleton is partitioned into five semantic body parts—Spine, Left Arm, Right Arm, Left Leg, and Right Leg—highlighted in different colors to ensure consistent representation across formats.
Body Part
NTU/PKU (25 joints)
NW-UCLA (20 joints)
Toyota (13 joints)
Spine
head, neck, spine, middle of spine, base of spine
head, spine, middle of spine, base of spine
head
Left Arm
left shoulder, left elbow, left wrist, left hand, left thumb, tip of left hand
left shoulder, left elbow, left wrist, left hand
left shoulder, left elbow, left wrist
Right Arm
right shoulder, right elbow, right wrist, right hand, right thumb, tip of right hand
right shoulder, right elbow, right wrist, right hand
right shoulder, right elbow, right wrist
Left Leg
left hip, left knee, left ankle, left foot
left hip, left knee, left ankle, left foot
left hip, left knee, left ankle
Right Leg
right hip, right knee, right ankle, right foot
right hip, right knee, right ankle, right foot
right hip, right knee, right ankle
Appendix
Table S5: Semantic body part partitioning across different skeleton formats, using joint names.
School of Cyber Science and Engineering, Southeast University, Nanjing, China · School of Computer Science and Engineering, Southeast University, Nanjing, China