cs.CVSep 7, 2026
SaveOne for All: Generalist Foundation Model for Cross-Sensor Skeleton Representation Learning
Organizations: Korea Advanced Institute of Science and Technology (KAIST)
Abstract
For learning generalizable motion representations from large-scale unlabeled data, Self-supervised learning (SSL) has become a widely adopted methodology. However, existing approaches are primarily limited by the inherent heterogeneity of skeleton data---characterized by varying joint counts, indexing protocols, and topological structures across different sensors---which typically necessitates training separate, sensor-specific, or even entirely dataset-specific models. To overcome this, we introduce SOfA (Skeleton One for All), the first generalist foundation model designed to achieve sensor-unified skeleton representation learning across diverse sensors. To accommodate the dimensional gap caused by varying joint counts, we introduce a fixed-size set of learnable Canonical Joint Slots, acting as a universal vessel that seamlessly accommodates arbitrary skeletal topologies. SOfA fills these slots via an attention mechanism that dynamically aggregates skeletal information from sensor-specific inputs. Furthermore, we resolve joint index misalignment between various sensors by introducing a Semantic Joint Embedding derived from a pre-trained text encoder, rather than relying on absolute positional embeddings. To validate our approach, we standardized ten 3D skeleton datasets for unified training. Extensive experiments demonstrate that SOfA can serve as a truly universal encoder, achieving state-of-the-art (SOTA) performance across a wide range of downstream tasks and sensor types, often outperforming dataset-specific specialist models with a single foundation model.
Figures & tables
| Property | Standard SSL | HSP (CVPR’25) | SOfA (Ours) |
| Cross-Sensor Capability | ✗ | Intra-scene (Paired) | Universal (Unpaired) |
| Alignment Strategy | ✗ | Skeletal interpolation | Canonical Joint Slots Semantic Joint Embedding |
| Sensor Flexibility | ✗ | Pre-defined 2D-3D pair | Arbitrary 3D sensor |
| Dataset Dependency | Homogeneous | Heterogeneous (Paired) | Heterogeneous (Independent) |
| Representation Scope | Domain-specific | Multi-modal fusion | Generalist foundation |
Table 1: Conceptual comparison of skeleton representation learning paradigms. Standard SSL methods are constrained by fixed topologies, and recent approach, HSP ( Wang et al., 2025a ) , is limited to intra-scene multi-modal fusion using paired data. In contrast, our SOfA operates as the first generalist foundation model, achieving universal generalization across entirely independent datasets and arbitrary sensor configurations.
Figure 1: Motivation of SOfA. (a) Previous methods require isolated encoders for varying sensor topologies, yielding incompatible sensor-specific representation. (b) SOfA (Ours) introduces fixed-size set of learnable Canonical Joint Slots, . By treating arbitrary sensor sequences as conditions, a single shared encoder dynamically fills these slots to extract a standardized, sensor-unified representation, regardless of original joint counts.
Figure 2: Overview of SOfA. Our framework establishes a sensor-unified representation within a teacher–student architecture, where both networks dynamically fill a fixed-size set of Canonical Joint Slots. Processing a masked view, the student encoder minimizes the canonical reconstruction error ( ) to match the slot representations of the unmasked teacher, alongside a global distillation loss ( ) to align high-level semantics. By treating varying sensor tokens as dynamic conditions, the model exclusively extracts the unified canonical representation for downstream tasks.
Figure 3: Semantic Joint Alignment and Canonical Joint Slot aggregation. ( Left ) to resolve the joint index misalignment, Semantic Joint Embedding ( SJE ) utilizes a pretrained text encoder to map heterogeneous joint names into a shared latent space, ensuring anatomically consistent representations regardless of the sensor-specific numerical indices. ( Right ) the attention mechanism jointly propagates the concatenated class token, canonical tokens, and condition patch tokens. This allows the model to effectively fill the canonical slots by aggregating information from the visible joints, while an attention mask is applied to exclude the influence of zero-padded tokens.
| Method | Publication | NTU-60 | NTU-120 | PKU-MMD | ||
| X-Sub | X-View | X-Sub | X-Set | X-Sub | ||
| Sensor-Specific Representation : | ||||||
| GL-Transformer ( Kim et al., 2022 ) | ECCV’22 | 76.3 | 83.8 | 66.0 | 68.7 | – |
| CPM ( Zhang et al., 2022 ) | ECCV’22 | 78.7 | 84.9 | 68.7 | 69.6 | 48.3 |
| CMD ( Mao et al., 2022 ) | ECCV’22 | 79.8 | 86.9 | 70.3 | 71.5 | 43.0 |
| AimCLR ( Guo et al., 2022 ) | AAAI’22 | 74.3 | 79.7 | 63.4 | 63.4 | – |
Table 2: Comparison with recent SSL methods on NTU-60, NTU-120, and PKU-MMD (25-joint). Results are reported as Top-1 accuracy (%) under the linear evaluation . Unlike baseline methods that require independent training for each dataset, SOfA employs a single, unified representation across all benchmarks. Bold indicates the best result, and underline indicates the second best. X-View ∗ denotes evaluation on leakage-free test samples only (see Supplementary Material ).
| Modules | 25-Joint Datasets | 20-Joint Datasets | ||||
| CJS | SJE | NTU-60 | NTU-120 | PKU-MMD | NW-UCLA | UT-Kinect |
| ✗ | ✗ | 75.2 | 70.7 | 50.9 | 68.9 | 77.0 |
| ✓ | ✗ | 84.9 | 78.7 | 64.7 | 72.3 | 80.0 |
| ✗ | ✓ | 76.4 | 71.4 | 52.3 | 86.7 | 91.0 |
| ✓ | ✓ | 85.7 | 79.2 | 65.4 | 92.9 | 97.0 |
Table 5: Ablation of SOfA components. We isolate the contributions of CJS and SJE across heterogeneous topologies.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Section | Contents |
| Sec. A | Additional results and discussion |
| Sec. B | More ablation studies on CJS |
| Sec. C | Pre-training corpus and data leakage |
| Sec. D | Datasets and preprocessing |
| Sec. E | Implementation details |
Table 6: Overview of the Supplementary Material .
| Method | Modality | NTU-60 | NTU-120 | PKU-MMD | ||
| X-Sub | X-View | X-Sub | X-Set | X-Sub | ||
| SOfA | J | 85.7 | 79.2 | 80.4 | 65.4 | |
| SOfA ( ) | J | 85.5 | 79.0 | 80.1 | 65.6 | |
| SOfA-L | J | 87.0 | 79.5 | 82.0 | 67.1 | |
Table 7: Network scalability on the 25-joint benchmarks (NTU-60, NTU-120, PKU-MMD). Results are Top-1 accuracy (%) under the linear evaluation protocol using the joint (J) modality. SOfA is the base model, SOfA ( ) is evaluated on an extended synthetic topology, and SOfA-L is the deeper and wider variant. Bold denotes the best result.
| Method | Tokens | GFLOPs | PKU-MMD | |
| Train | Inference | X-Sub | ||
| Sensor-Specific : | ||||
| SkeletonMAE ( Wu et al., 2023 ) | 19.67 | 28.32 | 36.1 | |
| MAMP ( Mao et al., 2023 ) | 19.67 | 28.32 | 53.8 | |
| S-JEPA ( Abdelfattah and Alahi, 2024 ) | 47.99 | 28.32 | 53.5 | |
| GFP ( Sun et al., 2025 ) | 4.18 | 28.32 | 56.2 | |
Table 8: Computational complexity comparison on PKU-MMD. We report the number of tokens, training and inference GFLOPs, and Top-1 accuracy (X-Sub). SOfA achieves state-of-the-art accuracy while substantially reducing inference cost.
| Slots | 25-Joint Datasets | 20-Joint Datasets | GFLOPs | |||
| CJS | NTU-60 | NTU-120 | PKU-MMD | NW-UCLA | UT-Kinect | Inference |
| 0 | 76.4 | 71.4 | 52.3 | 86.7 | 91.0 | 3.60 |
| 5 | 85.7 | 79.2 | 65.4 | 92.9 | 97.0 | 4.30 |
| 10 | 85.5 | 79.0 | 64.5 | 92.7 | 98.0 | 5.41 |
| 15 | 85.3 | 78.9 | 63.5 | 91.8 | 97.0 | 6.55 |
Table 13: Ablation study on the number of Canonical Joint Slots ( CJS ). We report linear evaluation accuracy (%) on 25-joint and 20-joint benchmarks, together with inference GFLOPs.
Figure 4: t-SNE visualization of Canonical Joint Slot assignments. The five slots autonomously converge toward the semantic centers of distinct anatomical clusters (torso and four limbs), consistent with the optimal in Table 13 .
Figure 5: Construction of the NTU pre-training corpus and evaluation sets. NTU-120 contains all NTU-60 sequences. NTU-60 and NTU-120 use the same X-Sub assignment for all NTU-60 subjects. However, NTU-60 X-View splits samples by camera ID, while NTU-120 X-Set splits them by setup ID. (b) Pre-training uses only sequences assigned to training by both NTU-120 X-Sub and X-Set. (c–e) The official NTU-120 X-Sub, NTU-120 X-Set, and NTU-60 X-Sub test sets are used directly because they do not overlap with the pre-training corpus. (f) For NTU-60 X-View, overlapping test sequences are removed before evaluation, and the remaining leakage-free subset is denoted X-View ∗ .
| Dataset | Tracker | Joints | Classes | Samples | Evaluation Setting |
| NTU-60 ( Shahroudy et al., 2016 ) | Kinect v2 | 25 | 60 | 56,880 | Pre-train & Downstream |
| NTU-120 ( Liu et al., 2019 ) | Kinect v2 | 25 | 120 | 114,480 | Pre-train & Downstream |
| PKU-MMD ( Liu et al., 2017a ) | Kinect v2 | 25 | 51 | 7,096 | Pre-train & Downstream |
| ETRI-Act ( Jang et al., 2020 ) | Kinect v2 | 25 | 55 | 112,620 | Pre-train & Downstream |
| ETRI-LivingLab ( Jang et al., 2020 ) | Kinect v2 | 25 | 55 | 8,605 | Pre-train & Downstream |
| MSR-Action3D ( Li et al., 2010 ) | Kinect v1 | 20 | 20 | 567 | Pre-train & Downstream |
Table 14: Summary of the ten heterogeneous 3D skeleton datasets. Datasets are grouped by tracking protocol and skeletal topology; the 15-joint protocols are reserved exclusively for unseen evaluation.
Figure 6: Representative normalized skeleton samples from the ten heterogeneous datasets. Despite variations in joint count, tracking protocol, and acquisition environment, the sequences are consistently aligned to a shared hip-centered coordinate frame.
Figure 7: Additional normalized skeleton examples from the ten heterogeneous datasets, illustrating that our preprocessing preserves dataset-specific motion patterns while reducing cross-dataset geometric discrepancies.
Figure 8: Additional qualitative visualization of normalized skeletons across the ten datasets, including unseen 15-joint protocols. The unified preprocessing remains effective under substantial topological variation and interaction-specific layouts.
| Dataset | Axis | Min | Max | Mean | Median |
| 25-Joint Protocols (Pre-train & Downstream) | |||||
| NTU-60 ( Shahroudy et al., 2016 ) | X | -5.3128 | 5.2244 | -0.0060 | -0.0022 |
| Y | -2.7981 | 2.4020 | 0.0414 | 0.0338 | |
| Z | -4.8746 | 3.8188 | -0.0657 | -0.0536 | |
| NTU-120 ( Liu et al., 2019 ) | X | -5.3128 | 5.2244 | 0.0012 | -0.0012 |
| Y | -2.7981 | 2.4369 | 0.0422 | 0.0375 | |
Table 15: Per-axis coordinate statistics of the ten unified datasets after preprocessing (in meters). The heterogeneous datasets are consistently aligned to a shared hip-centered coordinate frame with comparable spatial ranges.