Authors: Hongsong Wang, Pengbo Yan, Yang Zhang, Qiuxia Lai
Organizations: School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Southeast University - Monash University Joint Graduate School, SuZhou 215123, China · School of Computer Science and Software Engineering, Shenzhen University, Shenzhen 518060, China · State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, China
Action anticipation (AA) aims to recognize ongoing human or humanoids actions from partial observations, enabling robots to predict intentions before the actions are completed. Although skeleton-based AA offers efficiency advantages, existing approaches assume that all action classes are seen during training, which limits their deployment in real-world scenarios where novel actions inevitably arise. To address this gap, we study the new task of Zero-Shot Skeleton-Based Action Anticipation (ZS-SkAA). This task requires recognizing unseen action classes using only limited early-stage skeleton sequences, combining the challenges of partial observations, temporal dynamics, and zero-shot generalization. To establish foundational research for ZS-SkAA, we introduce:(1) A baseline model comprising a spatio-temporal feature extractor and a mutual information estimation and maximization module. This baseline model explicitly aligns partial visual features with semantic class embeddings across modalities by estimating and maximizing their mutual information, enhancing generalization to unseen classes.(2) A benchmark protocol using the NTU RGB+D dataset, which is adapted for rigorous ZS-SkAA evaluation. Experiments demonstrate the effectiveness of our model as a strong baseline for ZS-SkAA, achieving high zero-shot accuracy on NTU RGB+D. This work establishes ZS-SkAA as a vital research direction for real-world systems requiring generalization to novel actions.
Figures & tables
Figure 1: Comparison of skeleton-based action recognition and action anticipation.
Figure 2: Pipeline of our approach for zero-shot skeleton-based action anticipation.
Early Ratio
NTU RGB+D 60
NTU RGB+D 120
Joint
MultiFusion
Joint
MultiFusion
0.1
39.62
40.63
26.46
28.74
0.2
44.17
47.31
31.09
31.91
0.3
47.39
49.35
34.16
34.64
0.4
50.34
50.12
36.77
36.05
0.5
53.58
52.27
39.53
38.56
Table 1: Top-1 accuracy (%) of our model using joint-only and multi-modal inputs under varying early observation ratios. Fusion improves performance in early stages (0.1–0.3), while joint-only models perform better at later stages.
Dataset
Method
Early Observation Ratio
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
NTU60
SMIE [ 33 ]
36.50
40.48
43.47
45.31
49.16
53.88
57.13
60.82
63.58
Ours
39.62
44.17
47.39
50.34
53.58
57.59
59.34
61.13
63.64
NTU120
SMIE [ 33 ]
23.38
25.15
27.82
29.38
31.75
34.65
38.24
42.33
43.26
Ours
26.46
31.09
34.16
36.77
39.53
41.60
42.21
43.27
43.49
Table 2: Accuracy (%) comparison between our method and SMIE [ 33 ] under different early observation ratios on NTU RGB+D 60 and 120. Our method consistently outperforms the baseline, especially in low-ratio scenarios.
Figure 3: Ablation studies of our approach on the NTU-60 dataset.
Figure 4: Visualizations of results of skeleton-based action anticipation.
Zero-shot skeleton-based action recognition (ZSAR) aims to recognize unseen action categories by aligning skeleton features with textual semantics. However, existing methods rely on text-derived prototypes that inherently lack geometric structure and physical constraints, resulting in a pronounced semantic-kinematic gap. To bridge this gap, we propose GenPrior, the first framework to exploit generative priors from pre-trained Text-to-Motion (T2M) models for ZSAR. Specifically, we introduce Dispersion-Gated Feature Fusion, which distills kinematic prototypes and intra-class dispersion from generative motion sequences and employs a learned gating network to adaptively inject reliable structural cues into textual embeddings while suppressing synthetic artifacts. Furthermore, we propose Generative Prototype Refinement, which leverages these generation-enhanced prototypes as anchors to mine high-confidence unseen samples, calibrating class prototypes toward the true distribution and thereby unleashing strong performance gains. Extensive experiments on NTU-60, NTU-120, and PKU-MMD demonstrate that GenPrior achieves state-of-the-art performance under both zero-shot and generalized zero-shot settings. Code is available at https://github.com/jidongkuang/GenPrior.
Jidong Kuang, Hongsong Wang, Jie Gui
School of Cyber Science and Engineering, Southeast University Nanjing, China · School of Computer Science and Engineering, Southeast University Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications Nanjing, China · School of Cyber Science and Engineering, Southeast University Purple Mountain Laboratories Engineering Research Center of Blockchain Application, Supervision and Management Nanjing, China
Zero-shot skeleton-based action recognition (ZSAR) aims to recognize action classes without any training skeletons from those classes, relying instead on auxiliary semantics from text. Existing approaches frequently depend on explicit skeleton-text alignment, which can be brittle when action names underspecify fine-grained dynamics and when unseen classes are semantically confusable. We propose SCALE, a lightweight and deterministic Semantic- and Confidence-Aware Listwise Energy-based framework that formulates ZSAR as class-conditional energy ranking. SCALE builds a text-conditioned Conditional Variational Autoencoder where frozen text representations parameterize both the latent prior and the decoder, enabling likelihood-based evaluation for unseen classes without generating samples at test time. To separate competing hypotheses, we introduce a semantic- and confidence-aware listwise energy loss that emphasizes semantically similar hard negatives and incorporates posterior uncertainty to adapt decision margins and reweight ambiguous training instances. Additionally, we utilize a latent prototype contrast objective to align posterior means with text-derived latent prototypes, improving semantic organization and class separability without direct feature matching. Experiments on NTU-60 and NTU-120 datasets show that SCALE consistently improves over prior VAE- and alignment-based baselines while remaining competitive with diffusion-based methods.
Soroush Oraki, Feng Ding, Jie Liang
School of Engineering Science, Simon Fraser University, Burnaby, BC, Canada · School of Information Science and Technology, Eastern Institute of Technology, Ningbo, China
Zero-shot skeleton-based action recognition aims to recognize unseen actions by transferring knowledge from seen categories through semantic descriptions. Most existing methods typically align skeleton features with textual embeddings within a shared latent space. However, the absence of contextual cues, such as objects involved in the action, introduces an inherent gap between skeleton and semantic representations, making it difficult to distinguish visually similar actions. To address this, we propose SkeletonContext, a prompt-based framework that enriches skeletal motion representations with language-driven contextual semantics. Specifically, we introduce a Cross-Modal Context Prompt Module, which leverages a pretrained language model to reconstruct masked contextual prompts under guidance derived from LLMs. This design effectively transfers linguistic context to the skeleton encoder for instance-level semantic grounding and improved cross-modal alignment. In addition, a Key-Part Decoupling Module is incorporated to decouple motion-relevant joint features, ensuring robust action understanding even in the absence of explicit object interactions. Extensive experiments on multiple benchmarks demonstrate that SkeletonContext achieves state-of-the-art performance under both conventional and generalized zero-shot settings, validating its effectiveness in reasoning about context and distinguishing fine-grained, visually similar actions.
Ning Wang, Tieyue Wu, Naeha Sharif +5
Chang’an University · Xidian University · University of Western Australia +1