Skeleton-based Zero-Shot Spatio-Temporal Action Localization via Weakly-Supervised Pretraining

Koshiro Nagano; Fumiaki Sato; Ryo Hachiuma; Kazuki Tsutsukawa; Taiki Sekii

cs.CVAug 26, 2026

Skeleton-based Zero-Shot Spatio-Temporal Action Localization via Weakly-Supervised Pretraining

Authors: Koshiro Nagano, Fumiaki Sato, Ryo Hachiuma, Kazuki Tsutsukawa, Taiki Sekii

Organizations: Konica Minolta, Inc. · Keio University · CyberAgent · NVIDIA

Abstract

We propose a novel pretraining strategy for skeleton-based zero-shot spatio-temporal action localization to estimate unseen actions for person instances while overcoming high annotation costs for training via new target actions and pretraining using large-scale action scenery datasets. Specifically, our approach, termed Skeleton-Language feature Pooling Switching, introduces a weakly-supervised vision-language pretraining mechanism. This mechanism transitions pooling kernels from pretraining, which aggregates skeleton features at the video level and aligns them with each video's known action text embeddings, to the inference phase that computes instance-level features without training via target actions. Furthermore, we propose Scene-Mixed Discriminative Contrastive Learning to distinguish actions at the instance level within the combined scene through the MIL framework. Our experiments on four public spatio-temporal action localization and classification datasets demonstrate that the proposed method effectively addresses annotation limitations.

Figures & tables

Figure 1: Overview of proposed SLPS learning framework.

Figure 2: Overview of Scene-Mixed Discriminative Contrastive Learning for zero-shot spatio-temporal action localization. The modules same as those in Fig. 1 are abbreviated with the same color.

Table 1: Performance comparison with conventional methods. (a) spatio-temporal action localization methods on UCF101-24, (b) tracking-based methods on FDD, and (c) skeleton-based video classification methods on RWF and MF. Conventional methods are trained in a supervised (†) or weakly-supervised manner (‡).

Architecture	Training strategy	AP (%)
Ours	3-shot	$33.9$
	2-shot	$29.9$
	1-shot	$23.4$
	w/o SM-DCL	$31.2$
	w/ SM-DCL	$\mathbf{34.1}$

Table 2: Performance comparison for ablation studies on UCF101-24 ( cf. Sec. 4.3 ).

Method	Acc. (%)
X-CLIP [ 45 ]	$69.3$
EVA [ 19 ]	$71.3$
ViFi-CLIP [ 51 ]	$72.8$
Qwen2.5-VL-7B-Instruct [ 2 ]	$80.0$
Ours	$\mathbf{84.0}$

Table 3: Comparison with appearance-based methods on RWF-2000.

Figure 3: Robustness against degradation of input videos.

Loss weight	$0.0$	$0.2$	$0.4$	$0.6$	$0.8$
AP (%)	$24.8$	$30.4$	$30.5$	$31.2$	$30.1$

Table 12: Ablation study on the mixing ratios of action classification loss and contrastive loss.

Method	AP (%)
Ave. Pool. [ 71 ]	$30.6$
Attention [ 71 ]	$26.4$
Max Pool. (Ours)	$31.2$

Table 13: Ablation study on pooling switching mechanism on UCF101-24.

Figure 6: Qualitative zero-shot spatio-temporal action localization results of the proposed method on the FDD dataset. The input keypoints, including the fall behaviors, are visualized in the figure.

Method	Input information	Learn target actions in training (✗) or not (✓)	Supervision at instance (✗) or video (✓) level	Multi-person tracking free
[ 57 , 59 , 38 , 33 , 81 ]	RGB	✗		✗
[ 15 ]	Skeleton		✗	✗
[ 47 , 20 , 66 , 68 , 82 ]	RGB			✓
[ 1 , 12 , 17 ]	RGB			✗
[ 4 ]	RGB		✓	✓
[ 28 ]	Skeleton			✗

Table 15: Summary of spatio-temporal action localization approaches.

Explore similar work

Zero-Shot Skeleton-Based Action Anticipation
Aug 14, 2026Hongsong Wang, Pengbo Yan, Yang Zhang +1Zero-Shot Learning Skeleton-Based Action Recognition
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Southeast University - Monash University Joint Graduate School, SuZhou 215123, China · School of Computer Science and Software Engineering, Shenzhen University, Shenzhen 518060, China +1
Zero-Shot Temporal Action Localization Through Textual Guidance
May 21, 2026Benedetta Liberatori, Alessandro Conti, Lorenzo Vaquero +3Zero-Shot Learning Temporal Action Detection
University of Trento, Trento, Italy · Fondazione Bruno Kessler, Trento, Italy
SCALE: Semantic- and Confidence-Aware Conditional Variational Autoencoder for Zero-shot Skeleton-based Action Recognition
Apr 2, 2026Soroush Oraki, Feng Ding, Jie LiangSkeleton-Based Action Recognition Zero-Shot Learning
School of Engineering Science, Simon Fraser University, Burnaby, BC, Canada · School of Information Science and Technology, Eastern Institute of Technology, Ningbo, China

Aug 14, 2026J/K move · Enter open · S save

Zero-Shot Skeleton-Based Action Anticipation

Hongsong Wang, Pengbo Yan, Yang Zhang, Qiuxia Lai

School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Southeast University - Monash University Joint Graduate School, SuZhou 215123, China · School of Computer Science and Software Engineering, Shenzhen University, Shenzhen 518060, China+1

Zero-Shot Learning Skeleton-Based Action Recognition

Open paper AskPDF arXiv

Action anticipation (AA) aims to recognize ongoing human or humanoids actions from partial observations, enabling robots to predict intentions before the actions are completed. Although skeleton-based AA offers efficiency advantages, existing approaches assume that all action classes are seen during training, which limits their deployment in real-world scenarios where novel actions inevitably arise. To address this gap, we study the new task of Zero-Shot Skeleton-Based Action Anticipation (ZS-SkAA). This task requires recognizing unseen action classes using only limited early-stage skeleton sequences, combining the challenges of partial observations, temporal dynamics, and zero-shot generalization. To establish foundational research for ZS-SkAA, we introduce:(1) A baseline model comprising a spatio-temporal feature extractor and a mutual information estimation and maximization module. This baseline model explicitly aligns partial visual features with semantic class embeddings across modalities by estimating and maximizing their mutual information, enhancing generalization to unseen classes.(2) A benchmark protocol using the NTU RGB+D dataset, which is adapted for rigorous ZS-SkAA evaluation. Experiments demonstrate the effectiveness of our model as a strong baseline for ZS-SkAA, achieving high zero-shot accuracy on NTU RGB+D. This work establishes ZS-SkAA as a vital research direction for real-world systems requiring generalization to novel actions.

Abstract

Explore similar work

Zero-Shot Skeleton-Based Action Anticipation

Zero-Shot Temporal Action Localization Through Textual Guidance

SCALE: Semantic- and Confidence-Aware Conditional Variational Autoencoder for Zero-shot Skeleton-based Action Recognition