SCALE: Semantic- and Confidence-Aware Conditional Variational Autoencoder for Zero-shot Skeleton-based Action Recognition

Soroush Oraki; Feng Ding; Jie Liang

cs.CVApr 2, 2026

SCALE: Semantic- and Confidence-Aware Conditional Variational Autoencoder for Zero-shot Skeleton-based Action Recognition

Authors: Soroush Oraki, Feng Ding, Jie Liang

Organizations: School of Engineering Science, Simon Fraser University, Burnaby, BC, Canada · School of Information Science and Technology, Eastern Institute of Technology, Ningbo, China

Abstract

Zero-shot skeleton-based action recognition (ZSAR) aims to recognize action classes without any training skeletons from those classes, relying instead on auxiliary semantics from text. Existing approaches frequently depend on explicit skeleton-text alignment, which can be brittle when action names underspecify fine-grained dynamics and when unseen classes are semantically confusable. We propose SCALE, a lightweight and deterministic Semantic- and Confidence-Aware Listwise Energy-based framework that formulates ZSAR as class-conditional energy ranking. SCALE builds a text-conditioned Conditional Variational Autoencoder where frozen text representations parameterize both the latent prior and the decoder, enabling likelihood-based evaluation for unseen classes without generating samples at test time. To separate competing hypotheses, we introduce a semantic- and confidence-aware listwise energy loss that emphasizes semantically similar hard negatives and incorporates posterior uncertainty to adapt decision margins and reweight ambiguous training instances. Additionally, we utilize a latent prototype contrast objective to align posterior means with text-derived latent prototypes, improving semantic organization and class separability without direct feature matching. Experiments on NTU-60 and NTU-120 datasets show that SCALE consistently improves over prior VAE- and alignment-based baselines while remaining competitive with diffusion-based methods.

Figures & tables

Figure 1: Training pipeline of our proposed SCALE framework.

Figure 2: Inference pipeline of our proposed SCALE framework.

Method	Venue	NTU-60 (Acc, %)				NTU-120 (Acc, %)
Method	Venue	[55/5]	[48/12]	[40/20]	[30/30]	[110/10]	[96/24]	[80/40]	[60/60]
ReViSE [ 18 ]	ICCV2017	53.9	17.5	24.3	14.8	55.0	32.4	19.5	8.3
JPoSE [ 45 ]	ICCV2019	64.8	28.8	20.1	12.4	51.9	32.4	13.7	7.7
CADA-VAE [ 39 ]	CVPR2019	76.8	29.0	16.2	11.5	59.5	35.8	10.6	5.7
SynSE [ 15 ]	ICIP2021	75.8	33.3	19.9	12.0	62.7	38.7	13.6	7.7
SMIE [ 48 ]	ACMM2023	78.0	40.2	-	-	65.7	45.3	-	-

Table 1: Top-1 accuracy (%) comparison of ZSAR methods on the NTU-60 and NTU-120 datasets under multiple [seen/unseen] class splits under SynSE [ 15 ] and PURLS [ 50 ] benchmarks. Results in red denote the best performance, while those in blue indicate the second best and those in green indicate the third best.

Method	Parameters (M)	GFLOPs	Effective GFLOPs	Inference Time (ms)	Effective Inference Time (ms)
TDSM [ 10 ]	261.21	3.203	32.03 †	10.314	103.14 †
SCALE (Ours)	2.43	0.002	0.002	0.454	0.454
† TDSM averages over 10 inference trials.

Table 2: Computational efficiency comparison between SCALE and TDSM. Inference time and GFLOPs are measured per sample.

Training Loss	NTU-60 (Acc, %)		NTU-120 (Acc, %)
Training Loss	[55/5]	[48/12]	[110/10]	[96/24]
$-\mathrm{ELBO}^{+}$	78.7	43.3	65.1	47.1
$-\mathrm{ELBO}^{+}+\mathcal{L}_{\mathrm{proto}}$	81.1 ↑2.4	45.7 ↑2.4	69.0 ↑3.9	50.6 ↑3.5
$-\mathrm{ELBO}^{+}+\mathcal{L}_{\mathrm{SCALE}}$	82.7 ↑4.0	46.6 ↑3.3	70.6 ↑5.5	52.2 ↑5.1
$-\mathrm{ELBO}^{+}+\mathcal{L}_{\mathrm{SCALE}}+\mathcal{L}_{\mathrm{proto}}$	84.5 ↑5.8	50.0 ↑6.7	73.6 ↑8.5	54.9 ↑7.8

Table 3: Ablation study on the effect of different loss components. Top-1 accuracy (%) is reported on NTU-60 and NTU-120 under SynSE [ 15 ] settings.

$\alpha$	NTU-60 (Acc, %)		NTU-120 (Acc, %)
$\alpha$	[55/5]	[48/12]	[110/10]	[96/24]
$\alpha=0$	83.9	49.6	72.9	54.3
$\bm{\alpha}=\mathbf{1.0}$	84.5 ↑0.6	50.0 ↑0.4	73.6 ↑0.7	54.9 ↑0.6
$\alpha=2.0$	84.3 ↑0.4	50.1 ↑0.5	73.5 ↑0.6	54.4 ↑0.1
$\alpha=3.0$	84.4 ↑0.5	49.8 ↑0.2	73.3 ↑0.4	54.5 ↑0.2

Table 4: Ablation study on the semantic similarity bias parameter

\alpha

in the SCALE loss. Top-1 accuracy (%) is reported on NTU-60 and NTU-120 under SynSE [ 15 ] settings. Results in red denote the best performance.

Uncertainty Modeling	$\beta_{u}$	$\gamma$	NTU-60 (Acc, %)		NTU-120 (Acc, %)
Uncertainty Modeling	$\beta_{u}$	$\gamma$	[55/5]	[48/12]	[110/10]	[96/24]
Fixed margin, No reweighting	0	0	83.7	48.2	73.1	53.2
Uncertainty-adaptive margin only	0.5	0	84.3 ↑0.6	49.4 ↑1.2	73.5 ↑0.4	54.5 ↑1.3
Uncertainty-based reweighting only	0	2.0	84.0 ↑0.3	48.6 ↑0.4	73.3 ↑0.2	53.9 ↑0.7
Full $\bm{\mathcal{L}_{\mathrm{SCALE}}}$ loss	0.5	2.0	84.5 ↑0.8	50.0 ↑1.8	73.6 ↑0.5	54.9 ↑1.7

Table 5: Ablation study on uncertainty modeling in the SCALE loss. We analyze the individual and joint effects of uncertainty-adaptive margin relaxation (controlled by

\beta_{u}

) and uncertainty-based loss reweighting (controlled by

\gamma

). Results are shown on NTU-60 and NTU-120 under SynSE [ 15 ] splits.

Explore similar work

GenPrior: Unleashing Text-to-Motion Generative Priors for Zero-Shot Skeleton-based Action Recognition
Aug 3, 2026Jidong Kuang, Hongsong Wang, Jie GuiSkeleton-Based Action Recognition Zero-Shot Learning
School of Cyber Science and Engineering, Southeast University Nanjing, China · School of Computer Science and Engineering, Southeast University Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications Nanjing, China · School of Cyber Science and Engineering, Southeast University Purple Mountain Laboratories Engineering Research Center of Blockchain Application, Supervision and Management Nanjing, China
Visual Anchoring in Diffusion: Multimodal Zero-Shot Skeleton Action Recognition
Aug 5, 2026Zehao Bao, Shujun Guo, Bruce X. B. YuSkeleton-Based Action Recognition Diffusion Transformer
The University of Hong Kong Hong Kong SAR, China · Zhejiang University Hangzhou, China
Frequency-Enhanced Diffusion Models: Curriculum-Guided Semantic Alignment for Zero-Shot Skeleton Action Recognition
Apr 10, 2026Yuxi Zhou, Zhengbo Zhang, Jingyu Pan +2Skeleton-Based Action Recognition Diffusion Model Alignment
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan, China. · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore, Singapore. · School of Geodesy and Geomatics, Wuhan University, Wuhan, China. +2

Aug 3, 2026J/K move · Enter open · S save

GenPrior: Unleashing Text-to-Motion Generative Priors for Zero-Shot Skeleton-based Action Recognition

Jidong Kuang, Hongsong Wang, Jie Gui

School of Cyber Science and Engineering, Southeast University Nanjing, China · School of Computer Science and Engineering, Southeast University Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications Nanjing, China · School of Cyber Science and Engineering, Southeast University Purple Mountain Laboratories Engineering Research Center of Blockchain Application, Supervision and Management Nanjing, China

Skeleton-Based Action Recognition Zero-Shot Learning

Open paper AskPDF arXiv

Zero-shot skeleton-based action recognition (ZSAR) aims to recognize unseen action categories by aligning skeleton features with textual semantics. However, existing methods rely on text-derived prototypes that inherently lack geometric structure and physical constraints, resulting in a pronounced semantic-kinematic gap. To bridge this gap, we propose GenPrior, the first framework to exploit generative priors from pre-trained Text-to-Motion (T2M) models for ZSAR. Specifically, we introduce Dispersion-Gated Feature Fusion, which distills kinematic prototypes and intra-class dispersion from generative motion sequences and employs a learned gating network to adaptively inject reliable structural cues into textual embeddings while suppressing synthetic artifacts. Furthermore, we propose Generative Prototype Refinement, which leverages these generation-enhanced prototypes as anchors to mine high-confidence unseen samples, calibrating class prototypes toward the true distribution and thereby unleashing strong performance gains. Extensive experiments on NTU-60, NTU-120, and PKU-MMD demonstrate that GenPrior achieves state-of-the-art performance under both zero-shot and generalized zero-shot settings. Code is available at https://github.com/jidongkuang/GenPrior.

Abstract

Explore similar work

GenPrior: Unleashing Text-to-Motion Generative Priors for Zero-Shot Skeleton-based Action Recognition

Visual Anchoring in Diffusion: Multimodal Zero-Shot Skeleton Action Recognition

Frequency-Enhanced Diffusion Models: Curriculum-Guided Semantic Alignment for Zero-Shot Skeleton Action Recognition