Frequency-Enhanced Diffusion Models: Curriculum-Guided Semantic Alignment for Zero-Shot Skeleton Action Recognition

Yuxi Zhou; Zhengbo Zhang; Jingyu Pan; Zhiyu Lin; Zhigang Tu

cs.CVApr 10, 2026

Frequency-Enhanced Diffusion Models: Curriculum-Guided Semantic Alignment for Zero-Shot Skeleton Action Recognition

Authors: Yuxi Zhou, Zhengbo Zhang, Jingyu Pan, Zhiyu Lin, Zhigang Tu

Organizations: State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan, China. · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore, Singapore. · School of Geodesy and Geomatics, Wuhan University, Wuhan, China. · School of Mathematics and Statistics, Wuhan University, Wuhan, China. · Wuhan University Shenzhen Research Institute, Shenzhen, China.

Abstract

Human action recognition is pivotal in computer vision, with applications ranging from surveillance to human-robot interaction. Despite the effectiveness of supervised skeleton-based methods, their reliance on exhaustive annotation limits generalization to novel actions. Zero-Shot Skeleton Action Recognition (ZSAR) emerges as a promising paradigm, yet it faces challenges due to the spectral bias of diffusion models, which oversmooth high-frequency dynamics. Here, we propose Frequency-Aware Diffusion for Skeleton-Text Matching (FDSM), integrating a Semantic-Guided Spectral Residual Module, a Timestep-Adaptive Spectral Loss, and Curriculum-based Semantic Abstraction to address these challenges. Our approach effectively recovers fine-grained motion details, achieving state-of-the-art performance on NTU RGB+D, PKU-MMD, and Kinetics-skeleton datasets. Code has been made available at https://github.com/yuzhi535/FDSM. Project homepage: https://yuzhi535.github.io/FDSM.github.io/

Figures & tables

Figure 1: Frequency-domain visualization of skeleton sequences via DCT decomposition. Each row shows one action: (left) original skeleton with all frequency components; (center) low-frequency reconstruction capturing global pose topology; (right) high-frequency energy heat map, where bubble size at each joint indicates the magnitude of residual components. High-frequency micro-dynamics concentrate at extremities (hands, feet) in dynamic actions (e.g., Clapping, Waving hand) but are absent in near-static actions (e.g., Reading), illustrating why high-frequency suppression in the Diffusion Transformer degrades fine-grained action discrimination.

Figure 2: Overview of the proposed Frequency-Aware Diffusion Framework. The system integrates three synergistic modules: (1) Semantic-Guided Spectral Residual Module (SG-SRM) to amplify high-frequency dynamics based on LLM-derived kinematic priors; (2) Timestep-Adaptive Spectral Loss to enforce coherent spectral supervision during denoising; and (3) Curriculum-based Semantic Abstraction to bridge the semantic gap via a ‘rich-to-sparse’ training strategy.

Figure 3: Illustration of the Semantic-Guided Spectral Residual Module (SG-SRM). The module performs a DCT along the temporal dimension to decompose latent features into frequency coefficients. A semantic-guided gain filter, derived from the predicted kinematic intensity score, selectively amplifies high-frequency bands. The modulated spectrum is then reverted back to the temporal domain via IDCT, yielding enhanced latent features with restored fine-grained dynamics.

Figure 4: Illustration of the Curriculum-based Semantic Abstraction schedule. The probability

\gamma(e)

of using rich descriptions follows a cosine annealing curve, starting high to provide a semantic scaffold and decaying to zero to enforce self-reliant zero-shot inference.

Methods	NTU-60 (Acc, %)				NTU-120 (Acc, %)
Methods	55/5	48/12	40/20	30/30	110/10	96/24	80/40	60/60
ReViSE Hubert Tsai et al. (2017b)	53.91	17.49	24.26	14.81	55.04	32.38	19.47	8.27
JPoSE Wray et al. (2019)	64.82	28.75	20.05	12.39	51.93	32.44	13.71	7.65
CADA-VAE Schonfeld et al. (2019)	76.84	28.96	16.21	11.51	59.53	35.77	10.55	5.67
SynSE Gupta et al. (2021b)	75.81	33.30	19.85	12.00	62.69	38.70	13.64	7.73
SMIE Zhou et al. (2023a)	77.98	40.18	-	-	65.74	45.30	-	-

Table 1: Top-1 accuracy results of various zero-shot skeleton-based action recognition (ZSAR) methods evaluated on the SynSE and PURLS benchmarks for the NTU-60 and NTU-120 datasets. Each split is denoted as X/Y, where X represents the number of seen classes and Y the number of unseen classes. The results in bold highlight the best-performing model. For our method, the reported accuracy is the average value obtained from 10 trials, each with different Gaussian noise.

Methods	NTU-60 (Acc, %)	NTU-120 (Acc, %)	PKU-MMD (Acc, %)
Methods	55/5 split	110/10 split	46/5 split
ReViSE Hubert Tsai et al. (2017b)	60.94	44.90	59.34
JPoSE Wray et al. (2019)	59.44	46.69	57.17
CADA-VAE Schonfeld et al. (2019)	61.84	45.15	60.74
SynSE Gupta et al. (2021b)	64.19	47.28	53.85
SMIE Zhou et al. (2023a)	65.08	46.40	60.83

Table 2: Top-1 accuracy results of various ZSAR methods evaluated on the NTU-60, NTU-120, and PKU-MMD datasets under the SMIE benchmark. The reported values are the average performance across three splits.

Figure 5: T-SNE visualization of textual embeddings on NTU RGB+D 120. LLM-enriched descriptions (Right) form better-separated clusters compared to simple class labels (Left), facilitating more robust cross-modal alignment.

Methods	Kinetics-200 (Acc, %)
Methods	180/20 split	160/40 split	140/60 split	120/80 split
ReViSE Hubert Tsai et al. (2017b)	24.95	13.28	8.14	6.23
DeViSE Frome et al. (2013)	22.22	12.32	7.97	5.65
PURLS Zhu et al. (2024a)	32.22	22.56	12.01	11.75
TDSM Do and Kim (2025)	38.18	24.43	15.28	13.09
FDSM(Ours)	39.76	27.01	16.96	14.73

Table 3: Top-1 accuracy results of FDSM evaluated on the Kinetics-200 dataset under the PURLS Zhu et al. (2024a) benchmark.

Methods	Kinetics-400 (Acc, %)
Methods	360/40 split	320/80 split	300/100 split	280/120 split
ReViSE Hubert Tsai et al. (2017b)	20.84	11.82	9.49	8.23
DeViSE Frome et al. (2013)	18.37	10.23	9.47	8.34
PURLS Zhu et al. (2024a)	34.51	24.32	16.99	14.28
TDSM Do and Kim (2025)	38.92	26.24	18.45	16.10
FDSM(Ours)	40.25	28.17	20.13	17.88

Table 4: Top-1 accuracy results of FDSM evaluated on the Kinetics-400 dataset under the PURLS Zhu et al. (2024a) benchmark.

Methods	NTU-60 (Acc, %)			NTU-120 (Acc, %)
Methods	55/5	48/12	30/30	110/10	96/24	60/60
Full model	87.79	57.46	26.55	75.24	66.52	28.67
w.o. SG-SRM	86.52	56.11	25.42	74.18	65.13	27.53
w.o. $\mathcal{L}_{\text{Freq}}$	86.45	55.98	25.35	74.21	65.08	27.48
w.o. Curriculum	87.31	57.18	26.11	74.89	66.15	28.32

Table 5: Ablation study on the contribution of each component. SG-SRM: Semantic-Guided Spectral Residual Module;

\mathcal{L}_{\text{Freq}}

: Timestep-Adaptive Spectral Loss; Curriculum: Curriculum-based Semantic Abstraction. We report the performance degradation when removing each component from the full model.

Figure 6: DCT-based spectral energy of ST-GCN latent features along the temporal dimension, where

k

denotes the frequency index and

M{=}L/4

is the cutoff separating low- and high-frequency bands. All spectral profiles track closely in the low normalized-frequency band (

k/L<0.25

), confirming that all models capture global pose structure. Beyond the cutoff (

k/L\geq 0.25

), GT latents maintain substantial energy while both TDSM and FDSM without SG-SRM exhibit a sharp drop—this systematic gap is the Spectral Bias . FDSM (full model) recovers the suppressed high-frequency energy, closely tracking the GT curve.

Figure 7: Qualitative comparison of decoded latent trajectories predicted by FDSM without (top) and with (bottom) the Timestep-Adaptive Spectral Loss ( $\mathcal{L}_{\text{freq}}$ ) on a kicking sequence from NTU RGB+D 120. Without TASL, the predicted $\hat{z}_{0}$ trajectory collapses to an oversmoothed average, yielding nearly identical frames with minimal inter-frame variation. With TASL, the adaptive high-frequency supervision (Eq. 7 ) recovers richer micro-dynamics, including discriminative limb articulations that distinguish kinematically similar actions.

Gaussian noise $\bm{\epsilon}$	NTU-60 (Acc, %)			NTU-120 (Acc, %)
Gaussian noise $\bm{\epsilon}$	55/5	48/12	30/30	110/10	96/24	60/60
Fixed	77.68	45.52	19.82	65.18	53.47	18.55
Random(Ours)	87.79	57.46	26.55	75.24	66.52	28.67

Table 6: Ablation study on the effect of noise

\bm{\epsilon}

during training.

Cutoff $M$	NTU-60 (Acc, %)			NTU-120 (Acc, %)
Cutoff $M$	55/5	48/12	30/30	110/10	96/24	60/60
$L/8$	87.45	57.10	26.25	74.95	66.20	28.40
$L/4$ (Ours)	87.79	57.46	26.55	75.24	66.52	28.67
$L/2$	87.10	56.85	25.95	74.60	65.90	28.05

Table 7: Sensitivity to the frequency cutoff parameter

M

Schedule	NTU-60 (Acc, %)			NTU-120 (Acc, %)
Schedule	55/5	48/12	30/30	110/10	96/24	60/60
Fixed (0.5)	87.12	56.95	25.85	74.75	66.00	28.15
Step Decay	87.35	57.10	26.15	74.90	66.15	28.35
Linear Decay	87.55	57.25	26.35	75.05	66.30	28.50
Cosine (Ours)	87.79	57.46	26.55	75.24	66.52	28.67

Table 8: Ablation on curriculum schedules.

Gating Strategy	NTU-60 (Acc, %)			NTU-120 (Acc, %)
Gating Strategy	55/5	48/12	30/30	110/10	96/24	60/60
No Gain ( $s=0$ )	86.52	56.11	25.42	74.18	65.13	27.53
Random Gain	84.50	53.80	23.50	72.10	63.50	25.80
Uniform Gain ( $s=1$ )	85.80	55.10	24.85	73.40	64.80	26.90
Predicted (Ours)	87.79	57.46	26.55	75.24	66.52	28.67

Table 9: Ablation study on the semantic gating mechanism in SG-SRM.

Param	Value	NTU-60 (Acc, %)			NTU-120 (Acc, %)
Param	Value	55/5	48/12	30/30	110/10	96/24	60/60
$\lambda_{\text{freq}}$	0.5	87.15	56.80	25.90	74.65	65.95	28.05
	1.0 (Ours)	87.79	57.46	26.55	75.24	66.52	28.67
	5.0	86.30	55.50	25.10	73.80	64.80	27.20
$\alpha$	0.5	87.05	56.65	25.85	74.55	65.85	27.95
	1.0 (Ours)	87.79	57.46	26.55	75.24	66.52	28.67

Table 10: Sensitivity analysis of hyperparameters

\lambda_{\text{freq}}

and

\alpha

Method	Original	Temporal Scales (Cropping)		Sampling Rates (Downsampling)
Method	Original	$L=150$	$L=75$	1/2	1/4
TDSM Do and Kim (2025)	$86.49$	$82.70$	$75.45$	$83.60$	$78.15$
FDSM (Ours)	87.79	85.15	79.80	86.10	82.45

Table 11: Robustness to temporal scale and sampling rate on NTU-60 (55/5, Top-1 Acc, %).

Method	Params (M)	GFLOPs	Training	Inference	Accuracy (%)
Method	Params (M)	GFLOPs	Time	Time	55/5	80/40
TDSM Do and Kim (2025)	261.21	6.4545	6.7 h	0.43 s	86.49	36.95
FDSM (Ours)	261.21	6.4546	6.9 h	0.44 s	87.79 (+1.30)	39.16 (+2.21)

Table 12: Complexity and performance comparison between TDSM and FDSM. Training time is evaluated on a single NVIDIA L40S GPU for the NTU-60 55/5 split (50,000 iterations); inference time is measured per sequence.

Figure 8: Localized confusion matrix comparison between TDSM (Left) and our FDSM (Right) on selected NTU-120 unseen classes. FDSM significantly reduces the confusion between kinematically similar actions (e.g., “Clapping” vs. “Rubbing hands”) by recovering discriminative high-frequency details.

Figure 9: Success cases of FDSM on NTU-120 unseen classes (A027 “Jump up”, A050 “Punching/slapping other person”, A052 “Pushing other person”). Skeletons are overlaid on RGB frames using ground-truth Kinect-captured joint coordinates (colorX/colorY) for visualization; no pose estimator is applied. FDSM correctly classifies all three actions dominated by high-frequency micro-dynamics, while TDSM misclassifies them due to spectral smoothing.

Dataset	High Intensity (1)	Low Intensity (0)	Examples (High vs. Low)
NTU-60 Shahroudy et al. (2016)	53.3% (32)	46.7% (28)	Punching vs. Reading
NTU-120 Liu et al. (2019)	55.8% (67)	44.2% (53)	Butt kicks vs. Yawn
PKU-MMD Liu et al. (2017)	54.9% (28)	45.1% (23)	Kicking vs. Bow
Kinetics-200 Yan et al. (2018b)	58.0% (116)	42.0% (84)	High jump vs. Drinking
Kinetics-400 Kay et al. (2017)	58.2% (233)	41.8% (167)	Breakdancing vs. Dining

Table 13: Detailed distribution of LLM-derived binary motion intensity labels (

s_{y}^{GT}

) across the five evaluated benchmarks.

$t_{\text{test}}$ Value	10	20	25 (Ours)	30	40	50
NTU-60 (55/5)	84.62	86.95	87.79	86.31	82.54	72.40
NTU-120 (110/10)	72.15	74.42	75.24	73.95	70.62	61.35

Table 14: Zero-shot action recognition accuracy (%) under varying inference timesteps

t_{\text{test}}

on NTU-60 and NTU-120 datasets.

Methods	SG-SRM	$\mathcal{L}_{\text{Freq}}$	Curr.	NTU-60 (Acc, %)		NTU-120 (Acc, %)
Methods	SG-SRM	$\mathcal{L}_{\text{Freq}}$	Curr.	55/5	30/30	110/10	60/60
TDSM Do and Kim (2025)				86.49	25.88	74.15	27.21
+ SG-SRM	✓			87.03	26.02	74.61	28.16
+ $\mathcal{L}_{\text{Freq}}$ (TASL)	✓	✓		87.31	26.11	74.89	28.32
FDSM (Full)	✓	✓	✓	87.79	26.55	75.24	28.67

Table 15: Additive ablation study starting from the TDSM baseline. Each component addresses one distinct facet of the Spectral Bias, yielding cumulative performance gains.

Explore similar work

Visual Anchoring in Diffusion: Multimodal Zero-Shot Skeleton Action Recognition
Aug 5, 2026Zehao Bao, Shujun Guo, Bruce X. B. YuSkeleton-Based Action Recognition Diffusion Transformer
The University of Hong Kong Hong Kong SAR, China · Zhejiang University Hangzhou, China
GenPrior: Unleashing Text-to-Motion Generative Priors for Zero-Shot Skeleton-based Action Recognition
Aug 3, 2026Jidong Kuang, Hongsong Wang, Jie GuiSkeleton-Based Action Recognition Zero-Shot Learning
School of Cyber Science and Engineering, Southeast University Nanjing, China · School of Computer Science and Engineering, Southeast University Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications Nanjing, China · School of Cyber Science and Engineering, Southeast University Purple Mountain Laboratories Engineering Research Center of Blockchain Application, Supervision and Management Nanjing, China
SCALE: Semantic- and Confidence-Aware Conditional Variational Autoencoder for Zero-shot Skeleton-based Action Recognition
Apr 2, 2026Soroush Oraki, Feng Ding, Jie LiangSkeleton-Based Action Recognition Zero-Shot Learning
School of Engineering Science, Simon Fraser University, Burnaby, BC, Canada · School of Information Science and Technology, Eastern Institute of Technology, Ningbo, China

Aug 5, 2026J/K move · Enter open · S save

Visual Anchoring in Diffusion: Multimodal Zero-Shot Skeleton Action Recognition

Zehao Bao, Shujun Guo, Bruce X. B. Yu

The University of Hong Kong Hong Kong SAR, China · Zhejiang University Hangzhou, China

Skeleton-Based Action Recognition Diffusion Transformer Multimodal Diffusion Models

Open paper AskPDF arXiv

Zero-shot Skeleton Action Recognition (ZSAR) remains ambiguous when unseen actions share similar skeleton joint dynamics but differ in objects or scene context. RGB provides these missing cues, yet existing multimodal methods typically maintain independent skeleton and RGB scoring branches and fuse their outputs. Without using unlabeled test data for adaptation or fusion calibration, a fixed fusion weight cannot capture class-pair-dependent modality reliability, while an adaptive rule lacks target-side feedback for deciding which branch should dominate. We bypass this weight-selection problem via the classify-by-generation paradigm, where each class is scored by how accurately a text-conditioned denoiser predicts the noise added to the skeleton feature. This formulation separates the progressively corrupted skeleton from fixed conditioning, allowing RGB and text to jointly condition a single class-scoring function rather than produce independent scores. We instantiate this idea as Multimodal Triplet Diffusion for Skeleton-Text Matching (TDSM-MM), augmenting a text-conditioned denoising Transformer with a non-diffused RGB condition token that serves as a stable visual anchor during skeleton data reconstruction. Our proposed TDSM-MM has been ablated via extensive experiments and achieved the best inductive accuracy on three of four NTU-60/120 splits and surpasses the transductive state-of-the-art on NTU-120 96/24 (i.e., 71.3% vs. 69.1%), without test-time adaptation, suggesting that diffusion-based methods can be a promising direction for zero-shot learning.

Abstract

Explore similar work

Visual Anchoring in Diffusion: Multimodal Zero-Shot Skeleton Action Recognition

GenPrior: Unleashing Text-to-Motion Generative Priors for Zero-Shot Skeleton-based Action Recognition

SCALE: Semantic- and Confidence-Aware Conditional Variational Autoencoder for Zero-shot Skeleton-based Action Recognition