LLM Enhanced Action Recognition via Hierarchical Global-Local Skeleton-Language Model

Ruosi Wang; Fangwei Zuo; Lei Li; Zhaoqiang Xia

cs.CVMar 28, 2026

LLM Enhanced Action Recognition via Hierarchical Global-Local Skeleton-Language Model

Authors: Ruosi Wang, Fangwei Zuo, Lei Li, Zhaoqiang Xia

Organizations: Northwestern Polytechnical University, No.1 Dongxiang Road, Chang’an District , Xi’an, 710129, Shaanxi, China · Shandong University of Finance and Economics, No.40 Shungeng Road, Licheng District, Jinan, Jinan, Shandong, China

Abstract

Skeleton-based human action recognition has achieved remarkable progress in recent years. However, most existing GCN-based methods rely on short-range motion topologies, which not only struggle to capture long-range joint dependencies and complex temporal dynamics but also limit cross-modal semantic alignment and understanding due to insufficient modeling of action semantics. To address these challenges, we propose a hierarchical global-local skeleton-language model (HocSLM), enabling the large action model be more representative of action semantics. First, we design a hierarchical global-local network (HGLNet) that consists of a composite-topology spatial module and a dual-path hierarchical temporal module. By synergistically integrating multi-level global and local modules, HGLNet achieves dynamically collaborative modeling at both global and local scales while preserving prior knowledge of human physical structure, significantly enhancing the model's representation of complex spatio-temporal relationships. Then, a large vision-language model (VLM) is employed to generate textual descriptions by passing the original RGB video sequences to this model, providing the rich action semantics for further training the skeleton-language model. Furthermore, we introduce a skeleton-language sequential fusion module by combining the features from HGLNet and the generated descriptions, which utilizes a skeleton-language model (SLM) for aligning skeletal spatio-temporal features and textual action descriptions precisely within a unified semantic space. The SLM model could significantly enhance the HGLNet's semantic discrimination capabilities and cross-modal understanding abilities. Extensive experiments demonstrate that the proposed HocSLM achieves the state-of-the-art performance on three mainstream benchmark datasets: NTU RGB+D 60, NTU RGB+D 120, and Northwestern-UCLA.

Figures & tables

Figure 1: The architecture of the proposed HocSLM, including 1) Skeleton Component, 2) Language Component, and 3) Fusion Component.

Figure 2: Overall architecture of the CTS. The leftmost module is the feature preprocessing block. The upper orange region corresponds to the GSM, and the lower green region corresponds to the LSE.

Figure 3: Overall architecture of the DHT. The yellow region on the left half corresponds to the GTA, and the purple region on the right half corresponds to the LTC.

Figure 4: The overall architecture of SSF.

Figure 5: The overall architecture of 6-stream ensemble.

Method	X-Sub Acc (%)	X-View Acc (%)
ST-GCN Yan et al. (2018)	81.5	88.3
AS-GCN Li et al. (2019)	86.8	94.2
2s-AGCN Shi et al. (2019c)	88.5	95.1
SGN Zhang et al. (2020a)	89.0	94.5
AGC-LSTM Si et al. (2019)	89.2	95.0
DGNN Shi et al. (2019b)	89.9	96.1

Table 1: Comparison of different models on the NTU-RGB+D dataset.

Method	X-Sub Acc (%)	X-View Acc (%)
ST-GCN Yan et al. (2018)	70.7	73.2
2s-AGCN Shi et al. (2019c)	82.5	84.2
SGN Zhang et al. (2020a)	79.2	81.5
ST-TR-agcn Plizzari et al. (2021)	85.1	87.1
Shift-GCN Cheng et al. (2020b)	85.9	87.6
DC-GCN + ADG Cheng et al. (2020a)	86.5	88.1

Table 2: Comparison of different models on the NTU-RGB+D120 dataset.

Method	Acc (%)
Shift-GCN Cheng et al. (2020b)	94.6
GSTLN Dai et al. (2023)	94.8
CTR-GCN Chen et al. (2021a)	96.5
Info-GCN Chi et al. (2022)	96.6
MSS-GCN Jang et al. (2024)	96.6
SAN-GCN Tian et al. (2023)	96.8

Table 3: Comparison of different models on the Northwestern-UCLA dataset.

Configuration	CTS	DHT	SSF	Top-1 (%)	Performance
(Baseline + Components)					enhancement(%)
Baseline (CTR-GCN)	-	-	-	89.96	-
Baseline + CTS	$\checkmark$	-	-	90.31	0.35
Baseline + CTS + DHT	$\checkmark$	$\checkmark$	-	90.63	0.67
HocSLM	$\checkmark$	$\checkmark$	$\checkmark$	90.84	0.88

Table 4: Fine-grained Ablation on Different Core Components.

Configuration	CTS Module	DHT Module	Top-1 (%)
Baseline (CTR-GCN)	-	-	89.96
Only Local	Only LSE	Only LTC	89.71
w/o GSM	Only LSE	LTC + GTA	90.20
w/o GTA	LSE + GSM	Only LTC	90.37
HGLNet	LSE + GSM	LTC + GTA	90.63

Table 5: Fine-grained ablation on HGLNet components.

Strategy	Skeleton Encoder	MLP	LLM	Training Loss	Top-1 (%)
T0 (Baseline)	Trained (HGLNet)	–	–	$\mathcal{L}_{\text{cls}}$	90.63
T1	Frozen $\to$ Unfrozen	Trained	Frozen	Stage1: $\mathcal{L}_{\text{gen}}$ only	89.08
				Stage2: $\mathcal{L}_{\text{cls}}$ only
T2	Trained	Trained	Frozen	$\mathcal{L}_{\text{gen}}\rightarrow\mathcal{L}_{\text{cls}}$	90.73
T3 (Ours)	Trained	Trained	Frozen	$\mathcal{L}_{\text{gen}}+\mathcal{L}_{\text{cls}}$	90.84
T4	Trained from scratch	Trained	Frozen	$\mathcal{L}_{\text{gen}}+\mathcal{L}_{\text{cls}}$	68.23

Table 6: Ablation on training paradigms and effectiveness conditions of SSF.

Streams	Method	Top-1 (%)
1s	Joint only	90.84
	Bone only	91.51
	Joint Motion only	89.15
	Bone Motion only	88.72
	Joint 2nd-order only	91.42
	Vel 2nd-order only	88.87

Table 7: Ablation on multi-stream fusion strategies.

Figure 6: Visualization of skeleton diagrams and joint-Joint attention matrices for six typical actions.

Figure 7: The generated text examples by Qwen3-4B based HocSLM.

Explore similar work

SkelHCC: A Hyperbolic CLIP-Driven Cache Adaptation Framework for Skeleton-based One-Shot Action Recognition
Jun 2, 2026Yanan Liu, Anqi Zhu, Jingmin Zhu +6Skeleton-Based Action Recognition Cross-Modal Representation Learning
Yunnan University, Kunming, China · Monash University · Lancaster University +1
Towards Universal Skeleton-Based Action Recognition
Apr 18, 2026Jidong Kuang, Hongsong Wang, Jie GuiSkeleton-Based Action Recognition Human Activity Recognition
School of Cyber Science and Engineering, Southeast University, Nanjing, China · School of Computer Science and Engineering, Southeast University, Nanjing, China
T-MOR: Learning Motion-Aware Skeleton Representations for Human Action Recognition
Jun 19, 2026Di Yang, Mahmoud Ali, Quan Kong +2Skeleton-Based Action Recognition Representation Learning
Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, China · Inria Center at Universit´e Cˆote d’Azur, Valbonne, France · Woven by Toyota, Tokyo, Japan +1

Jun 2, 2026J/K move · Enter open · S save

SkelHCC: A Hyperbolic CLIP-Driven Cache Adaptation Framework for Skeleton-based One-Shot Action Recognition

Yanan Liu, Anqi Zhu, Jingmin Zhu, Jun Liu+5

Yunnan University, Kunming, China · Monash University · Lancaster University+1

Skeleton-Based Action Recognition Cross-Modal Representation Learning Hierarchical Representation Learning Few-Shot Learning Hyperbolic Representation Learning

Open paper AskPDF arXiv

Skeleton-based action recognition aims to understand human behaviors from body joint sequences and is especially challenging in the one-shot setting, where only a single labeled exemplar is available for each novel action. A key challenge is learning representations that capture the hierarchical and compositional structure of human motion while aligning effectively with high-level action semantics under extreme data scarcity. Existing approaches, largely based on Euclidean embeddings and low-level motion cues, struggle to model the tree-like organization of skeleton data, limiting cross-modal alignment and generalization to unseen action categories. We propose SkelHCC, a unified skeleton hyperbolic CLIP-driven cache adaptation framework for one-shot skeleton-based action recognition. SkelHCC introduces an Explicitly Hierarchical Hyperbolic CLIP (EH-HCLIP) module that embeds skeleton sequences and action language into a shared hyperbolic space. By leveraging the negative curvature and exponential volume growth of hyperbolic geometry, EH-HCLIP naturally encodes the joint-part-body hierarchy of human anatomy and yields structurally consistent cross-modal representations. To support efficient one-shot adaptation, SkelHCC further integrates a training-free LLM-guided Multi-granularity Voting Cache (LMV-Cache) for context-aware inference. Experiments on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD demonstrate that SkelHCC consistently outperforms state-of-the-art methods.

Abstract

Explore similar work

SkelHCC: A Hyperbolic CLIP-Driven Cache Adaptation Framework for Skeleton-based One-Shot Action Recognition

Towards Universal Skeleton-Based Action Recognition

T-MOR: Learning Motion-Aware Skeleton Representations for Human Action Recognition