LLM Enhanced Action Recognition via Hierarchical Global-Local Skeleton-Language Model

Ruosi Wang; Fangwei Zuo; Lei Li; Zhaoqiang Xia

cs.CVMar 28, 2026

LLM Enhanced Action Recognition via Hierarchical Global-Local Skeleton-Language Model

Authors: Ruosi Wang, Fangwei Zuo, Lei Li, Zhaoqiang Xia

Organizations: Northwestern Polytechnical University, No.1 Dongxiang Road, Chang’an District , Xi’an, 710129, Shaanxi, China · Shandong University of Finance and Economics, No.40 Shungeng Road, Licheng District, Jinan, Jinan, Shandong, China

Abstract

Skeleton-based human action recognition has achieved remarkable progress in recent years. However, most existing GCN-based methods rely on short-range motion topologies, which not only struggle to capture long-range joint dependencies and complex temporal dynamics but also limit cross-modal semantic alignment and understanding due to insufficient modeling of action semantics. To address these challenges, we propose a hierarchical global-local skeleton-language model (HocSLM), enabling the large action model be more representative of action semantics. First, we design a hierarchical global-local network (HGLNet) that consists of a composite-topology spatial module and a dual-path hierarchical temporal module. By synergistically integrating multi-level global and local modules, HGLNet achieves dynamically collaborative modeling at both global and local scales while preserving prior knowledge of human physical structure, significantly enhancing the model's representation of complex spatio-temporal relationships. Then, a large vision-language model (VLM) is employed to generate textual descriptions by passing the original RGB video sequences to this model, providing the rich action semantics for further training the skeleton-language model. Furthermore, we introduce a skeleton-language sequential fusion module by combining the features from HGLNet and the generated descriptions, which utilizes a skeleton-language model (SLM) for aligning skeletal spatio-temporal features and textual action descriptions precisely within a unified semantic space. The SLM model could significantly enhance the HGLNet's semantic discrimination capabilities and cross-modal understanding abilities. Extensive experiments demonstrate that the proposed HocSLM achieves the state-of-the-art performance on three mainstream benchmark datasets: NTU RGB+D 60, NTU RGB+D 120, and Northwestern-UCLA.

Figures & tables

Figure 1: The architecture of the proposed HocSLM, including 1) Skeleton Component, 2) Language Component, and 3) Fusion Component.

Figure 2: Overall architecture of the CTS. The leftmost module is the feature preprocessing block. The upper orange region corresponds to the GSM, and the lower green region corresponds to the LSE.

Figure 3: Overall architecture of the DHT. The yellow region on the left half corresponds to the GTA, and the purple region on the right half corresponds to the LTC.

Figure 4: The overall architecture of SSF.

Figure 5: The overall architecture of 6-stream ensemble.

Method	X-Sub Acc (%)	X-View Acc (%)
ST-GCN Yan et al. (2018)	81.5	88.3
AS-GCN Li et al. (2019)	86.8	94.2
2s-AGCN Shi et al. (2019c)	88.5	95.1
SGN Zhang et al. (2020a)	89.0	94.5
AGC-LSTM Si et al. (2019)	89.2	95.0
DGNN Shi et al. (2019b)	89.9	96.1

Table 1: Comparison of different models on the NTU-RGB+D dataset.

Method	X-Sub Acc (%)	X-View Acc (%)
ST-GCN Yan et al. (2018)	70.7	73.2
2s-AGCN Shi et al. (2019c)	82.5	84.2
SGN Zhang et al. (2020a)	79.2	81.5
ST-TR-agcn Plizzari et al. (2021)	85.1	87.1
Shift-GCN Cheng et al. (2020b)	85.9	87.6
DC-GCN + ADG Cheng et al. (2020a)	86.5	88.1

Table 2: Comparison of different models on the NTU-RGB+D120 dataset.

Method	Acc (%)
Shift-GCN Cheng et al. (2020b)	94.6
GSTLN Dai et al. (2023)	94.8
CTR-GCN Chen et al. (2021a)	96.5
Info-GCN Chi et al. (2022)	96.6
MSS-GCN Jang et al. (2024)	96.6
SAN-GCN Tian et al. (2023)	96.8

Table 3: Comparison of different models on the Northwestern-UCLA dataset.

Configuration	CTS	DHT	SSF	Top-1 (%)	Performance
(Baseline + Components)					enhancement(%)
Baseline (CTR-GCN)	-	-	-	89.96	-
Baseline + CTS	$\checkmark$	-	-	90.31	0.35
Baseline + CTS + DHT	$\checkmark$	$\checkmark$	-	90.63	0.67
HocSLM	$\checkmark$	$\checkmark$	$\checkmark$	90.84	0.88

Table 4: Fine-grained Ablation on Different Core Components.

Configuration	CTS Module	DHT Module	Top-1 (%)
Baseline (CTR-GCN)	-	-	89.96
Only Local	Only LSE	Only LTC	89.71
w/o GSM	Only LSE	LTC + GTA	90.20
w/o GTA	LSE + GSM	Only LTC	90.37
HGLNet	LSE + GSM	LTC + GTA	90.63

Table 5: Fine-grained ablation on HGLNet components.

Strategy	Skeleton Encoder	MLP	LLM	Training Loss	Top-1 (%)
T0 (Baseline)	Trained (HGLNet)	–	–	$\mathcal{L}_{\text{cls}}$	90.63
T1	Frozen $\to$ Unfrozen	Trained	Frozen	Stage1: $\mathcal{L}_{\text{gen}}$ only	89.08
				Stage2: $\mathcal{L}_{\text{cls}}$ only
T2	Trained	Trained	Frozen	$\mathcal{L}_{\text{gen}}\rightarrow\mathcal{L}_{\text{cls}}$	90.73
T3 (Ours)	Trained	Trained	Frozen	$\mathcal{L}_{\text{gen}}+\mathcal{L}_{\text{cls}}$	90.84
T4	Trained from scratch	Trained	Frozen	$\mathcal{L}_{\text{gen}}+\mathcal{L}_{\text{cls}}$	68.23

Table 6: Ablation on training paradigms and effectiveness conditions of SSF.

Streams	Method	Top-1 (%)
1s	Joint only	90.84
	Bone only	91.51
	Joint Motion only	89.15
	Bone Motion only	88.72
	Joint 2nd-order only	91.42
	Vel 2nd-order only	88.87

Table 7: Ablation on multi-stream fusion strategies.

Figure 6: Visualization of skeleton diagrams and joint-Joint attention matrices for six typical actions.

Figure 7: The generated text examples by Qwen3-4B based HocSLM.

Explore similar work

Jun 2, 2026

SkelHCC: A Hyperbolic CLIP-Driven Cache Adaptation Framework for Skeleton-based One-Shot Action Recognition

Skeleton-based action recognition aims to understand human behaviors from body joint sequences and is especially challenging in the one-shot setting, where only a single labeled exemplar is available for each novel action. A key challenge is learning representations that capture the hierarchical and compositional structure of human motion while aligning effectively with high-level action semantics under extreme data scarcity. Existing approaches, largely based on Euclidean embeddings and low-level motion cues, struggle to model the tree-like organization of skeleton data, limiting cross-modal alignment and generalization to unseen action categories. We propose SkelHCC, a unified skeleton hyperbolic CLIP-driven cache adaptation framework for one-shot skeleton-based action recognition. SkelHCC introduces an Explicitly Hierarchical Hyperbolic CLIP (EH-HCLIP) module that embeds skeleton sequences and action language into a shared hyperbolic space. By leveraging the negative curvature and exponential volume growth of hyperbolic geometry, EH-HCLIP naturally encodes the joint-part-body hierarchy of human anatomy and yields structurally consistent cross-modal representations. To support efficient one-shot adaptation, SkelHCC further integrates a training-free LLM-guided Multi-granularity Voting Cache (LMV-Cache) for context-aware inference. Experiments on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD demonstrate that SkelHCC consistently outperforms state-of-the-art methods.

Yanan Liu, Anqi Zhu, Jingmin Zhu +6

Yunnan University, Kunming, China · Monash University · Lancaster University +1

Skeleton-Based Action Recognition Cross-Modal Representation Learning Hierarchical Representation Learning Few-Shot Learning Hyperbolic Representation Learning

Ask PDF

Apr 18, 2026

Towards Universal Skeleton-Based Action Recognition

With the development of robotics, skeleton-based action recognition has become increasingly important, as human-robot interaction requires understanding the actions of humans and humanoid robots. Due to different sources of human skeletons and structures of humanoid robots, skeleton data naturally exhibit heterogeneity. However, previous works overlook the data heterogeneity of skeletons and solely construct models using homogeneous skeletons. Moreover, open-vocabulary action recognition is also essential for real-world applications. To this end, this work studies the challenging problem of heterogeneous skeleton-based action recognition with open vocabularies. We construct a large-scale Heterogeneous Open-Vocabulary (HOV) Skeleton dataset by integrating and refining multiple representative large-scale skeleton-based action datasets. To address universal skeleton-based action recognition, we propose a Transformer-based model that comprises three key components: unified skeleton representation, motion encoder for skeletons, and multi-grained motion-text alignment. The motion encoder feeds multi-modal skeleton embeddings into a two-stream Transformer-based encoder to learn spatio-temporal action representations, which are then mapped to a semantic space to align with text embeddings. Multi-grained motion-text alignment incorporates contrastive learning at three levels: global instance alignment, stream-specific alignment, and fine-grained alignment. Extensive experiments on popular benchmarks with heterogeneous skeleton data demonstrate both the effectiveness and the generalization ability of the proposed method. Code is available at https://github.com/jidongkuang/Universal-Skeleton.

Jidong Kuang, Hongsong Wang, Jie Gui

School of Cyber Science and Engineering, Southeast University, Nanjing, China · School of Computer Science and Engineering, Southeast University, Nanjing, China

Skeleton-Based Action Recognition Human Activity Recognition

Ask PDF

Jun 19, 2026

T-MOR: Learning Motion-Aware Skeleton Representations for Human Action Recognition

Vision-language models such as CLIP have recently achieved strong performance on a wide range of visual understanding tasks. However, most existing models rely primarily on appearance-level supervision from images or videos, and do not explicitly model human motion, which is essential for fine-grained and human-centric action recognition task as actions are defined by temporally structured and physically grounded body movements. To address this problem, we propose Transferable skeleton MOtion Representation (T-MOR), a motion-aware framework that learns transferable action representations from skeleton sequences with the aid of video and language supervision during training. T-MOR adopts a multi-modal contrastive learning scheme that aligns skeleton motion with visual and textual representations, while performing inference using only lightweight skeleton inputs. To support large-scale pre-training, we construct PoseCap-1M, a new dataset that contains over one million synchronized video, skeleton, and text triplets covering diverse human activities. We evaluate T-MOR on a range of human-centric action recognition benchmarks, including action classification and frame-wise temporal detection. Experimental results show that T-MOR consistently improves performance across multiple datasets, such as Toyota Smarthome, Penn Action, UAV-Human, TSU, and Charades. In addition, T-MOR demonstrates strong generalization ability in few-shot and zero-shot settings, highlighting the effectiveness of motion-centric and embodied representations for transferable action understanding.

Di Yang, Mahmoud Ali, Quan Kong +2

Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, China · Inria Center at Universit´e Cˆote d’Azur, Valbonne, France · Woven by Toyota, Tokyo, Japan +1

Skeleton-Based Action Recognition Representation Learning Cross-Modal Representation Learning Motion Representation Learning Temporal Action Detection Temporal Video Understanding

Ask PDF