Geometry-Constrained Dynamic Hypergraph Convolutional Network with Contrastive Score Refinement for Skeleton-Based Action Recognition

Minh Ta Tran Nhat; Yuning Ye; Yuseok Ban

cs.CVNov 2, 2026

Geometry-Constrained Dynamic Hypergraph Convolutional Network with Contrastive Score Refinement for Skeleton-Based Action Recognition

Authors: Minh Ta Tran Nhat, Yuning Ye, Yuseok Ban

Organizations: Department of Electronics Engineering Chungbuk National University Cheongju, South Korea

Abstract

Skeleton-based action recognition is a pivotal task in computer vision; however, conventional Graph Convolutional Networks (GCNs) are often constrained by pairwise joint modeling, which fails to capture the intricate high-order interactions inherent in human movement. While recent hypergraph-based methods attempt to address this limitation, they typically rely on static structures derived from anatomical priors, lacking the flexibility to capture dynamic, action-specific joint correlations. To this end, we propose DyGCN, a Geometry-Constrained Dynamic Hypergraph Convolutional Network for skeleton-based action recognition. At the core of DyGCN is the Dynamic Hypergraph Construction Network with Contrastive Score Refinement (DyGCT) block, which comprises two novel components: the Geometry-Constrained Dynamic Hypergraph Construction (GeoHC) module and the Contrastive Score Refinement (CSR) module. Specifically, GeoHC dynamically constructs action-specific hyperedges by leveraging geometry-constrained second-order compatibility, transcending the limitations of static hyperedge structures. Furthermore, CSR explicitly refines the pairwise compatibility scores during hypergraph construction through prototype-based contrastive learning, fostering more discriminative action-specific hyperedges, with zero inference-time overhead as CSR operates only during training. Extensive experiments on NTU RGB+D, NTU RGB+D 120, and Northwestern-UCLA validate the effectiveness of DyGCN, achieving state-of-the-art performance on widely used benchmarks. The source code is available at https://github.com/marvineta0405/DyGCN.

Figures & tables

Figure 1: Overview of our proposed method. (1) The overall pipeline takes skeleton sequences $\mathbf{X}$ as input, followed by embedding, spatio-temporal modeling via stacked DyGCN and MS-TC [ 5 ] modules, and global average pooling with Fully Connected Layers (FC) for action classification. (2) The detailed architecture of the Dynamic Hypergraph Convolutional Network (DyGCN)

Figure 2: The framework of the DyGCT block, comprising the Geometry-Constrained Dynamic Hypergraph Construction (GeoHC) module and the Contrastive Score Refinement (CSR) module.

Types	Methods	Publication	NTU RGB+D		NTU RGB+D 120		NW-UCLA
Types	Methods	Publication	X-Sub(%)	X-View(%)	X-Sub(%)	X-Set(%)	Top-1(%)
GCN-based	ST-GCN [ 39 ]	AAAI2018	81.5	88.3	70.7	73.2	-
	2s-AGCN [ 29 ]	CVPR2019	88.5	95.1	82.5	84.2	-
	CTR-GCN [ 4 ]	ICCV2021	92.4	96.8	88.9	90.6	96.5
	EfficientGCN-B4 [ 32 ]	TPAMI2022	91.7	95.7	88.3	89.1	97.0
	InfoGCN [ 7 ]	CVPR2022	93.0	97.1	89.8	91.2	97.0

Table 1: Comparison with state-of-the-art methods on NTU RGB+D [ 28 ] , NTU RGB+D 120 [ 22 ] , and NW-UCLA [ 35 ] . * denotes methods using additional supervision signals or non-standard input data.

Method	DyGCT			Params (M)	GFLOPs (G)	Acc (%)
Method	TPE	CM	CSR	Params (M)	GFLOPs (G)	Acc (%)
Baseline	–	–	–	1.5	1.97	89.8
DyGCN	–	$\checkmark$	–	2.24	2.7	91.2 ( $\uparrow$ 1.4)
	$\checkmark$	$\checkmark$	–	2.28	2.7	91.7 ( $\uparrow$ 1.9)
	$\checkmark$	$\checkmark$	$\checkmark$	2.3	2.7	92.0 ( $\uparrow$ 2.2)

Table 2: Ablation study on the contribution of each component in DyGCT under the NTU RGB+D X-Sub setting with the joint input modality only.

Methods	$K_{1}$	w/ CSR	w/o CSR
DyGCT	3	91.8	91.5
	5	92.0	91.7
	7	91.5	91.2
	9	91.3	91.1

Table 3: The impact of the hyperparameter

K_{1}

in DyGCT (left) and Compatibility Measuring Component

\mathbf{C}_{\text{triad}}

and

\mathbf{C}_{\text{edge}}

(right) under NTU RGB+D X-Sub setting.

	CTR-GCN [ 4 ]	FR-Head [ 42 ]	HD-GCN [ 18 ]	Ours
NTU60-Hard	62.1	67.5	67.4	$\textbf{71.1}^{\text{+3.6}}$
NTU60-Medium	83.8	83.9	84.9	$\textbf{87.9}^{\text{+3.0}}$
NTU60-Easy	95.2	95.8	95.5	$\textbf{96.1}^{\text{+0.6}}$

Table 4: Performance comparison of DyGCN variants on ambiguous actions.

Panel of Figure 3 — Figure 3: Left: Ablation study on the influence of weight $\gamma$ (upper) and memory capacity $M_{p}$ (lower) under the NTU X-Sub setting. Right: Visualization of the learned hyperedges across 8 frames for four different action categories.

Panel of Figure 4 — Figure 4: t-SNE visualization of the output features from the Baseline and DyGCN. Different colors denote different action categories.

Skeleton-Based Action Recognition

Skeleton-based action recognition identifies human activities from sequences of body-joint positions, focusing on motion while limiting reliance on appearance.

See all 54 papers

NeuroPath: Brain-Inspired Dual-Pathway Graph Convolutional Networks for Skeleton-Based Action Recognition
Aug 18, 2026Kanglei Zhou, Ruizhi Cai, Hubert P. H. Shum +2Skeleton-Based Action Recognition Spatiotemporal GNNs
Department of Psychological and Cognitive Sciences, Tsinghua University, No. 30 Shuangqing Road, Haidian District, Beijing, 100084, China · State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, No. 37 Xueyuan Road, Haidian District, Beijing, 100191, China · Department of Computer Science, Durham University, Stockton Rd, Durham, DH1 3LE, United Kingdom +1
KGS-GCN: Kinematics-Driven Gaussian Splatting and Probabilistic Topology for Skeleton-Based Action Recognition
Mar 16, 2026Yuhan Chen, Yicui Shi, Guofa Li +4Skeleton-Based Action Recognition Human Activity Recognition
College of Mechanical and Vehicle Engineering, Chongqing University, Chongqing 400044, China · Department of Mathematical Sciences, Tsinghua University, Beijing 100084, China · School of Vehicle and Mobility, Tsinghua University, Beijing 100084, China +1
GCN-DevLSTM: Path Development for Skeleton-Based Action Recognition
Mar 22, 2024Lei Jiang, Weixin Yang, Xin Zhang +1Skeleton-Based Action Recognition Human Activity Recognition
University College London · University of Oxford · South China University of Technology

Aug 18, 2026J/K move · Enter open · S save

NeuroPath: Brain-Inspired Dual-Pathway Graph Convolutional Networks for Skeleton-Based Action Recognition

Kanglei Zhou, Ruizhi Cai, Hubert P. H. Shum, Frederick W. B. Li+1

Department of Psychological and Cognitive Sciences, Tsinghua University, No. 30 Shuangqing Road, Haidian District, Beijing, 100084, China · State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, No. 37 Xueyuan Road, Haidian District, Beijing, 100191, China · Department of Computer Science, Durham University, Stockton Rd, Durham, DH1 3LE, United Kingdom+1

Skeleton-Based Action Recognition Spatiotemporal GNNs

Open paper AskPDF arXiv

Skeleton-based action recognition aims to recognize human actions from sequences of human joint coordinates. Most existing Spatial-Temporal Graph Convolutional Networks (STGCNs) have achieved promising results by modeling skeletal structures with implicit spatial-temporal representations. However, our empirical study reveals a clear performance imbalance across different skeletal modalities, indicating that implicitly coupling spatial and temporal information limits the full exploitation of complementary structural and motion cues. Inspired by the ventral and dorsal pathways in human perception, we propose Dual-Pathway Graph Convolutional Networks (NeuroPath), which adopt a dual-pathway architecture for separate yet collaborative modeling of spatial and temporal information. Specifically, transformation units first convert the input into pathway-specific skeletal representations, allowing each pathway to focus on complementary aspects of human motion. To further capture coordinated joint behaviors and their interrelationships, we introduce a group graph convolution block that dynamically identifies key body parts and models their spatial-temporal dependencies. In addition, inter-pathway dynamic fusion modules integrate complementary inter-modal information across pathways, facilitating higher-level semantic interpretation of actions. Extensive experiments on Kinetics Skeleton 400, NTU RGB+D 60, and NTU RGB+D 120 demonstrate consistent performance improvements, validating the effectiveness of dual-pathway spatial-temporal modeling for skeleton-based action recognition.

Abstract

Skeleton-Based Action Recognition

Explore similar work

NeuroPath: Brain-Inspired Dual-Pathway Graph Convolutional Networks for Skeleton-Based Action Recognition

KGS-GCN: Kinematics-Driven Gaussian Splatting and Probabilistic Topology for Skeleton-Based Action Recognition

GCN-DevLSTM: Path Development for Skeleton-Based Action Recognition