NeuroPath: Brain-Inspired Dual-Pathway Graph Convolutional Networks for Skeleton-Based Action Recognition

Kanglei Zhou; Ruizhi Cai; Hubert P. H. Shum; Frederick W. B. Li; Xiaohui Liang

cs.CVAug 18, 2026

NeuroPath: Brain-Inspired Dual-Pathway Graph Convolutional Networks for Skeleton-Based Action Recognition

Authors: Kanglei Zhou, Ruizhi Cai, Hubert P. H. Shum, Frederick W. B. Li, Xiaohui Liang

Organizations: Department of Psychological and Cognitive Sciences, Tsinghua University, No. 30 Shuangqing Road, Haidian District, Beijing, 100084, China · State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, No. 37 Xueyuan Road, Haidian District, Beijing, 100191, China · Department of Computer Science, Durham University, Stockton Rd, Durham, DH1 3LE, United Kingdom · Zhongguancun Laboratory, Beijing, China

Abstract

Skeleton-based action recognition aims to recognize human actions from sequences of human joint coordinates. Most existing Spatial-Temporal Graph Convolutional Networks (STGCNs) have achieved promising results by modeling skeletal structures with implicit spatial-temporal representations. However, our empirical study reveals a clear performance imbalance across different skeletal modalities, indicating that implicitly coupling spatial and temporal information limits the full exploitation of complementary structural and motion cues. Inspired by the ventral and dorsal pathways in human perception, we propose Dual-Pathway Graph Convolutional Networks (NeuroPath), which adopt a dual-pathway architecture for separate yet collaborative modeling of spatial and temporal information. Specifically, transformation units first convert the input into pathway-specific skeletal representations, allowing each pathway to focus on complementary aspects of human motion. To further capture coordinated joint behaviors and their interrelationships, we introduce a group graph convolution block that dynamically identifies key body parts and models their spatial-temporal dependencies. In addition, inter-pathway dynamic fusion modules integrate complementary inter-modal information across pathways, facilitating higher-level semantic interpretation of actions. Extensive experiments on Kinetics Skeleton 400, NTU RGB+D 60, and NTU RGB+D 120 demonstrate consistent performance improvements, validating the effectiveness of dual-pathway spatial-temporal modeling for skeleton-based action recognition.

Figures & tables

Figure 1: The bubble plot of model computational performance in the X-Sub setting on the NTU RGB+D 60 dataset. The horizontal axis represents computational operations (GFLOPs), the vertical axis represents accuracy (%), and the size of the bubbles represents the number of parameters (M). Different bubble colors represent different kinds of models.

Panel of Figure 2 — Figure 2: Motivation and empirical observations. (a) Common skeletal modality representations derived from joint sequences, including joint positions, bone vectors, joint motion, and bone motion. (b) Inspiration from the two-stream hypothesis of the human visual system, where spatial structure and motion are processed by two parallel yet interacting pathways. (c) Empirical results of CTR-GCN [ 5 ] on the X-Sub protocol of NTU RGB+D 120 under different skeletal modalities. We report Top-1 Accuracy (denoted as Accuracy). Single-stream results exhibit clear performance imbalance across modalities, while combining structural and motion information consistently improves performance, highlighting their complementarity. (d) Results of our method, which significantly alleviates modality imbalance and achieves more balanced performance across different skeletal modalities.

Figure 3: Overview of the proposed NeuroPath framework. NeuroPath adopts a brain-inspired dual-pathway architecture to explicitly model the complementary spatial-aware and temporal-aware information in skeletal data. The two pathways are conceptually motivated by the ventral and dorsal streams in the human visual system, which specialize in spatial structure and motion processing, respectively. In contrast to existing multi-stream networks that fuse modalities only at the output level, NeuroPath introduces dynamic inter-pathway fusion modules to enable effective interaction and coordination between spatial-aware and temporal-aware representations throughout the network.

Figure 4: Core design intuition of STGGCB . Discriminative actions are characterized by coordinated groups of joints with correlated temporal dynamics. STGGCB first summarizes local temporal variation at the group level and then conditions spatial aggregation on this motion-aware context, enabling graph convolution to emphasize structurally and temporally relevant joint interactions.

Panel of Figure 5 — Figure 5: The network architecture of STGGCB : (a) Group Graph Convolution (G-GC) and (b) Multi-Scale Temporal Convolution (MS-TC). G-GC comprises group aggregation (see Eq. 4 ) and adaptive structural aggregation (see Eq. 5 ).

Panel of Figure 6 — Figure 6: Illustration of multi-hop aggregation strategies on a simple chain graph. (a) A 7-node chain graph, where node 4 is selected as the center node and 1-hop, 2-hop, and 3-hop relations are shown by different colors. (b) Exact $k$ -hop adjacency supports $\tilde{A}_{k}$ , where each matrix contains only node pairs whose shortest-path distance is exactly $k$ . (c) Conventional baselines using cumulative multi-hop aggregation ( $A^{1}+A^{2}+A^{3}$ ), which entangle different hop distances within a single representation. (d) Our method, which applies learnable masks $M_{k}$ over exact $k$ -hop supports, enabling adaptive reweighting while preserving explicit separation of neighborhood ranges.

Figure 7: The illustration of the spatial-temporal attention module. ‘

\cdot

’ and ‘

\times

’ represent matrix dot-product and matrix multiplication operations, respectively.

Figure 8: A comparison of STGCN network architectures: (a) our dual-pathway with dynamic fusion, (b) single pathway, (c) dual-pathway with hard fusion.

Ensemble stream	Method	Publication	Model type	Kinetics Skeleton 400		NTU RGB+D 60		NTU RGB+D 120
Ensemble stream	Method	Publication	Model type	Top-1	Top-5	X-Sub	X-View	X-Sub	X-Setup
Joint- only	ST-GCN [ 36 ]	AAAI 2018	(2+1)D	30.7	52.8	81.5	88.3	70.7	73.2
	AGCN [ 25 ]	CVPR 2019	(2+1)D	34.8	56.5	86.8	94.2	77.9	78.5
	Shift-GCN [ 7 ]	CVPR 2020	(2+1)D	34.7	56.8	87.8	95.1	80.9	83.2
	CTR-GCN [ 5 ]	ICCV 2021	(2+1)D	36.5	58.5	89.0	94.8	83.6	84.5
	InfoGCN [ 8 ]	CVPR 2022	(2+1)D	36.7	58.7	90.4	95.0	84.1	85.2

Table 1: Accuracy (%) comparison. The symbol “–” denotes unreported results. Underlined values indicate results taken from prior work [ 11 ] . Bold values indicate the best result in each column.

Configuration	Param. (M)	Acc. (%)
Baseline w/o Group	2.51	88.7
Baseline w/ Overlapping	2.80 ↑0.29	90.8 ↑2.1
Baseline w/ Non-Overlapping	2.80 ↑0.29	90.2 ↑1.5

Table 2: Accuracy (%) with various grouping strategies. The changes are relative to the first row.

Configuration	Window Size			Param. (M)	Acc. (%)
Configuration	B1	B2	B3	Param. (M)	Acc. (%)
w/o Group	✗	✗	✗	2.51	88.7
w/ Group	✓	✗	✗	2.65 ↑0.14	90.4 ↑1.7
	✓	✓	✗	2.80 ↑0.29	90.8 ↑2.1
	✓	✓	✓	3.36 ↑0.85	90.4 ↑1.7
	✗	✓	✗	2.66 ↑0.15	90.1 ↑1.4

Table 3: Accuracy (%) with various group configurations. The changes are relative to the first row.

Configuration	Param. (M)	Acc. (%)
AGCN (Js) [ 26 ]	3.47	87.4
AGCN (Js) w/ $\mathbf{M}_{\mathrm{att}}$	3.87 ↑0.40	88.2 ↑0.8
AGCN (Js) w/ Group	4.45 ↑0.98	89.1 ↑1.7
AGCN (Js) w/ Group + $\mathbf{M}_{\mathrm{att}}$	4.85 ↑1.38	89.7 ↑2.3

Table 4: Plug-and-play accuracy (%) of proposed modules. The changes are relative to the first row.

Configuration					Param. (M)	Acc. (%)
T	M	B	MK	MD	Param. (M)	Acc. (%)
✓	✗	✗	✗	✓	2.80	90.2
✗	✓	✗	✗	✓	2.80	90.6
✗	✗	✓	✗	✓	2.80	90.8
✗	✗	✓	✓	✗	3.04 ↑0.24	90.5 ↓0.3

Table 5: Accuracy (%) in various spatial and temporal configurations. The changes are relative to the third row.

Configuration	Param. (M)	Acc. (%)
Fig. 8 w/ $\odot\mathbf{M}_{k}$ (see Eq. 7 )	2.80	90.8
Fig. 8 w/ $+\mathbf{M}_{k}$ (see Eq. 7 )	2.80 ↑0.00	90.2 ↓0.6
Fig. 8 w/ $\odot\mathbf{M}_{k}$ (see Eq. 7 )	1.40 ↓1.40	88.5 ↓2.3
Fig. 8 w/ $\odot\mathbf{M}_{k}$ (see Eq. 7 )	2.80 ↑0.00	90.5 ↓0.3
Fig. 8 w/o $\mathcal{M}_{\mathrm{s}},\mathcal{M}_{\mathrm{t}}$	2.80 ↑0.00	89.7 ↓1.1
Fig. 8 w/o $\mathbf{M}_{\mathrm{att}}$ (see Eq. 10 )	2.23 ↓0.57	89.9 ↓0.9

Table 6: Accuracy (%) with various configurations. The changes are relative to the first row.

Figure 9: A comparison of the training loss and testing accuracy between CTR-GCN [ 5 ] and our method. The results demonstrate the effectiveness of our proposed model, as it achieves lower training loss and higher testing accuracy compared to CTR-GCN.

Panel of Figure 10 — Figure 10: Heatmaps of (a) the fusion matrix and (b) the confusion matrix. $\lambda_{1}$ and $\lambda_{2}$ are coefficient weights.

Panel of Figure 11 — Figure 11: Selected frames with their human skeletons of two failure cases.

Figure 12: An accuracy comparison of six challenging classes in the X-Sub setting on NTU RGB+D 60.

#	Action	CTR-GCN [ 5 ]	Ours
1	Reading (GT)	0.21	0.82
1	Typing on a keyboard	0.53	0.05
2	Typing on a keyboard (GT)	0.12	0.59
2	Writing	0.47	0.28
3	Touching neck (GT)	0.02	0.88
3	Brushing hair	0.96	0.10

Table 7: Prediction scores of CTR-GCN [ 5 ] and NeuroPath.

Panel of Figure 13 — Figure 13: Selected challenging and well-recognized samples.

Panel of Figure 14 — Figure 14: Attention maps in the first STGGCB: (a) ‘s1’, ‘s2’, and ‘s3’ are heatmaps of three STGGCBs in the spatial-aware pathway, (b) ‘t1’, ‘t2’, and ‘t3’ are heatmaps of three STGGCBs in the temporal-aware pathway. The darker color denotes the higher attention weight for the corresponding joint.

Explore similar work

GCN-DevLSTM: Path Development for Skeleton-Based Action Recognition
Mar 22, 2024Lei Jiang, Weixin Yang, Xin Zhang +1Skeleton-Based Action Recognition Human Activity Recognition
University College London · University of Oxford · South China University of Technology
LLM Enhanced Action Recognition via Hierarchical Global-Local Skeleton-Language Model
Mar 28, 2026Ruosi Wang, Fangwei Zuo, Lei Li +1Skeleton-Based Action Recognition Cross-Modal Alignment
Northwestern Polytechnical University, No.1 Dongxiang Road, Chang’an District , Xi’an, 710129, Shaanxi, China · Shandong University of Finance and Economics, No.40 Shungeng Road, Licheng District, Jinan, Jinan, Shandong, China
KGS-GCN: Kinematics-Driven Gaussian Splatting and Probabilistic Topology for Skeleton-Based Action Recognition
Mar 16, 2026Yuhan Chen, Yicui Shi, Guofa Li +4Skeleton-Based Action Recognition Human Activity Recognition
College of Mechanical and Vehicle Engineering, Chongqing University, Chongqing 400044, China · Department of Mathematical Sciences, Tsinghua University, Beijing 100084, China · School of Vehicle and Mobility, Tsinghua University, Beijing 100084, China +1

Mar 22, 2024J/K move · Enter open · S save

GCN-DevLSTM: Path Development for Skeleton-Based Action Recognition

Lei Jiang, Weixin Yang, Xin Zhang, Hao Ni

University College London · University of Oxford · South China University of Technology

Skeleton-Based Action Recognition Human Activity Recognition Spatiotemporal GNNs

Open paper AskPDF arXiv

Skeleton-based action recognition (SAR) in videos is an important but challenging task in computer vision. The recent state-of-the-art (SOTA) models for SAR are primarily based on graph convolutional neural networks (GCNs), which are powerful in extracting the spatial information from skeleton data. However, their ability to capture temporal dynamics remains limited. To address this, we propose the G-Dev layer, which leverages path development-a principled and parsimonious representation for sequential data based on Lie group structures-to enhance temporal modeling. By integrating the G-Dev layer, the proposed DevLSTM module summarizes local temporal dynamics, reducing the time dimension while retaining high-frequency information. It can be conveniently applied to any temporal graph data, complementing existing advanced GCN-based models. Our empirical studies on the NTU-60, NTU-120 and Chalearn2013 datasets demonstrate that our proposed GCN-DevLSTM network consistently improves the strong GCN baseline models and achieves competitive performance. The code repository is publicly available at https://github.com/DeepIntoStreams/GCN-DevLSTM.

Abstract

Explore similar work

GCN-DevLSTM: Path Development for Skeleton-Based Action Recognition

LLM Enhanced Action Recognition via Hierarchical Global-Local Skeleton-Language Model

KGS-GCN: Kinematics-Driven Gaussian Splatting and Probabilistic Topology for Skeleton-Based Action Recognition