NeuroPath: Brain-Inspired Dual-Pathway Graph Convolutional Networks for Skeleton-Based Action Recognition
Authors: Kanglei Zhou, Ruizhi Cai, Hubert P. H. Shum, Frederick W. B. Li, Xiaohui Liang
Organizations: Department of Psychological and Cognitive Sciences, Tsinghua University, No. 30 Shuangqing Road, Haidian District, Beijing, 100084, China · State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, No. 37 Xueyuan Road, Haidian District, Beijing, 100191, China · Department of Computer Science, Durham University, Stockton Rd, Durham, DH1 3LE, United Kingdom · Zhongguancun Laboratory, Beijing, China
Skeleton-based action recognition aims to recognize human actions from sequences of human joint coordinates. Most existing Spatial-Temporal Graph Convolutional Networks (STGCNs) have achieved promising results by modeling skeletal structures with implicit spatial-temporal representations. However, our empirical study reveals a clear performance imbalance across different skeletal modalities, indicating that implicitly coupling spatial and temporal information limits the full exploitation of complementary structural and motion cues. Inspired by the ventral and dorsal pathways in human perception, we propose Dual-Pathway Graph Convolutional Networks (NeuroPath), which adopt a dual-pathway architecture for separate yet collaborative modeling of spatial and temporal information. Specifically, transformation units first convert the input into pathway-specific skeletal representations, allowing each pathway to focus on complementary aspects of human motion. To further capture coordinated joint behaviors and their interrelationships, we introduce a group graph convolution block that dynamically identifies key body parts and models their spatial-temporal dependencies. In addition, inter-pathway dynamic fusion modules integrate complementary inter-modal information across pathways, facilitating higher-level semantic interpretation of actions. Extensive experiments on Kinetics Skeleton 400, NTU RGB+D 60, and NTU RGB+D 120 demonstrate consistent performance improvements, validating the effectiveness of dual-pathway spatial-temporal modeling for skeleton-based action recognition.
Figures & tables
Figure 1: The bubble plot of model computational performance in the X-Sub setting on the NTU RGB+D 60 dataset. The horizontal axis represents computational operations (GFLOPs), the vertical axis represents accuracy (%), and the size of the bubbles represents the number of parameters (M). Different bubble colors represent different kinds of models.
Figure 2: Motivation and empirical observations. (a) Common skeletal modality representations derived from joint sequences, including joint positions, bone vectors, joint motion, and bone motion. (b) Inspiration from the two-stream hypothesis of the human visual system, where spatial structure and motion are processed by two parallel yet interacting pathways. (c) Empirical results of CTR-GCN [ 5 ] on the X-Sub protocol of NTU RGB+D 120 under different skeletal modalities. We report Top-1 Accuracy (denoted as Accuracy). Single-stream results exhibit clear performance imbalance across modalities, while combining structural and motion information consistently improves performance, highlighting their complementarity. (d) Results of our method, which significantly alleviates modality imbalance and achieves more balanced performance across different skeletal modalities.
Figure 3: Overview of the proposed NeuroPath framework. NeuroPath adopts a brain-inspired dual-pathway architecture to explicitly model the complementary spatial-aware and temporal-aware information in skeletal data. The two pathways are conceptually motivated by the ventral and dorsal streams in the human visual system, which specialize in spatial structure and motion processing, respectively. In contrast to existing multi-stream networks that fuse modalities only at the output level, NeuroPath introduces dynamic inter-pathway fusion modules to enable effective interaction and coordination between spatial-aware and temporal-aware representations throughout the network.
Figure 4: Core design intuition of STGGCB . Discriminative actions are characterized by coordinated groups of joints with correlated temporal dynamics. STGGCB first summarizes local temporal variation at the group level and then conditions spatial aggregation on this motion-aware context, enabling graph convolution to emphasize structurally and temporally relevant joint interactions.
Figure 5: The network architecture of STGGCB : (a) Group Graph Convolution (G-GC) and (b) Multi-Scale Temporal Convolution (MS-TC). G-GC comprises group aggregation (see Eq. 4 ) and adaptive structural aggregation (see Eq. 5 ).
Figure 6: Illustration of multi-hop aggregation strategies on a simple chain graph. (a) A 7-node chain graph, where node 4 is selected as the center node and 1-hop, 2-hop, and 3-hop relations are shown by different colors. (b) Exact k -hop adjacency supports A~k , where each matrix contains only node pairs whose shortest-path distance is exactly k . (c) Conventional baselines using cumulative multi-hop aggregation ( A1+A2+A3 ), which entangle different hop distances within a single representation. (d) Our method, which applies learnable masks Mk over exact k -hop supports, enabling adaptive reweighting while preserving explicit separation of neighborhood ranges.
Figure 7: The illustration of the spatial-temporal attention module. ‘ ⋅ ’ and ‘ × ’ represent matrix dot-product and matrix multiplication operations, respectively.
Figure 8: A comparison of STGCN network architectures: (a) our dual-pathway with dynamic fusion, (b) single pathway, (c) dual-pathway with hard fusion.
Ensemble stream
Method
Publication
Model type
Kinetics Skeleton 400
NTU RGB+D 60
NTU RGB+D 120
Top-1
Top-5
X-Sub
X-View
X-Sub
X-Setup
Joint- only
ST-GCN [ 36 ]
AAAI 2018
(2+1)D
30.7
52.8
81.5
88.3
70.7
73.2
AGCN [ 25 ]
CVPR 2019
(2+1)D
34.8
56.5
86.8
94.2
77.9
78.5
Shift-GCN [ 7 ]
CVPR 2020
(2+1)D
34.7
56.8
87.8
95.1
80.9
83.2
CTR-GCN [ 5 ]
ICCV 2021
(2+1)D
36.5
58.5
89.0
94.8
83.6
84.5
InfoGCN [ 8 ]
CVPR 2022
(2+1)D
36.7
58.7
90.4
95.0
84.1
85.2
Table 1: Accuracy (%) comparison. The symbol “–” denotes unreported results. Underlined values indicate results taken from prior work [ 11 ] . Bold values indicate the best result in each column.
Configuration
Param. (M)
Acc. (%)
Baseline w/o Group
2.51
88.7
Baseline w/ Overlapping
2.80 ↑0.29
90.8 ↑2.1
Baseline w/ Non-Overlapping
2.80 ↑0.29
90.2 ↑1.5
Table 2: Accuracy (%) with various grouping strategies. The changes are relative to the first row.
Configuration
Window Size
Param. (M)
Acc. (%)
B1
B2
B3
w/o Group
✗
✗
✗
2.51
88.7
w/ Group
✓
✗
✗
2.65 ↑0.14
90.4 ↑1.7
✓
✓
✗
2.80 ↑0.29
90.8 ↑2.1
✓
✓
✓
3.36 ↑0.85
90.4 ↑1.7
✗
✓
✗
2.66 ↑0.15
90.1 ↑1.4
Table 3: Accuracy (%) with various group configurations. The changes are relative to the first row.
Configuration
Param. (M)
Acc. (%)
AGCN (Js) [ 26 ]
3.47
87.4
AGCN (Js) w/ Matt
3.87 ↑0.40
88.2 ↑0.8
AGCN (Js) w/ Group
4.45 ↑0.98
89.1 ↑1.7
AGCN (Js) w/ Group + Matt
4.85 ↑1.38
89.7 ↑2.3
Table 4: Plug-and-play accuracy (%) of proposed modules. The changes are relative to the first row.
Configuration
Param. (M)
Acc. (%)
T
M
B
MK
MD
✓
✗
✗
✗
✓
2.80
90.2
✗
✓
✗
✗
✓
2.80
90.6
✗
✗
✓
✗
✓
2.80
90.8
✗
✗
✓
✓
✗
3.04 ↑0.24
90.5 ↓0.3
Table 5: Accuracy (%) in various spatial and temporal configurations. The changes are relative to the third row.
Configuration
Param. (M)
Acc. (%)
Fig. 8 w/ ⊙Mk (see Eq. 7 )
2.80
90.8
Fig. 8 w/ +Mk (see Eq. 7 )
2.80 ↑0.00
90.2 ↓0.6
Fig. 8 w/ ⊙Mk (see Eq. 7 )
1.40 ↓1.40
88.5 ↓2.3
Fig. 8 w/ ⊙Mk (see Eq. 7 )
2.80 ↑0.00
90.5 ↓0.3
Fig. 8 w/o Ms,Mt
2.80 ↑0.00
89.7 ↓1.1
Fig. 8 w/o Matt (see Eq. 10 )
2.23 ↓0.57
89.9 ↓0.9
Table 6: Accuracy (%) with various configurations. The changes are relative to the first row.
Figure 9: A comparison of the training loss and testing accuracy between CTR-GCN [ 5 ] and our method. The results demonstrate the effectiveness of our proposed model, as it achieves lower training loss and higher testing accuracy compared to CTR-GCN.
Figure 10: Heatmaps of (a) the fusion matrix and (b) the confusion matrix. λ1 and λ2 are coefficient weights.
Figure 11: Selected frames with their human skeletons of two failure cases.
Figure 12: An accuracy comparison of six challenging classes in the X-Sub setting on NTU RGB+D 60.
#
Action
CTR-GCN [ 5 ]
Ours
1
Reading (GT)
0.21
0.82
Typing on a keyboard
0.53
0.05
2
Typing on a keyboard (GT)
0.12
0.59
Writing
0.47
0.28
3
Touching neck (GT)
0.02
0.88
Brushing hair
0.96
0.10
Table 7: Prediction scores of CTR-GCN [ 5 ] and NeuroPath.
Figure 13: Selected challenging and well-recognized samples.
Figure 14: Attention maps in the first STGGCB: (a) ‘s1’, ‘s2’, and ‘s3’ are heatmaps of three STGGCBs in the spatial-aware pathway, (b) ‘t1’, ‘t2’, and ‘t3’ are heatmaps of three STGGCBs in the temporal-aware pathway. The darker color denotes the higher attention weight for the corresponding joint.
Skeleton-based action recognition (SAR) in videos is an important but challenging task in computer vision. The recent state-of-the-art (SOTA) models for SAR are primarily based on graph convolutional neural networks (GCNs), which are powerful in extracting the spatial information from skeleton data. However, their ability to capture temporal dynamics remains limited. To address this, we propose the G-Dev layer, which leverages path development-a principled and parsimonious representation for sequential data based on Lie group structures-to enhance temporal modeling. By integrating the G-Dev layer, the proposed DevLSTM module summarizes local temporal dynamics, reducing the time dimension while retaining high-frequency information. It can be conveniently applied to any temporal graph data, complementing existing advanced GCN-based models. Our empirical studies on the NTU-60, NTU-120 and Chalearn2013 datasets demonstrate that our proposed GCN-DevLSTM network consistently improves the strong GCN baseline models and achieves competitive performance. The code repository is publicly available at https://github.com/DeepIntoStreams/GCN-DevLSTM.
Lei Jiang, Weixin Yang, Xin Zhang +1
University College London · University of Oxford · South China University of Technology
Skeleton-based human action recognition has achieved remarkable progress in recent years. However, most existing GCN-based methods rely on short-range motion topologies, which not only struggle to capture long-range joint dependencies and complex temporal dynamics but also limit cross-modal semantic alignment and understanding due to insufficient modeling of action semantics. To address these challenges, we propose a hierarchical global-local skeleton-language model (HocSLM), enabling the large action model be more representative of action semantics. First, we design a hierarchical global-local network (HGLNet) that consists of a composite-topology spatial module and a dual-path hierarchical temporal module. By synergistically integrating multi-level global and local modules, HGLNet achieves dynamically collaborative modeling at both global and local scales while preserving prior knowledge of human physical structure, significantly enhancing the model's representation of complex spatio-temporal relationships. Then, a large vision-language model (VLM) is employed to generate textual descriptions by passing the original RGB video sequences to this model, providing the rich action semantics for further training the skeleton-language model. Furthermore, we introduce a skeleton-language sequential fusion module by combining the features from HGLNet and the generated descriptions, which utilizes a skeleton-language model (SLM) for aligning skeletal spatio-temporal features and textual action descriptions precisely within a unified semantic space. The SLM model could significantly enhance the HGLNet's semantic discrimination capabilities and cross-modal understanding abilities. Extensive experiments demonstrate that the proposed HocSLM achieves the state-of-the-art performance on three mainstream benchmark datasets: NTU RGB+D 60, NTU RGB+D 120, and Northwestern-UCLA.
Ruosi Wang, Fangwei Zuo, Lei Li +1
Northwestern Polytechnical University, No.1 Dongxiang Road, Chang’an District , Xi’an, 710129, Shaanxi, China · Shandong University of Finance and Economics, No.40 Shungeng Road, Licheng District, Jinan, Jinan, Shandong, China
Skeleton-based action recognition is widely applied in sensor-based systems, including human-computer interaction and intelligent surveillance. However, typical sensors produce sparse and discrete joint coordinates, often leading to the loss of fine-grained spatiotemporal information during dynamic movements. Furthermore, predefined physical topologies restrict modeling potential long-range dependencies. To address these challenges, we propose KGS-GCN, which integrates kinematics-driven Gaussian splatting and probabilistic topology within a graph convolutional network. A Gaussian splatting module constructs anisotropic covariance matrices by extracting instantaneous joint velocity vectors, rendering sparse skeleton sequences into multi-view continuous heatmaps rich in spatiotemporal semantics. Additionally, a probabilistic topology construction strategy transcends physical connectivity limitations by utilizing the Bhattacharyya distance to quantify statistical correlations between joint Gaussian distributions, generating an adaptive prior adjacency matrix. Finally, the lightweight multi-view rendering branch and topological GCN backbone are unified through a visual context gating mechanism, enabling seamless fusion of continuous dynamic cues with structural priors while maintaining high computational efficiency, requiring only 1.4M parameters and 1.3 GFLOPs. Extensive experiments on multiple benchmark datasets demonstrate that KGS-GCN significantly enhances the modeling of complex spatiotemporal dynamics and achieves competitive performance at low computational cost, establishing an efficient paradigm for improving the perceptual robustness of low-fidelity sensor data.
Yuhan Chen, Yicui Shi, Guofa Li +4
College of Mechanical and Vehicle Engineering, Chongqing University, Chongqing 400044, China · Department of Mathematical Sciences, Tsinghua University, Beijing 100084, China · School of Vehicle and Mobility, Tsinghua University, Beijing 100084, China +1