Many modern video-based human action recognition (HAR) approaches use 2D skeleton as the intermediate representation in their prediction pipelines. Despite overall encouraging results, these approaches still struggle in many common scenes, mainly because the skeleton does not capture critical action-related information pertaining to the depth of the joints, contour of the human body, and interaction between the human and objects. To address this, we propose an effective approach to augment skeleton with a representation capturing action-related information in the pipeline of HAR. The representation, termed Scale-Body-Flow (SBF), consists of three distinct components, namely a scale map volume given by the scale (and hence depth information) of each joint, a body map outlining the human subject, and a flow map indicating human-object interaction given by pixel-wise optical flow values. To predict SBF, we further present SFSNet, a novel segmentation network supervised by the skeleton and optical flow without extra annotation overhead beyond the existing skeleton extraction. Extensive experiments across different datasets demonstrate that our pipeline based on SBF and SFSNet achieves significantly higher HAR accuracy with similar compactness and efficiency as compared with the state-of-the-art skeleton-only approaches.
Figures & tables
Figure 1: Comparison of video-based HAR pipelines. Our proposed pipeline (c) employs SBF predicted by SFSNet to augment skeleton for effective HAR, addressing the limitations of the skeleton-only approach (b).
Figure 2: Failure cases of HAR based on extracting 2D skeletons from videos, addressed with our SBF representation predicted via SFSNet. Top row: original video frame; middle row: skeleton with prediction error; bottom row: correct prediction using skeleton+SBF, owing to the capturing of the action-related information. Red circles highlight key regions possibly leading to the error. (a) Joint depth: The 2D skeleton’s flat nature causes the ambiguity of the depth of each joint, e.g., confusing “squatting down” and “sitting down” from a front view. (b) Body contour: The contour of human body, an encircling of all body parts, provides richer features than skeleton. E.g., “reaching into a pocket” is mis-predicted as “drop” because the subject’s left hand is not shown overlapping with his body in the skeleton. (c) Human-object interaction (HOI): The skeleton misses interactions between the subject and objects, e.g., “throwing” misinterpreted as “waving” due to the ignored motion between the subject and the thrown object.
Figure 3: An example video frame, its extracted skeleton, and our proposed SBF.
Figure 4: The overall structure of SFSNet. The flow estimator is pretrained via unsupervised learning.
Figure 5: A conceptual example of the “waving” action for our annotation generation method in SPR. In each segmentation task, the white region denotes Ppos (positive labels), the black region represents Pneg (negative labels), and the grey region indicates areas excluded from point sampling.
Method
Category
Add. Anno.
Params (M)
FLOPs (G)
Clips
NTU
NTU120
X-Sub
X-View
X-Sub
X-Set
ST-GCN [ 46 ]
Skeleton
✗
12.3
15.3
10
92.4
98.3
84.7
89.0
AA-GCN [ 30 ]
Skeleton
✗
15.0
17.4
10
93.0
98.2
85.5
89.9
MS-G3D [ 24 ]
Skeleton
✗
11.8
27.4
10
94.1
98.3
87.4
90.9
CTR-GCN [ 5 ]
Skeleton
✗
5.7
7.8
10
93.6
98.4
86.6
90.1
ST-GCN++ [ 10 ]
Skeleton
✗
5.6
11.2
10
93.2
98.5
86.4
90.3
Table 1: Comparison of model size, computational cost and accuracy (%) with the state-of-the-art HAR methods on NTU and NTU120. All methods shown in the table extract intermediate representations (skeleton, object and SBF) from videos. Number of parameters and FLOPs are calculated for the downstream HAR model alone. Both 1-clip and 10-clip testing are applied to PoseConv3D and SBFConv3D. ”J” and ”L” indicate the joint and limb variants of SBF defined in Sec. 3.5 .
Table 7
Method
Category
Front
Side
45∘
PoseConv3D [ 11 ]
Skeleton
87.4
83.5
86.9
SBFConv3D
Skl+SBF
90.1
87.3
89.9
Δ
-
+2.7
+3.8
+3.0
Table 4: Comparison of accuracy (%) on videos captured from different viewing angles on NTU120 X-Sub.
Method
Category
Hard
Medium
Easy
PoseConv3D [ 11 ]
Skeleton
52.2
82.2
96.5
SBFConv3D
Skl+SBF
61.1
87.1
97.4
Δ
-
+8.9
+4.9
+0.9
Table 5: Comparison of accuracy (%) on actions with different difficulty levels on NTU120 X-Sub.
Figure 6: The accuracy difference (%) between our SBFConv3D and PoseConv3D for hard actions on NTU120 X-Sub.
Skeleton
S
B
F
Acc (%)
✓
93.5
✓
90.3
✓
86.7
✓
87.8
✓
✓
93.6
✓
✓
✓
93.8
Table 6: Ablation Study on SBF components on NTU X-Sub.
Method
Network
Acc (%)
Params
FLOPs
PoseConv3D [ 11 ]
SO-R50
93.7
2.0M
15.8G
C3D-s
92.9
3.4M
16.8G
X3D-s
92.3
241K
0.6G
SBFConv3D
SO-R50
94.6
2.0M
16.2G
C3D-s
93.4
3.4M
17.1G
X3D-s
92.4
242K
0.7G
Table 7: Comparison with PoseConv3D using different downstream networks on NTU X-Sub.
Figure 7: Visualization of SBF components predicted by SFSNet on NTU120 X-Sub. Each joint’s scale map has a distinct color.
Table 14Table 15
Skeleton
S
B
F
Acc (%)
✓
93.5
✓
90.3
✓
86.7
✓
87.8
✓
✓
93.6
✓
✓
93.6
Table 12: Complete Ablation Study on SBF components on NTU X-Sub.
Figure 8: Comparison of Implicit PointRend [ 6 ] and our Simplified PointRend architectures.
Figure 9: The accuracy difference (%) between our SBFConv3D and PoseConv3D for “Medium” actions on NTU120 X-Sub.
Figure 10: Visualization of SBF components predicted by SFSNet on NTU120 [ 23 ] (row 1-6), HMDB51 [ 19 ] (row 7-8) and UCF101 [ 33 ] (row 9-10). Each joint is depicted in a distinct color.
Human action recognition plays an important role when developing intelligent interactions between humans and machines. While there is a lot of active research on improving the machine learning algorithms for skeleton-based action recognition, not much attention has been given to the quality of the input skeleton data itself. This work demonstrates that by making use of multiple camera views to triangulate more accurate 3D~skeletons, the performance of state-of-the-art action recognition models can be improved significantly. This suggests that the quality of the input data is currently a limiting factor for the performance of these models. Based on these results, it is argued that the cost-benefit ratio of using multiple cameras is very favorable in most practical use-cases, therefore future research in skeleton-based action recognition should consider multi-view applications as the standard setup.
Daniel Bermuth, Alexander Poeppel, Wolfgang Reif
Institute for Software and Systems Engineering, University of Augsburg, Germany
Skeleton-based human action recognition is a powerful approach for understanding human behaviour from pose data, but collecting large-scale, diverse, and well-annotated 3D skeleton datasets is both expensive and labor-intensive. To address this challenge, we propose a conditional generative pipeline for data augmentation in skeleton action recognition. Our method learns the distribution of real skeleton sequences under the constraint of action labels, enabling the synthesis of diverse and high-fidelity data. Even with limited training samples, it can effectively generate skeleton sequences and achieve competitive recognition performance in low-data scenarios, demonstrating strong generalisation in downstream tasks. Specifically, we introduce a Transformer-based encoder-decoder architecture, combined with a generative refinement module and a dropout mechanism, to balance fidelity and diversity during sampling. Experiments on HumanAct12 and the refined NTU-RGBD (NTU-VIBE) dataset show that our approach consistently improves the accuracy of multiple skeleton-based action recognition models, validating its effectiveness in both few-shot and full-data settings. The source code can be found at here.
Xu Dong, Wanqing Li, Anthony Adeyemi-Ejeye +1
Innovative Media Lab, University of Surrey, Guildford, UK · Advanced Multimedia Research Lab, University of Wollongong, Wollongong, Australia
Fine-grained human action recognition (FHAR) must distinguish visually similar actions that differ mainly in body configuration, timing, or local appearance. RGB representations retain visual context but often suppress joint-level geometry, whereas skeleton representations encode kinematics but discard dense spatial detail. We introduce FineX, which factorizes fine-grained cues into RGB appearance, pose heatmap geometry, and skeletal-graph topology. Pairwise cross-attention enables symmetric, stream-preserving information exchange, followed by a streamwise latent sparse Mixture-of-Experts that routes each representation to a content-dependent subset of shared experts, regularized by a load-balancing objective. FineX achieves state-of-the-art results on Gym99, Gym288, and Diving48. On the long-tailed Gym288, it raises mean class accuracy from 68.6% to 76.2% (+7.6 points) without textual supervision or large-scale vision-language pre-training, demonstrating the benefit of structured visual-pose-graph fusion and conditional expert refinement for FHAR.
Imtiaz Ul Hassan, Tasweer Ahmad, Nik Bessis +1
Computer Science Department, Edge Hill University, Ormskirk L39 4QP, United Kingdom