Many modern video-based human action recognition (HAR) approaches use 2D skeleton as the intermediate representation in their prediction pipelines. Despite overall encouraging results, these approaches still struggle in many common scenes, mainly because the skeleton does not capture critical action-related information pertaining to the depth of the joints, contour of the human body, and interaction between the human and objects. To address this, we propose an effective approach to augment skeleton with a representation capturing action-related information in the pipeline of HAR. The representation, termed Scale-Body-Flow (SBF), consists of three distinct components, namely a scale map volume given by the scale (and hence depth information) of each joint, a body map outlining the human subject, and a flow map indicating human-object interaction given by pixel-wise optical flow values. To predict SBF, we further present SFSNet, a novel segmentation network supervised by the skeleton and optical flow without extra annotation overhead beyond the existing skeleton extraction. Extensive experiments across different datasets demonstrate that our pipeline based on SBF and SFSNet achieves significantly higher HAR accuracy with similar compactness and efficiency as compared with the state-of-the-art skeleton-only approaches.
Figures & tables
Figure 1: Comparison of video-based HAR pipelines. Our proposed pipeline (c) employs SBF predicted by SFSNet to augment skeleton for effective HAR, addressing the limitations of the skeleton-only approach (b).
Figure 2: Failure cases of HAR based on extracting 2D skeletons from videos, addressed with our SBF representation predicted via SFSNet. Top row: original video frame; middle row: skeleton with prediction error; bottom row: correct prediction using skeleton+SBF, owing to the capturing of the action-related information. Red circles highlight key regions possibly leading to the error. (a) Joint depth: The 2D skeleton’s flat nature causes the ambiguity of the depth of each joint, e.g., confusing “squatting down” and “sitting down” from a front view. (b) Body contour: The contour of human body, an encircling of all body parts, provides richer features than skeleton. E.g., “reaching into a pocket” is mis-predicted as “drop” because the subject’s left hand is not shown overlapping with his body in the skeleton. (c) Human-object interaction (HOI): The skeleton misses interactions between the subject and objects, e.g., “throwing” misinterpreted as “waving” due to the ignored motion between the subject and the thrown object.
Figure 3: An example video frame, its extracted skeleton, and our proposed SBF.
Figure 4: The overall structure of SFSNet. The flow estimator is pretrained via unsupervised learning.
Figure 5: A conceptual example of the “waving” action for our annotation generation method in SPR. In each segmentation task, the white region denotes Ppos (positive labels), the black region represents Pneg (negative labels), and the grey region indicates areas excluded from point sampling.
Method
Category
Add. Anno.
Params (M)
FLOPs (G)
Clips
NTU
NTU120
X-Sub
X-View
X-Sub
X-Set
ST-GCN [ 46 ]
Skeleton
✗
12.3
15.3
10
92.4
98.3
84.7
89.0
AA-GCN [ 30 ]
Skeleton
✗
15.0
17.4
10
93.0
98.2
85.5
89.9
MS-G3D [ 24 ]
Skeleton
✗
11.8
27.4
10
94.1
98.3
87.4
90.9
CTR-GCN [ 5 ]
Skeleton
✗
5.7
7.8
10
93.6
98.4
86.6
90.1
ST-GCN++ [ 10 ]
Skeleton
✗
5.6
11.2
10
93.2
98.5
86.4
90.3
Table 1: Comparison of model size, computational cost and accuracy (%) with the state-of-the-art HAR methods on NTU and NTU120. All methods shown in the table extract intermediate representations (skeleton, object and SBF) from videos. Number of parameters and FLOPs are calculated for the downstream HAR model alone. Both 1-clip and 10-clip testing are applied to PoseConv3D and SBFConv3D. ”J” and ”L” indicate the joint and limb variants of SBF defined in Sec. 3.5 .
Table 7
Method
Category
Front
Side
45∘
PoseConv3D [ 11 ]
Skeleton
87.4
83.5
86.9
SBFConv3D
Skl+SBF
90.1
87.3
89.9
Δ
-
+2.7
+3.8
+3.0
Table 4: Comparison of accuracy (%) on videos captured from different viewing angles on NTU120 X-Sub.
Method
Category
Hard
Medium
Easy
PoseConv3D [ 11 ]
Skeleton
52.2
82.2
96.5
SBFConv3D
Skl+SBF
61.1
87.1
97.4
Δ
-
+8.9
+4.9
+0.9
Table 5: Comparison of accuracy (%) on actions with different difficulty levels on NTU120 X-Sub.
Figure 6: The accuracy difference (%) between our SBFConv3D and PoseConv3D for hard actions on NTU120 X-Sub.
Skeleton
S
B
F
Acc (%)
✓
93.5
✓
90.3
✓
86.7
✓
87.8
✓
✓
93.6
✓
✓
✓
93.8
Table 6: Ablation Study on SBF components on NTU X-Sub.
Method
Network
Acc (%)
Params
FLOPs
PoseConv3D [ 11 ]
SO-R50
93.7
2.0M
15.8G
C3D-s
92.9
3.4M
16.8G
X3D-s
92.3
241K
0.6G
SBFConv3D
SO-R50
94.6
2.0M
16.2G
C3D-s
93.4
3.4M
17.1G
X3D-s
92.4
242K
0.7G
Table 7: Comparison with PoseConv3D using different downstream networks on NTU X-Sub.
Figure 7: Visualization of SBF components predicted by SFSNet on NTU120 X-Sub. Each joint’s scale map has a distinct color.
Table 14Table 15
Skeleton
S
B
F
Acc (%)
✓
93.5
✓
90.3
✓
86.7
✓
87.8
✓
✓
93.6
✓
✓
93.6
Table 12: Complete Ablation Study on SBF components on NTU X-Sub.
Figure 8: Comparison of Implicit PointRend [ 6 ] and our Simplified PointRend architectures.
Figure 9: The accuracy difference (%) between our SBFConv3D and PoseConv3D for “Medium” actions on NTU120 X-Sub.
Figure 10: Visualization of SBF components predicted by SFSNet on NTU120 [ 23 ] (row 1-6), HMDB51 [ 19 ] (row 7-8) and UCF101 [ 33 ] (row 9-10). Each joint is depicted in a distinct color.