Insights from Visual Cognition: Understanding Human Action Dynamics with Overall Glance and Refined Gaze Transformer
Authors: Bohao Xing, Deng Li, Rong Gao, Xin Liu, Heikki Kälviäinen
Organizations: Department of Computational Engineering, Lappeenranta-Lahti University of Technology LUT, Finland · Faculty of Information Technology, Brno University of Technology, Czech Republic
Recently, Transformer has made significant progress in various vision tasks. To balance computation and efficiency in video tasks, recent works heavily rely on factorized or window-based self-attention. However, these approaches split spatiotemporal correlations between regions of interest in videos, limiting the models' ability to capture motion and long-range dependencies. In this paper, we argue that, similar to the human visual system, the importance of temporal and spatial information varies across different time scales, and attention is allocated sparsely over time through glance and gaze behavior. Is equal consideration of time and space crucial for success in video tasks? Motivated by this understanding, we propose a dual-path network called the Overall Glance and Refined Gaze (OG-ReG) Transformer. The Glance path extracts coarse-grained overall spatiotemporal information, while the Gaze path supplements the Glance path by providing local details. Our model achieves state-of-the-art results on the Kinetics-400, Something-Something v2, and Diving-48, demonstrating its competitive performance. The code will be available at https://github.com/linuxsino/OG-ReG.
Figures & tables
Figure 1: The differences between window-based attention and glance-like attention. (a) Frame sequences ( pushing notebook so that it falls off the table ) from SSv2 [ 25 ] dataset. (b) In window-based self-attention (e.g., Video-Swin [ 48 ] ), a square represents a window within which self-attention performs its calculations. When an object (marked with red) zooms, moves, and rotates across many different windows in frame sequences, the tokens in the yellow or green window of the first frame merely communicate with the tokens of the same object in subsequent frames at the later stage of the network. It is difficult to obtain an excellent receptive field to learn spatiotemporal information, particularly to model motion. (c) After spatial downsampling, each square (which represents a token ) can communicate freely with others, resulting in an attention mechanism that resembles a glance.
Figure 2: By visualizing the similarity matrices and their corresponding three video sequences ((a) a fast tempo, (b) a slow tempo, (c) a slower tempo) from the same action category, we demonstrate that matrix A can effectively capture the tempo characteristics of the actions.
Figure 3: The framework of OG-ReG Transformer. (a) An overview of the proposed Overall Glance and Refined Gaze (OG-ReG) Transformer. (b) The fundamental building block of OG-ReG, namely the OG-ReG block.
Figure 4: Details of the OG-ReG block (neglecting Layer-Norm).
Figure 5: Details of the MDConv.
Method
Pretrain
Top-1
Top-5
Views
FLOPs
Param
R(2+1)D [ 73 ]
-
72.0
90.0
10×1
75
61.8
I3D [ 10 ]
ImageNet-1K
72.1
90.3
1×1
108
28.0
NL-I3D [ 79 ]
ImageNet-1K
77.7
93.3
6×10
32
35.3
CoST [ 35 ]
ImageNet-1K
77.5
93.2
3×10
33
35.3
SlowFast-R50 [ 20 ]
ImageNet-1K
75.6
92.1
3×10
36
32.4
X3D-XL [ 21 ]
-
79.1
93.9
3×10
48
11.0
Table 1: Comparison with state-of-the-art on K400.
Method
Pretrain
Top-1
Top-5
Views
FLOPs
Param
TSM [ 46 ]
K400
63.4
88.5
3×2
65
24.3
TEINet [ 49 ]
ImageNet-1K
62.1
-
1×1
66
30.4
TDN [ 75 ]
ImageNet-1K
65.3
89.5
1×1
72
24.8
ACTION-Net [ 81 ]
ImageNet-1K
64.0
89.3
1×1
70
28.1
SlowFast-R101, 8x8 [ 20 ]
K400
63.1
87.6
3×1
106
53.3
MSNet [ 32 ]
ImageNet-1K
64.7
89.4
1×1
101
24.6
Table 2: Comparison with state-of-the-art on SSv2.
Method
Pretrain
Top-1
Top-5
Views
FLOPs
Param
SlowFast-R101, 8x8 [ 20 ]
K400
77.6
-
3×1
106
53.3
Timesformer [ 4 ]
ImageNet-21K
74.9
-
3×1
196
121.4
Timesformer-HR [ 4 ]
ImageNet-21K
78.0
-
3×1
1703
121.4
Timesformer-L [ 4 ]
ImageNet-21K
81.0
-
3×1
2380
121.4
DVT-B [ 74 ]
ImageNet-1K
86.0
-
-
128
-
PST-T [ 85 ]
ImageNet-1K
79.2
98.2
3×1
72
28.5
Table 3: Comparison with state-of-the-art on Diving-48 v2.
Spatial
Temporal
Pixel Shuffle
K400
SSv2
Top-1
Top-5
Top-1
Top-5
2D
w/o
2D
79.5
94.2
68.9
91.2
3D
3D
78.6
93.5
68.2
90.9
2D
1D
R(2+1)D
76.2
92.5
66.3
90.0
Table 4: Ablation study on Clip Level on K400 and SSv2.
Pattern
K400
SSv2
Top-1
Top-5
Top-1
Top-5
2D
79.2
94.0
68.5
91.3
3D
79.1
93.8
68.6
91.2
fixed 2D + 3D
79.2
94.0
68.5
91.2
factorized 3D
79.3
93.7
68.6
91.2
2D+3D
79.5
94.2
68.9
91.2
Table 5: Ablation study on Frame Level pattern on K400 and SSv2.
SoDA
MDConv
K400
SSv2
Top-1
Top-5
Top-1
Top-5
✓
78.5
93.6
68.2
90.9
✓
77.0
92.7
66.0
90.0
✓
✓
79.5
94.2
68.9
91.2
Video-Swin-T [ 48 ]
78.8
93.6
66.2
90.8
PST-T [ 85 ]
78.2
92.2
67.3
90.5
Table 6: Ablation study on Contributions of the Two Paths on K400 and SSv2.
Figure 6: Fourier spectrum of features in OG-ReG-T and Video-Swin-B.
Manner
K400
SSv2
Top-1
Top-5
Top-1
Top-5
Dynamic Conv [ 12 ]
79.3
94.1
68.5
91.3
Ours
79.5
94.2
68.9
91.2
Table 7: Ablation study on Guide Manner on K400 and SSv2.
Frame
K400
SSv2
Top-1
Top-5
Top-1
Top-5
32 w. D
78.6
93.5
68.2
90.9
32 w/o D
79.5
94.2
68.9
91.2
64 w. D
79.8
94.1
69.6
91.7
64 w/o D
80.2
94.3
69.5
91.6
Table 8: Ablation study on Frame Number on K400 and SSv2.
Conv in MSA
K400
SSv2
Top-1
Top-5
Top-1
Top-5
2D
79.5
94.2
68.9
91.2
3D
79.0
93.8
68.5
91.1
Table 9: Ablation study on Convolution in Self-Attention on K400 and SSv2.
Figure 7: Grad-Cam and Visual tempo Visualization.