Organizations: School of Artificial Intelligence, Beijing Normal University · School of Computer Science and Technology, Beihang University · AI Lab, Lenovo Research
Extensive occlusions in real-world scenarios pose challenges to gait recognition due to missing and noisy information, as well as body misalignment in position and scale. We argue that rich dynamic contextual information within a gait sequence inherently possesses occlusion-solving traits: 1) Adjacent frames with gait continuity allow holistic body regions to infer occluded body regions; 2) Gait cycles allow information integration between holistic actions and occluded actions. Therefore, we introduce an action detection perspective where a gait sequence is regarded as a composition of actions. To detect accurate actions under complex occlusion scenarios, we propose an Action Detection Based Mixture of Experts (GaitMoE), consisting of Mixture of Temporal Experts (MTE) and Mixture of Action Experts (MAE). MTE adaptively constructs action anchors by temporal experts and MAE adaptively constructs action proposals from action anchors by action experts. Especially, action detection as a proxy task with gait recognition is an end-to-end joint training only with ID labels. In addition, due to the lack of a unified occluded benchmark, we construct a pioneering Occluded Gait database (OccGait), containing rich occlusion scenarios and annotations of occlusion types. Extensive experiments on OccGait, OccCASIA-B,Gait3D and GREW demonstrate the superior performance of GaitMoE.OccGait is available at https://github.com/BNU-IVC/OccGait.
Figures & tables
Figure 1: (a) The main four occlusion issues in gait recognition. (b) The dynamic contextual information within a gait sequence can infer and integrate occlusion information.
Figure 2: Action composition. Each temporal expert focuses on one body region with individual temporal size, constructing action anchors. Each action expert integrates similar action anchors from different gait cycles, constructing one action proposal.
Figure 3: The overview of GaitMoE. Best viewed in colors, the input sequence is firstly extracted to frame-level features by C2D Block ( e.g . , 2D CNNs or a residual block), and Mixture of Temporal Experts (MTE) dispatch temporal experts (TE) with different temporal bounding boxes to the corresponding channel segments, forming action anchors. After horizontal partitioning (HP), Mixture of Action Experts (MAE) is independent for each part-level feature. For each channel segment, one action expert (AE) integrates similar action anchors along the temporal dimension by weighted sum operation, forming one action proposal. Finally, The concatenated action proposals as part features for identification.
Figure 4: (a) Mixture of Temporal Experts. Dila Conv (DC) represents Dilated Convolution, predefining action anchors with different dilated ratios. (b) Mixture of Action Experts. LP denotes the linear projection. Similar action anchors adaptively integrate into action proposals.
Figure 5: (a) The layout and camera views during collection. (b) The 3 types of occlusion scenarios where the number ①② are for Carrying Occlusion, ③④ are for Crowd Occlusion and ⑤⑥ are for Static Occlusion, and None Occlusion is omitted for simplicity.
Method
Probe View
Average
0∘
45∘
90∘
135∘
180∘
225∘
270∘
315∘
NM
GaitSet [ 4 ]
65.7
91.7
89.1
90.7
66.7
91.9
89.4
92.1
84.7
GaitPart [ 12 ]
62.9
92.0
88.9
89.6
59.6
89.9
87.3
90.7
82.6
GaitGL [ 32 ]
73.9
94.3
92.6
93.4
68.1
93.1
91.7
92.6
87.5
STOR [ 37 ]
73.7
94.6
92.0
93.3
73.3
94.1
92.6
92.6
88.3
GaitBase [ 11 ]
68.4
91.6
88.7
91.1
74.9
93.6
88.1
91.7
86.0
Table 1: The Rank-1 accuracy (%) on OccGait for different probe views excluding the identical-view cases. For evaluation, the sequences of NM01 and NM02 for each subject are taken as the gallery. The benchmark adopts None Occlusion (NO), Carrying Occlusion (CA), Crowd Occlusion (CR) and Static Occlusion (ST).
Methods
NO
CO
SO
DO
NM
BG
CL
Mean
NM
BG
CL
Mean
NM
BG
CL
Mean
NM
BG
CL
Mean
GaitSet [ 4 ]
92.4
83.0
65.2
80.2
80.7
69.6
50.4
66.9
86.0
76.6
57.8
73.5
85.7
72.7
52.7
70.4
GaitPart [ 12 ]
92.3
84.9
68.9
82.0
80.2
70.6
53.3
68.1
85.0
77.3
60.5
74.3
80.7
68.4
52.4
67.2
GaitGL [ 32 ]
94.5
89.2
75.3
86.4
84.4
74.8
57.7
72.3
87.4
81.8
66.9
78.7
86.2
76.2
61.5
74.6
STOR [ 37 ]
95.9
90.9
77.1
88.0
88.8
80.7
64.3
77.9
91.2
85.6
69.9
82.3
93.2
86.3
70.1
83.2
GaitBase [ 11 ]
94.4
88.5
68.7
83.9
87.6
78.0
57.5
74.4
90.1
82.7
62.0
78.3
89.9
80.4
58.5
76.3
Table 2: The Rank-1 accuracy (%) on OccCAISA-B across different views, excluding the identical-view cases. The NO, CO, SO, and DO denote the testing sets of Non-Occlusion, Crowd Occlusion, Static Occlusion, and Detection Occlusion accordingly. Based on the walking condition, probe sequences are grouped into Normal Walking (NM), Carrying Bags (BG), and Cloth-changing Condition (CL).
Method
Venue
Gait3D
GREW
Rank-1
mAP
Rank-1
Rank-5
GaitSet [ 4 ]
AAAI19
36.7
30.0
46.3
63.6
GaitPart [ 12 ]
CVPR20
28.2
47.6
44.0
60.7
GaitGL [ 32 ]
ICCV21
29.7
22.3
47.3
63.6
SMPLGait [ 62 ]
CVPR22
46.3
37.2
-
-
MTSGait [ 62 ]
MM22
48.7
37.6
55.3
71.3
Table 3: Comparisons on Gait3D and GREW.
OccGait
OccCASIA-B
Gait3D
MTE
MAE
NM
CA
CR
ST
NO
CO
SO
DO
Rank-1
mAP
87.7
69.0
75.1
77.2
85.8
75.9
79.6
80.0
59.2
48.6
✓
89.7
78.3
77.3
81.2
88.2
76.9
82.1
83.6
66.2
56.6
✓
89.4
78.4
77.4
81.8
87.6
78.3
81.7
82.5
67.2
57.3
✓
✓
91.4
82.1
79.9
84.7
89.5
80.5
83.9
85.3
71.3
62.5
Table 4: Impact of MTE and MAE on OccGait, OccCASIA-B and Gait3D.
MTE
MAE
NM
CA
CR
ST
Mean
✓
1
89.5
78.3
78.0
82.1
82.0
✓
8
90.6
80.7
79.1
83.3
83.4
✓
16
91.4
82.1
79.9
84.7
84.5
✓
32
90.8
81.2
79.9
84.6
84.1
Table 5: The number of Action Experts in MAE on OccGait.
Figure 6: The visualization of action composition. Here are an occluded gait sequence, action proposals and the selection map of action anchors. Alphabet denotes the action proposal. Each rectangle denotes one action proposal, and each row within one action proposal denotes an action anchor.
Figure 7: The model comparisons on accuracy and efficiency. Rank-1 (%), Param. (M) and FLOPs. (G) on Gait3D.