Gait silhouettes, which can be encoded into binary gait codes, are widely adopted to representing motion patterns of pedestrian. Recent approaches commonly leverage visual backbones to encode gait silhouettes, achieving successful performance. However, they primarily focus on continuous visual features, overlooking the discrete nature of binary silhouettes that inherently share a discrete encoding space with natural language. Large Language Models (LLMs) have demonstrated exceptional capability in extracting discriminative features from discrete sequences and modeling long-range dependencies, highlighting their potential to capture temporal motion patterns by identifying subtle variations. Motivated by these observations, we explore bridging binary gait silhouettes and natural language within a binary encoding space. However, the encoding spaces of text tokens and binary gait silhouettes remain misaligned, primarily due to differences in token frequency and density. To address this issue, we propose the Contour-Velocity Tokenizer, which encodes binary gait silhouettes while reshaping their distribution to better align with the text token space. We then establish a dual-branch framework termed Silhouette Language Model, which enhances visual silhouettes by integrating discrete linguistic embeddings derived from LLMs. Implemented on mainstream gait backbones, SilLang consistently improves state-of-the-art methods across SUSTech1K, GREW, and Gait3D.
Figures & tables
Figure 1: Binary gait silhouettes are encoded with the proposed Contour-Velocity Tokenizer, thereby forming an implicit silhouette vocabulary within the text token space. Since low-frequency silhouette tokens denote subtle walking patterns, shifts in the token frequency distribution capture fine-grained motion cues.
Figure 2: Aligning and translating binary gait silhouettes into natural language within a shared binary encoding space.
Figure 3: Pipeline of the silhouette language model. (a) The silhouette sequences are encoded with a Contour-Velocity Tokenizer for silhouette language branch, and the Adapter adapts the text embedding ( embt ) to the visual embedding ( embv ). In the visual branch, it keeps the same structure as other silhouette-based gait recognition model [ 7 , 27 ] . (b) Extract contour map c(t) and velocity map v(t) from silhouette s(t) . (c) Visualization of the normalized token frequencies in silhouette and text vocabulary when align silhouette to language.
Density
SUSTech1K
GREW
Gait3D
ACR
Ps(1)
21.2%
25.0%
20.2%
100%
Pc(1)
4.5%
4.1%
4.1%
19.3%
Pv(1)
1.8%
2.0%
2.1%
9.0%
Table 1: Analysis of token density in silhouettes, with 1,000 silhouette ( 64×44 ) sequences sampled randomly from SUSTech1K [ 30 ] , GREW [ 50 ] and Gait3D [ 49 ] . Token density Ps(1) , Pc(1) , and Pv(1) denote the probabilities of a pixel being 1 in the silhouette, contour map, and velocity map, respectively, approximated by their observed frequencies. ACR represents the Average Compression Rate relative to the original token density.
Fusion method
Rank-1
Rank-5
mAP
Attention
73.3
85.7
64.3
Token Concat
71.4
86.0
63.8
Channel Concat
75.2
87.2
67.1
Table 2: Recognition results of different embedding fusion methods on Gait3D with DeepGaitV2-P3D.
Method
Publication
Gait3D
GREW
Rank-1
Rank-5
mAP
Rank-1
Rank-5
GaitSet [ 3 ]
AAAI 2019
36.7
58.3
30.0
46.3
63.6
GaitPart [ 9 ]
CVPR 2020
28.2
47.6
21.6
44.0
60.7
GaitGL [ 17 ]
ICCV 2021
29.7
48.5
22.3
47.3
63.6
SMPLGait [ 49 ]
CVPR 2022
53.2
71.0
42.4
-
-
DyGait [ 37 ]
ICCV 2023
66.3
80.8
56.4
71.4
83.2
Table 3: Comparison of Rank-1, 5 accuracy and mean Average Precision (%) on Gait3D and GREW, where *-P3D and *-GL refer to the gait backbone, DeepGaitV2-P3D and GLGait, respectively. The superscript † denotes results obtained using 128 channels, whereas our experiments are conducted with 64 channels due to limited computational resources.
Method
Probe Sequence (Rank-1)
Overall
Normal
Bag
Clothing
Carrying
Umbrella
Uniform
Occlusion
Night
Rank-1
GaitSet [ 3 ]
69.1
68.3
37.4
65.0
63.1
61.0
67.2
23.0
65.0
GaitPart [ 9 ]
62.2
62.8
33.1
59.5
57.3
54.9
57.2
21.8
59.2
GaitGL [ 17 ]
67.1
66.2
35.9
63.3
61.6
58.1
66.6
17.9
63.1
GaitBase [ 8 ]
81.5
77.5
49.6
75.8
75.6
76.7
81.4
25.9
76.1
DeepGaitV2-P3D ∗ [ 7 ]
86.9
82.4
48.9
79.2
83.3
81.8
85.0
27.7
80.1
Table 4: Results with different attributes on SUSTech1K, where ∗ indicates that the method is reproduced with the same configuration.
v(t)
EA
Rank-1
Rank-5
mAP
✗
✗
73.7
87.1
64.9
✗
✔
74.0
88.1
66.2
✔
✗
74.5
86.9
66.2
✔
✔
75.2
87.2
67.1
Table 5: Ablation study on components of the Contour-Velocity Tokenizer, including the velocity extractor ( v(t) ) and dimension expansion and align module ( EA ).
Backbone
LLM
Rank-1
Rank-5
mAP
✗
✗
71.6
86.8
64.4
✗
✔
72.1
86.1
64.0
✔
✗
75.2
87.2
67.1
✔
✔
73.5
87.2
65.4
Table 6: Ablation study on branches selected for training, where the Backbone and LLM refer to the gait backbone (DeepGaitV2-P3D) and the large language model (Qwen3-Embedding-0.6B).
SL ( Height×Width )
Rank-1
Rank-5
mAP
-
74.2
86.9
67.1
704 ( 32×22 )
74.7
86.6
65.6
2816 ( 64×44 )
75.2
87.2
67.1
Table 7: Ablation study on silhouette vocabulary size SL (pixels), where - is the reproduced baseline without language model.
Language Model
Rank-1
Rank-5
mAP
Qwen-Embedding-0.6B
76.9
88.2
68.8
Qwen-Embedding-4B
77.2
88.8
69.3
Table 8: Ablation study on different LLM size.
Figure 4: Distribution of normalized embeddings and tokens across different branches. (a) Embeddings in visual, silhouette language and text branches; (b) Input tokens for LLMs.
Figure 5: Performance improvement of SilLang on Gait3D. The number of correctly recognized samples in test set are partitioned according to the frame-length ranges. And the average distance is defined as the mean distance between negative samples minus that between positive samples within each frame-length range.
Figure 6: Normalized frequency heatmaps of s(t) , c(t) , and v(t) on Gait3D [ 49 ] , GREW [ 50 ] , and SUSTech1K [ 30 ] .
Figure 7: Original frequency heatmaps of s(t) on Gait3D [ 49 ] , GREW [ 50 ] and SUSTech1K [ 30 ] .
Figure 8: Visualization of the silhouette vocabulary ( 64×44 ). The frequency heatmap is extracted from Gait3D [ 49 ] . These text tokens are not directly provided as encoded silhouette tokens to the LLMs, which are measured by embedding similarity of tokens for visualization.