SilLang: Improving Gait Recognition with Silhouette Language Encoding

Ruiyi Zhan; Guozhen Peng; Canyu Chen; Jian Lei; Annan Li

cs.CVMar 25, 2026

SilLang: Improving Gait Recognition with Silhouette Language Encoding

Authors: Ruiyi Zhan, Guozhen Peng, Canyu Chen, Jian Lei, Annan Li

Organizations: Beihang University · Tsinghua University

Abstract

Gait silhouettes, which can be encoded into binary gait codes, are widely adopted to representing motion patterns of pedestrian. Recent approaches commonly leverage visual backbones to encode gait silhouettes, achieving successful performance. However, they primarily focus on continuous visual features, overlooking the discrete nature of binary silhouettes that inherently share a discrete encoding space with natural language. Large Language Models (LLMs) have demonstrated exceptional capability in extracting discriminative features from discrete sequences and modeling long-range dependencies, highlighting their potential to capture temporal motion patterns by identifying subtle variations. Motivated by these observations, we explore bridging binary gait silhouettes and natural language within a binary encoding space. However, the encoding spaces of text tokens and binary gait silhouettes remain misaligned, primarily due to differences in token frequency and density. To address this issue, we propose the Contour-Velocity Tokenizer, which encodes binary gait silhouettes while reshaping their distribution to better align with the text token space. We then establish a dual-branch framework termed Silhouette Language Model, which enhances visual silhouettes by integrating discrete linguistic embeddings derived from LLMs. Implemented on mainstream gait backbones, SilLang consistently improves state-of-the-art methods across SUSTech1K, GREW, and Gait3D.

Figures & tables

Figure 1: Binary gait silhouettes are encoded with the proposed Contour-Velocity Tokenizer, thereby forming an implicit silhouette vocabulary within the text token space. Since low-frequency silhouette tokens denote subtle walking patterns, shifts in the token frequency distribution capture fine-grained motion cues.

Figure 2: Aligning and translating binary gait silhouettes into natural language within a shared binary encoding space.

Figure 3: Pipeline of the silhouette language model. (a) The silhouette sequences are encoded with a Contour-Velocity Tokenizer for silhouette language branch, and the Adapter adapts the text embedding ( $\boldsymbol{emb}_{t}$ ) to the visual embedding ( $\boldsymbol{emb}_{v}$ ). In the visual branch, it keeps the same structure as other silhouette-based gait recognition model [ 7 , 27 ] . (b) Extract contour map $\boldsymbol{c}(t)$ and velocity map $\boldsymbol{v}(t)$ from silhouette $\boldsymbol{s}(t)$ . (c) Visualization of the normalized token frequencies in silhouette and text vocabulary when align silhouette to language.

Density	SUSTech1K	GREW	Gait3D	ACR
$P_{s}(1)$	21.2%	25.0%	20.2%	100%
$P_{c}(1)$	4.5%	4.1%	4.1%	19.3%
$P_{v}(1)$	1.8%	2.0%	2.1%	9.0%

Table 1: Analysis of token density in silhouettes, with 1,000 silhouette (

64\times 44

) sequences sampled randomly from SUSTech1K [ 30 ] , GREW [ 50 ] and Gait3D [ 49 ] . Token density

P_{s}(1)

P_{c}(1)

, and

P_{v}(1)

denote the probabilities of a pixel being 1 in the silhouette, contour map, and velocity map, respectively, approximated by their observed frequencies. ACR represents the Average Compression Rate relative to the original token density.

Fusion method	Rank-1	Rank-5	mAP
Attention	73.3	85.7	64.3
Token Concat	71.4	86.0	63.8
Channel Concat	75.2	87.2	67.1

Table 2: Recognition results of different embedding fusion methods on Gait3D with DeepGaitV2-P3D.

Method	Publication	Gait3D			GREW
Method	Publication	Rank-1	Rank-5	mAP	Rank-1	Rank-5
GaitSet [ 3 ]	AAAI 2019	36.7	58.3	30.0	46.3	63.6
GaitPart [ 9 ]	CVPR 2020	28.2	47.6	21.6	44.0	60.7
GaitGL [ 17 ]	ICCV 2021	29.7	48.5	22.3	47.3	63.6
SMPLGait [ 49 ]	CVPR 2022	53.2	71.0	42.4	-	-
DyGait [ 37 ]	ICCV 2023	66.3	80.8	56.4	71.4	83.2

Table 3: Comparison of Rank-1, 5 accuracy and mean Average Precision (%) on Gait3D and GREW, where *-P3D and *-GL refer to the gait backbone, DeepGaitV2-P3D and GLGait, respectively. The superscript

\dagger

denotes results obtained using 128 channels, whereas our experiments are conducted with 64 channels due to limited computational resources.

Method	Probe Sequence (Rank-1)								Overall
Method	Normal	Bag	Clothing	Carrying	Umbrella	Uniform	Occlusion	Night	Rank-1
GaitSet [ 3 ]	69.1	68.3	37.4	65.0	63.1	61.0	67.2	23.0	65.0
GaitPart [ 9 ]	62.2	62.8	33.1	59.5	57.3	54.9	57.2	21.8	59.2
GaitGL [ 17 ]	67.1	66.2	35.9	63.3	61.6	58.1	66.6	17.9	63.1
GaitBase [ 8 ]	81.5	77.5	49.6	75.8	75.6	76.7	81.4	25.9	76.1
DeepGaitV2-P3D ∗ [ 7 ]	86.9	82.4	48.9	79.2	83.3	81.8	85.0	27.7	80.1

Table 4: Results with different attributes on SUSTech1K, where

*

indicates that the method is reproduced with the same configuration.

$\boldsymbol{v}(t)$	$EA$	Rank-1	Rank-5	mAP
✗	✗	73.7	87.1	64.9
✗	✔	74.0	88.1	66.2
✔	✗	74.5	86.9	66.2
✔	✔	75.2	87.2	67.1

Table 5: Ablation study on components of the Contour-Velocity Tokenizer, including the velocity extractor (

\boldsymbol{v}(t)

) and dimension expansion and align module (

EA

Backbone	LLM	Rank-1	Rank-5	mAP
✗	✗	71.6	86.8	64.4
✗	✔	72.1	86.1	64.0
✔	✗	75.2	87.2	67.1
✔	✔	73.5	87.2	65.4

Table 6: Ablation study on branches selected for training, where the Backbone and LLM refer to the gait backbone (DeepGaitV2-P3D) and the large language model (Qwen3-Embedding-0.6B).

$S_{L}$ ( $Height\times Width$ )	Rank-1	Rank-5	mAP
-	74.2	86.9	67.1
704 ( $32\times 22$ )	74.7	86.6	65.6
2816 ( $64\times 44$ )	75.2	87.2	67.1

Table 7: Ablation study on silhouette vocabulary size

S_{L}

(pixels), where - is the reproduced baseline without language model.

Language Model	Rank-1	Rank-5	mAP
Qwen-Embedding-0.6B	76.9	88.2	68.8
Qwen-Embedding-4B	77.2	88.8	69.3

Table 8: Ablation study on different LLM size.

Figure 4: Distribution of normalized embeddings and tokens across different branches. (a) Embeddings in visual, silhouette language and text branches; (b) Input tokens for LLMs.

Figure 5: Performance improvement of SilLang on Gait3D. The number of correctly recognized samples in test set are partitioned according to the frame-length ranges. And the average distance is defined as the mean distance between negative samples minus that between positive samples within each frame-length range.

Figure 6: Normalized frequency heatmaps of $\boldsymbol{s}(t)$ , $\boldsymbol{c}(t)$ , and $\boldsymbol{v}(t)$ on Gait3D [ 49 ] , GREW [ 50 ] , and SUSTech1K [ 30 ] .

Figure 7: Original frequency heatmaps of $\boldsymbol{s}(t)$ on Gait3D [ 49 ] , GREW [ 50 ] and SUSTech1K [ 30 ] .

Figure 8: Visualization of the silhouette vocabulary ( $64\times 44$ ). The frequency heatmap is extracted from Gait3D [ 49 ] . These text tokens are not directly provided as encoded silhouette tokens to the LLMs, which are measured by embedding similarity of tokens for visualization.

Explore similar work

Mar 9, 2026

BioGait-VLM: A Tri-Modal Vision-Language-Biomechanics Framework for Interpretable Clinical Gait Assessment

Video-based Clinical Gait Analysis often suffers from poor generalization as models overfit environmental biases instead of capturing pathological motion. To address this, we propose BioGait-VLM, a tri-modal Vision-Language-Biomechanics framework for interpretable clinical gait assessment. Unlike standard video encoders, our architecture incorporates a Temporal Evidence Distillation branch to capture rhythmic dynamics and a Biomechanical Tokenization branch that projects 3D skeleton sequences into language-aligned semantic tokens. This enables the model to explicitly reason about joint mechanics independent of visual shortcuts. To ensure rigorous benchmarking, we augment the public GAVD dataset with a high-fidelity Degenerative Cervical Myelopathy (DCM) cohort to form a unified 8-class taxonomy, establishing a strict subject-disjoint protocol to prevent data leakage. Under this setting, BioGait-VLM achieves state-of-the-art recognition accuracy. Furthermore, a blinded expert study confirms that biomechanical tokens significantly improve clinical plausibility and evidence grounding, offering a path toward transparent, privacy-enhanced gait assessment.

Erdong Chen, Yuyang Ji, Jacob K. Greenberg +5

Department of Computer Science, Drexel University · Department of Neurological Surgery, Washington University · University of California, Berkeley

Gait Analysis Medical VLMs VLM Interpretability

Ask PDF

Oct 10, 2026

Attribute-Centric Vision-Language Alignment for Fine-Grained Gait Understanding

Walking sequences reveal human-interpretable attributes related to body structure, posture, and motion. Gait attribute recognition provides a concrete route to fine-grained gait understanding by predicting these attributes. However, existing benchmarks provide limited fine-grained annotations, while classifier-based gait attribute recognition methods typically represent candidate classes through learned output weights rather than directly matching visual evidence with textual class representations. To address the annotation gap, we present RA-GAR, a large-scale, richly annotated multimodal benchmark for gait attribute recognition, containing 1,001 subjects, 242,729 walking sequences, and annotations for 15 static and motion gait attributes. We further propose Gait-VL, an attribute-centric vision-language alignment framework that recognizes gait attributes by matching attribute-specific multi-frame visual evidence with textual class prototypes. On the visual side, attribute-specific learnable queries aggregate RGB features across spatial locations and frames into per-attribute visual embeddings. On the language side, diverse descriptions expanded by a large language model are combined with context about relevant body regions to construct textual class prototypes. Beyond attribute recognition, we evaluate the transferability of the attribute-pretrained visual encoder by using it to replace generic upstream large vision models in identityoriented gait recognition pipelines; attribute pre-training uses no identity classification or metric-learning objectives. Experiments demonstrate state-of-the-art attribute recognition performance on RA-GAR and consistent improvements in downstream identity recognition across multiple pipelines and benchmarks, supporting the effectiveness and transferability of attribute-centric gait representation learning.

Saihui Hou, Chenye Wang, Wenpeng Lang +5

School of Artificial Intelligence, Beijing Normal University · The Hong Kong University of Science and Technology · Zhejiang University +3

Gait Recognition Vision-Language Alignment Motion Representation Learning

Ask PDF

Sep 16, 2026

Vocabulary-Guided Gait Recognition

What is a gait? Appearance-based gait networks consider a gait as the human shape and motion information from images. Model-based gait networks treat a gait as the human inherent structure from points. However, the considerations remain vague for humans to comprehend truly. In this work, we introduce a novel paradigm Vocabulary-Guided Gait Recognition, dubbed Gait-World, which attempts to explore gait concepts through human vocabularies with Vision-Language Models (VLMs). Although VLMs have achieved the remarkable progress in various vision tasks, the cognitive capability regarding gait modalities remains limited. The success element in Gait-World is the proper vocabulary prompt where this paradigm carefully selects gait cycle actions as Vocabulary Base, bridging the gait and vocabulary feature spaces and further promoting human understanding for the gait. How to extract gait features? Although previous gait networks have made significant progress, learning solely from gait modalities on limited gait databases makes it difficult to learn universal gait features for practicality. Therefore, we propose the first Gait-World model, dubbed α-Gait, which guides the gait network learning with vocabulary knowledge from VLMs. However, due to the heterogeneity of the modalities, directly integrating vocabulary and gait features is highly challenging as they reside in different embedding spaces. To address the issues, α-Gait designs Vocabulary Relation Mapper and Gait Fine grained Detector to map and establish vocabulary relations in the gait space for detecting corresponding gait features. Extensive experiments on CASIA-B, CCPG, SUSTech1K, Gait3D and GREW reveal the potential value and research directions of vocabulary information from VLMs in the gait field.

Panjian Huang, Saihui Hou, Chunshui Cao +2

School of Artificial Intelligence, Beijing Normal University

Gait Recognition

Ask PDF