BioGait-VLM: A Tri-Modal Vision-Language-Biomechanics Framework for Interpretable Clinical Gait Assessment
Authors: Erdong Chen, Yuyang Ji, Jacob K. Greenberg, Benjamin Steel, Faraz Arkam, Abigail Lewis, Pranay Singh, Feng Liu
Organizations: Department of Computer Science, Drexel University · Department of Neurological Surgery, Washington University · University of California, Berkeley
Video-based Clinical Gait Analysis often suffers from poor generalization as models overfit environmental biases instead of capturing pathological motion. To address this, we propose BioGait-VLM, a tri-modal Vision-Language-Biomechanics framework for interpretable clinical gait assessment. Unlike standard video encoders, our architecture incorporates a Temporal Evidence Distillation branch to capture rhythmic dynamics and a Biomechanical Tokenization branch that projects 3D skeleton sequences into language-aligned semantic tokens. This enables the model to explicitly reason about joint mechanics independent of visual shortcuts. To ensure rigorous benchmarking, we augment the public GAVD dataset with a high-fidelity Degenerative Cervical Myelopathy (DCM) cohort to form a unified 8-class taxonomy, establishing a strict subject-disjoint protocol to prevent data leakage. Under this setting, BioGait-VLM achieves state-of-the-art recognition accuracy. Furthermore, a blinded expert study confirms that biomechanical tokens significantly improve clinical plausibility and evidence grounding, offering a path toward transparent, privacy-enhanced gait assessment.
Figures & tables
Figure 1: Overview of BioGait-VLM. The framework integrates three streams: (Top) Temporal Evidence Distillation (TED) for rhythmic dynamics; (Middle) Visual Context; and (Bottom) Biomechanical Tokenization, projecting 3D kinematics into semantic tokens. Fused within a frozen LVLM, these modalities enable state-of-the-art classification and evidence-grounded clinical reporting.
Figure 2: DCM Dataset Acquisition Setup. (a) The recording environment at the outpatient clinic. The scene includes realistic background clutter ( e.g. , treadmills, medical equipment) rather than a sterile lab background, providing a rigorous testbed for evaluating model robustness against visual shortcuts. (b) Top-down schematic of the standardized capture protocol. A tripod-mounted RGB camera is positioned to capture the sagittal (side) view of the patient traversing the instrumented gait mat, ensuring consistent biomechanical visibility.
Method
Acc. (%)
Macro F1
Per-Class F1 Score (%)
DCM
Myo.
Abnorm.
CP
Park.
Norm.
Style
Exer.
End-to-End Video Encoders (Fine-tuned)
SlowFast [ 4 ]
32.2
31.1
88.2
54.1
21.5
0.0
57.1
0.0
3.9
24.3
TSN [ 22 ]
37.1
35.9
100.0
47.6
23.4
12.1
75.0
0.0
0.0
29.2
General-Purpose LVLMs (Zero-Shot)
Qwen3-VL-2B [ 1 ]
21.9
7.8
0.0
0.0
46.7
0.0
0.0
15.7
0.0
0.0
Table 1: Results & Ablation. Performance on the 8 -class clinical gait benchmark. Top: Comparison against state-of-the-art video and VLM baselines. Bottom: Ablation study isolating the impact of the Temporal Evidence Distillation (TED) and Biomechanical Tokenization branches. Note: Test set class distribution is imbalanced; consequently, F1 scores on underrepresented classes may exhibit higher variance.
Figure 3: Qualitative Visualization of Interpretable Reasoning. Our BioGait-VLM generates detailed clinical rationales anchored in quantitative evidence. (Top) Myopathic: The model identifies “muscle weakness” via specific metrics, citing “high steps per minute (173.68)” and “increased double support time.” (Middle) Abnormal: It detects potential coordination issues by calculating precise asymmetry: “left step time significantly shorter (0.5s vs. 0.81s)” and “imbalance in limb propulsion.” (Bottom) Abnormal: It distinguishes a different irregularity by counting specific events, noting “more toe-offs on the right side (4) compared to the left (3)” and quantifying swing percentage. These examples demonstrate how Biomechanical Tokenization enables granular, evidence-based differentiation even within broad diagnostic categories.
Model Variant
Grounding
Explainability
Usefulness
Consistency
InternVL3.5-1B [ 28 ] (Zero-Shot)
2.31
2.98
2.46
3.85
BioGait-VLM ( w/o Biomechanical Branch )
1.98
2.12
1.83
2.13
BioGait-VLM (Full)
4.23
3.98
3.69
4.15
Table 2: Human Expert Evaluation. Average likert scores (Range: 1–5, higher is better). Our Biomechanical Tokenization Branch significantly boosts Evidence Grounding and Usefulness compared to vision-only approaches, reducing hallucinations.
Figure 4: Interface for Blinded Expert Evaluation. To ensure unbiased assessment, we developed a custom web-based annotation tool. Clinicians view the patient gait video alongside three anonymized and randomized model-generated reports (labeled Diagnosis A, B, C). Experts rate each rationale independently on four clinical dimensions using Likert scales before selecting the superior model, ensuring that rankings are based solely on clinical quality rather than model identity.
Current automated gait analysis for clinical applications relies on uninterpretable black-box classifiers. Although Vision-Language Models (VLMs) offer strong reasoning capabilities, applying them directly to gait videos often leads to hallucinations, because they struggle to measure subtle geometric deviations from raw visual contexts. To address this, we introduce DrGait, a training-free agentic framework that shifts the VLM's role from a direct visual reasoner to a clinical planner. DrGait decouples semantic reasoning from geometric perception through a structured Triage-Verification-Synthesis (TVS) workflow. Given an input video and a set of basic spatiotemporal metrics, the DrGait agent first performs a heuristic triage to propose diagnostic hypotheses, which are then verified by autonomously calling deterministic biomechanical tools that operate on reconstructed 3D mesh trajectories, segmented 2D pose tracks, and event-centered video evidence. Finally, a closed-loop mechanism recursively updates the agent's reasoning context based on the feedback. By anchoring VLM's reasoning in verifiable geometric and temporal measurements, DrGait reduces hallucinations, achieving competitive diagnostic accuracy while generating transparent and audit-ready clinical reports.
What is a gait? Appearance-based gait networks consider a gait as the human shape and motion information from images. Model-based gait networks treat a gait as the human inherent structure from points. However, the considerations remain vague for humans to comprehend truly. In this work, we introduce a novel paradigm Vocabulary-Guided Gait Recognition, dubbed Gait-World, which attempts to explore gait concepts through human vocabularies with Vision-Language Models (VLMs). Although VLMs have achieved the remarkable progress in various vision tasks, the cognitive capability regarding gait modalities remains limited. The success element in Gait-World is the proper vocabulary prompt where this paradigm carefully selects gait cycle actions as Vocabulary Base, bridging the gait and vocabulary feature spaces and further promoting human understanding for the gait. How to extract gait features? Although previous gait networks have made significant progress, learning solely from gait modalities on limited gait databases makes it difficult to learn universal gait features for practicality. Therefore, we propose the first Gait-World model, dubbed α-Gait, which guides the gait network learning with vocabulary knowledge from VLMs. However, due to the heterogeneity of the modalities, directly integrating vocabulary and gait features is highly challenging as they reside in different embedding spaces. To address the issues, α-Gait designs Vocabulary Relation Mapper and Gait Fine grained Detector to map and establish vocabulary relations in the gait space for detecting corresponding gait features. Extensive experiments on CASIA-B, CCPG, SUSTech1K, Gait3D and GREW reveal the potential value and research directions of vocabulary information from VLMs in the gait field.
Panjian Huang, Saihui Hou, Chunshui Cao +2
School of Artificial Intelligence, Beijing Normal University
Walking sequences reveal human-interpretable attributes related to body structure, posture, and motion. Gait attribute recognition provides a concrete route to fine-grained gait understanding by predicting these attributes. However, existing benchmarks provide limited fine-grained annotations, while classifier-based gait attribute recognition methods typically represent candidate classes through learned output weights rather than directly matching visual evidence with textual class representations. To address the annotation gap, we present RA-GAR, a large-scale, richly annotated multimodal benchmark for gait attribute recognition, containing 1,001 subjects, 242,729 walking sequences, and annotations for 15 static and motion gait attributes. We further propose Gait-VL, an attribute-centric vision-language alignment framework that recognizes gait attributes by matching attribute-specific multi-frame visual evidence with textual class prototypes. On the visual side, attribute-specific learnable queries aggregate RGB features across spatial locations and frames into per-attribute visual embeddings. On the language side, diverse descriptions expanded by a large language model are combined with context about relevant body regions to construct textual class prototypes. Beyond attribute recognition, we evaluate the transferability of the attribute-pretrained visual encoder by using it to replace generic upstream large vision models in identityoriented gait recognition pipelines; attribute pre-training uses no identity classification or metric-learning objectives. Experiments demonstrate state-of-the-art attribute recognition performance on RA-GAR and consistent improvements in downstream identity recognition across multiple pipelines and benchmarks, supporting the effectiveness and transferability of attribute-centric gait representation learning.
Saihui Hou, Chenye Wang, Wenpeng Lang +5
School of Artificial Intelligence, Beijing Normal University · The Hong Kong University of Science and Technology · Zhejiang University +3