BioGait-VLM: A Tri-Modal Vision-Language-Biomechanics Framework for Interpretable Clinical Gait Assessment

Erdong Chen; Yuyang Ji; Jacob K. Greenberg; Benjamin Steel; Faraz Arkam; Abigail Lewis; Pranay Singh; Feng Liu

cs.CVMar 9, 2026

BioGait-VLM: A Tri-Modal Vision-Language-Biomechanics Framework for Interpretable Clinical Gait Assessment

Authors: Erdong Chen, Yuyang Ji, Jacob K. Greenberg, Benjamin Steel, Faraz Arkam, Abigail Lewis, Pranay Singh, Feng Liu

Organizations: Department of Computer Science, Drexel University · Department of Neurological Surgery, Washington University · University of California, Berkeley

Abstract

Video-based Clinical Gait Analysis often suffers from poor generalization as models overfit environmental biases instead of capturing pathological motion. To address this, we propose BioGait-VLM, a tri-modal Vision-Language-Biomechanics framework for interpretable clinical gait assessment. Unlike standard video encoders, our architecture incorporates a Temporal Evidence Distillation branch to capture rhythmic dynamics and a Biomechanical Tokenization branch that projects 3D skeleton sequences into language-aligned semantic tokens. This enables the model to explicitly reason about joint mechanics independent of visual shortcuts. To ensure rigorous benchmarking, we augment the public GAVD dataset with a high-fidelity Degenerative Cervical Myelopathy (DCM) cohort to form a unified 8-class taxonomy, establishing a strict subject-disjoint protocol to prevent data leakage. Under this setting, BioGait-VLM achieves state-of-the-art recognition accuracy. Furthermore, a blinded expert study confirms that biomechanical tokens significantly improve clinical plausibility and evidence grounding, offering a path toward transparent, privacy-enhanced gait assessment.

Figures & tables

Figure 1: Overview of BioGait-VLM. The framework integrates three streams: (Top) Temporal Evidence Distillation (TED) for rhythmic dynamics; (Middle) Visual Context; and (Bottom) Biomechanical Tokenization, projecting 3D kinematics into semantic tokens. Fused within a frozen LVLM, these modalities enable state-of-the-art classification and evidence-grounded clinical reporting.

Figure 2: DCM Dataset Acquisition Setup. (a) The recording environment at the outpatient clinic. The scene includes realistic background clutter ( e.g. , treadmills, medical equipment) rather than a sterile lab background, providing a rigorous testbed for evaluating model robustness against visual shortcuts. (b) Top-down schematic of the standardized capture protocol. A tripod-mounted RGB camera is positioned to capture the sagittal (side) view of the patient traversing the instrumented gait mat, ensuring consistent biomechanical visibility.

Method	Acc. (%)	Macro F1	Per-Class F1 Score (%)
Method	Acc. (%)	Macro F1	DCM	Myo.	Abnorm.	CP	Park.	Norm.	Style	Exer.
End-to-End Video Encoders (Fine-tuned)
SlowFast [ 4 ]	$32.2$	$31.1$	$88.2$	$54.1$	$21.5$	$0.0$	$57.1$	$0.0$	$3.9$	$24.3$
TSN [ 22 ]	$37.1$	$35.9$	$100.0$	$47.6$	$23.4$	$12.1$	$75.0$	$0.0$	$0.0$	$29.2$
General-Purpose LVLMs (Zero-Shot)
Qwen3-VL-2B [ 1 ]	$21.9$	$7.8$	$0.0$	$0.0$	$46.7$	$0.0$	$0.0$	$15.7$	$0.0$	$0.0$

Table 1: Results & Ablation. Performance on the

8

-class clinical gait benchmark. Top: Comparison against state-of-the-art video and VLM baselines. Bottom: Ablation study isolating the impact of the Temporal Evidence Distillation (TED) and Biomechanical Tokenization branches. Note: Test set class distribution is imbalanced; consequently, F1 scores on underrepresented classes may exhibit higher variance.

Figure 3: Qualitative Visualization of Interpretable Reasoning. Our BioGait-VLM generates detailed clinical rationales anchored in quantitative evidence. (Top) Myopathic: The model identifies “muscle weakness” via specific metrics, citing “high steps per minute (173.68)” and “increased double support time.” (Middle) Abnormal: It detects potential coordination issues by calculating precise asymmetry: “left step time significantly shorter (0.5s vs. 0.81s)” and “imbalance in limb propulsion.” (Bottom) Abnormal: It distinguishes a different irregularity by counting specific events, noting “more toe-offs on the right side (4) compared to the left (3)” and quantifying swing percentage. These examples demonstrate how Biomechanical Tokenization enables granular, evidence-based differentiation even within broad diagnostic categories.

Model Variant	Grounding	Explainability	Usefulness	Consistency
InternVL3.5-1B [ 28 ] (Zero-Shot)	$2.31$	$2.98$	$2.46$	$3.85$
BioGait-VLM ( w/o Biomechanical Branch )	$1.98$	$2.12$	$1.83$	$2.13$
BioGait-VLM (Full)	4.23	3.98	3.69	4.15

Table 2: Human Expert Evaluation. Average likert scores (Range: 1–5, higher is better). Our Biomechanical Tokenization Branch significantly boosts Evidence Grounding and Usefulness compared to vision-only approaches, reducing hallucinations.

Figure 4: Interface for Blinded Expert Evaluation. To ensure unbiased assessment, we developed a custom web-based annotation tool. Clinicians view the patient gait video alongside three anonymized and randomized model-generated reports (labeled Diagnosis A, B, C). Experts rate each rationale independently on four clinical dimensions using Likert scales before selecting the superior model, ensuring that rankings are based solely on clinical quality rather than model identity.

Explore similar work

DrGait: Biomechanically Grounded Visual Reasoning for Interpretable Clinical Gait Analysis
Sep 23, 2026Xiangyu Yin, Shiqi Wang, Abrar Alamri +4Vision-Language Models Gait Analysis
University of Pittsburgh
Vocabulary-Guided Gait Recognition
Sep 16, 2026Panjian Huang, Saihui Hou, Chunshui Cao +2Gait Recognition
School of Artificial Intelligence, Beijing Normal University
Attribute-Centric Vision-Language Alignment for Fine-Grained Gait Understanding
Oct 10, 2026Saihui Hou, Chenye Wang, Wenpeng Lang +5Gait Recognition Vision-Language Alignment
School of Artificial Intelligence, Beijing Normal University · The Hong Kong University of Science and Technology · Zhejiang University +3

Sep 23, 2026J/K move · Enter open · S save

DrGait: Biomechanically Grounded Visual Reasoning for Interpretable Clinical Gait Analysis

Xiangyu Yin, Shiqi Wang, Abrar Alamri, Yasir Aljohani+3

University of Pittsburgh

Vision-Language Models Gait Analysis VLM Reasoning Visually Grounded Reasoning

Open paper AskPDF arXiv

Current automated gait analysis for clinical applications relies on uninterpretable black-box classifiers. Although Vision-Language Models (VLMs) offer strong reasoning capabilities, applying them directly to gait videos often leads to hallucinations, because they struggle to measure subtle geometric deviations from raw visual contexts. To address this, we introduce DrGait, a training-free agentic framework that shifts the VLM's role from a direct visual reasoner to a clinical planner. DrGait decouples semantic reasoning from geometric perception through a structured Triage-Verification-Synthesis (TVS) workflow. Given an input video and a set of basic spatiotemporal metrics, the DrGait agent first performs a heuristic triage to propose diagnostic hypotheses, which are then verified by autonomously calling deterministic biomechanical tools that operate on reconstructed 3D mesh trajectories, segmented 2D pose tracks, and event-centered video evidence. Finally, a closed-loop mechanism recursively updates the agent's reasoning context based on the feedback. By anchoring VLM's reasoning in verifiable geometric and temporal measurements, DrGait reduces hallucinations, achieving competitive diagnostic accuracy while generating transparent and audit-ready clinical reports.

Abstract

Explore similar work

DrGait: Biomechanically Grounded Visual Reasoning for Interpretable Clinical Gait Analysis

Vocabulary-Guided Gait Recognition

Attribute-Centric Vision-Language Alignment for Fine-Grained Gait Understanding