The Gait Signature of Frailty: Transfer Learning based Deep Gait Models for Scalable Frailty Assessment
Authors: Laura McDaniel, Basudha Pal, Crystal Szczesny, Yuxiang Guo, Zhaoyang Wang, Ryan Roemmich, Peter Abadir, Rama Chellappa
Organizations: Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA. · Division of Geriatrics and Gerontology Medicine, Johns Hopkins University School of Medicine, Baltimore, MD, USA. · Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA. · Center for Movement Studies, Kennedy Krieger Institute, Baltimore, MD, USA. · Department of Physical Medicine and Rehabilitation, Johns Hopkins University School of Medicine, Baltimore, MD, USA. · Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA.
Frailty is a condition in aging medicine characterized by diminished physiological reserve and increased vulnerability to stressors. However, frailty assessment remains subjective, heterogeneous, and difficult to scale in clinical practice. Gait is a sensitive marker of biological aging, capturing multisystem decline before overt disability. Yet the application of modern computer vision to gait-based frailty assessment has been limited by small, imbalanced datasets and a lack of clinically representative benchmarks. In this work, we introduce a publicly available silhouette-based frailty gait dataset collected in a clinically realistic setting, spanning the full frailty spectrum and including older adults who use walking aids. Using this dataset, we evaluate how pretrained gait recognition models can be adapted for frailty classification under limited data conditions. We study both convolutional and hybrid attention-based architectures and show that predictive performance depends primarily on how pretrained representations are transferred rather than architectural complexity alone. Across models, selectively freezing low-level gait representations while allowing higher-level features to adapt yields more stable and generalizable performance than either full fine-tuning or rigid freezing. Conservative handling of class imbalance further improves training stability, and combining complementary learning objectives enhances discrimination between clinically adjacent frailty states. Interpretability analyses reveal consistent model attention to lower-limb and pelvic regions, aligning with established biomechanical correlates of frailty. Together, these findings establish gait-based representation learning as a scalable, non-invasive, and interpretable framework for frailty assessment and support the integration of modern biometric modeling approaches into aging research and clinical practice.
Figures & tables
Figure 1: The figure presents the overall framework from clinical motivation to model evaluation. Frailty is introduced as a multidimensional geriatric syndrome motivating objective gait based assessment. Clinical labels are derived using the Fried frailty phenotype defined by five criteria exhaustion, weight loss, slow gait speed, weak grip strength, and low energy, with mobility impairment particularly slow gait speed serving as a core marker of physical frailty. Gait data are collected during a ten meter self paced walk using markerless capture, followed by segmentation and silhouette frame sequencing to construct a labeled silhouette dataset. The conceptual trajectory of age related physical frailty highlights progressive gait deterioration as a primary functional manifestation. The dataset is used to train and compare SwinGait and DeepGait architectures under different finetuning strategies. The bottom panel summarizes demographic characteristics of the study cohort stratified by gender.
Experiment
Micro AUC (%)
Cohen’s Kappa
CNN
PE
Swin S1
Swin S2
SwinGait M1
0.7721 ± 0.0664
0.5923 ± 0.1128
×
×
×
×
SwinGait M2
0.7790 ± 0.0910
0.6190 ± 0.1465
✓
×
×
×
SwinGait M3
0.7513 ± 0.0838
0.5903 ± 0.1301
✓
✓
×
×
SwinGait M4
0.7250 ± 0.0865
0.5103 ± 0.2126
✓
✓
✓
×
SwinGait M5
0.6316 ± 0.0713
0.2110 ± 0.1447
✓
✓
✓
✓
Table 1: Performance summary under different freezing strategies. Patch Embedding (PE) is treated as a learnable projection layer. Here ✓denotes freezing while × denotes trainable.
Experiment
Frozen Layers
Micro AUC (%)
Cohen’s Kappa
DeepGaitV2 D0
None
0.7348 ± 0.0548
0.5787 ± 0.0335
DeepGaitV2 D1
Layer 0
0.7788 ± 0.0312
0.5228 ± 0.0889
DeepGaitV2 D2
Layers 0–1
0.7612 ± 0.0594
0.5657 ± 0.0764
DeepGaitV2 D3
Layers 0–2
0.7407 ± 0.0306
0.4947 ± 0.1241
DeepGaitV2 D4
Layers 0–3
0.7458 ± 0.0623
0.5308 ± 0.0584
DeepGaitV2 D5
Layers 0–4
0.6961 ± 0.0567
0.4412 ± 0.0811
Table 2: Performance summary for DeepGaitV2 under different backbone freezing strategies. Convolutional layers are indexed from shallow (Layer 0) to deep (Layer 4).
Backbone
Configuration
Frozen Components
Class Weights
Micro AUC
Cohen’s Kappa
SwinGait
Frozen CNN (M2)
CNN frozen; PE, S1, S2 trainable
No
0.7790 ± 0.0910
0.6190 ± 0.1465
SwinGait
Frozen CNN (M2)
CNN frozen; PE, S1, S2 trainable
Yes
0.7683 ± 0.0858
0.5658 ± 0.0824
SwinGait
Unfrozen CNN
All backbone components trainable
No
0.7721 ± 0.0664
0.5923 ± 0.1128
SwinGait
Unfrozen CNN
All backbone components trainable
Yes
0.7631 ± 0.0685
0.6091 ± 0.0981
DeepGaitV2
Partially Frozen (D1)
Layer 0 frozen
No
0.7788 ± 0.0312
0.5228 ± 0.0889
DeepGaitV2
Partially Frozen (D1)
Layer 0 frozen
Yes
0.7852 ± 0.0558
0.5248 ± 0.0765
Table 3: Transfer-learning comparison between SwinGait and DeepGaitV2 under selected best freezing strategies with and without inverse square-root class weighting. For SwinGait, the “Frozen CNN” configuration corresponds to M2 (CNN frozen, patch embedding and both Swin stages trainable). For DeepGaitV2, the partially frozen configuration corresponds to freezing convolutional Layer 0.
Figure 2: Exemplar of gradient activation maps for best performing models. Rows 1 and 2 : SwinGait M2 and its corresponding GradCAM. Rows 3 and 4 : DeepGait D1 and its corresponding GradCAM.
Figure 3: Overview of the proposed frailty assessment framework. (a) In-clinic RGB videos are converted to privacy-preserving silhouette sequences using Detectron2. (b) Two transfer-learning backbones, DeepGaitV2 and SwinGait, extract gait representations for frailty prediction. In DeepGaitV2, the Basic Block 2D consists of two spatial convolution layers with batch normalization and ReLU activation, followed by an identity shortcut, while the Basic Block P3D replaces full 3D convolution with a pseudo-3D design composed of spatial and temporal convolutions, optional downsampling, synchronized batch normalization, and a 3D residual shortcut. (c) Clinician-facing interpretation of the predicted frailty status and potential downstream applications, including screening, longitudinal monitoring, and care planning.
Video-based Clinical Gait Analysis often suffers from poor generalization as models overfit environmental biases instead of capturing pathological motion. To address this, we propose BioGait-VLM, a tri-modal Vision-Language-Biomechanics framework for interpretable clinical gait assessment. Unlike standard video encoders, our architecture incorporates a Temporal Evidence Distillation branch to capture rhythmic dynamics and a Biomechanical Tokenization branch that projects 3D skeleton sequences into language-aligned semantic tokens. This enables the model to explicitly reason about joint mechanics independent of visual shortcuts. To ensure rigorous benchmarking, we augment the public GAVD dataset with a high-fidelity Degenerative Cervical Myelopathy (DCM) cohort to form a unified 8-class taxonomy, establishing a strict subject-disjoint protocol to prevent data leakage. Under this setting, BioGait-VLM achieves state-of-the-art recognition accuracy. Furthermore, a blinded expert study confirms that biomechanical tokens significantly improve clinical plausibility and evidence grounding, offering a path toward transparent, privacy-enhanced gait assessment.
Erdong Chen, Yuyang Ji, Jacob K. Greenberg +5
Department of Computer Science, Drexel University · Department of Neurological Surgery, Washington University · University of California, Berkeley
Frailty and functional decline strongly influence treatment tolerance and outcomes in older patients with cancer, yet assessment is typically limited to infrequent clinic visits. We propose a multimodal wearable framework to estimate frailty-related functional change between visits in elderly breast cancer patients enrolled in the multicenter CARDIOCARE study. Free-living smartwatch physical activity and sleep features are combined with ECG-derived heart rate variability (HRV) features from a chest strap and organized into patient-horizon bags aligned to month 3 (M3) and month 6 (M6) follow-ups. Our innovation is an attention-based multiple instance learning (MIL) formulation that fuses irregular, multimodal wearable instances under real-world missingness and weak supervision. An attention-based MIL model with modality-specific multilayer perceptron (MLP) encoders with embedding dimension 128 aggregates variable-length and partially missing longitudinal instances to predict discretized change-from-baseline classes (worsened, stable, improved) for FACIT-F and handgrip strength. Under subject-independent leave-one-subject-out (LOSO) evaluation, the full multimodal model achieved balanced accuracy/F1 of 0.68 +/- 0.08/0.67 +/- 0.09 at M3 and 0.70 +/- 0.10/0.69 +/- 0.08 at M6 for handgrip, and 0.59 +/- 0.04/0.58 +/- 0.06 at M3 and 0.64 +/- 0.05/0.63 +/- 0.07 at M6 for FACIT-F. Ablation results indicated that smartwatch activity and sleep provide the strongest predictive information for frailty-related functional changes, while HRV contributes complementary information when fused with smartwatch streams.
Ioannis Kyprakis, Vasileios Skaramagkas, Georgia Karanasiou +10
Walking sequences reveal human-interpretable attributes related to body structure, posture, and motion. Gait attribute recognition provides a concrete route to fine-grained gait understanding by predicting these attributes. However, existing benchmarks provide limited fine-grained annotations, while classifier-based gait attribute recognition methods typically represent candidate classes through learned output weights rather than directly matching visual evidence with textual class representations. To address the annotation gap, we present RA-GAR, a large-scale, richly annotated multimodal benchmark for gait attribute recognition, containing 1,001 subjects, 242,729 walking sequences, and annotations for 15 static and motion gait attributes. We further propose Gait-VL, an attribute-centric vision-language alignment framework that recognizes gait attributes by matching attribute-specific multi-frame visual evidence with textual class prototypes. On the visual side, attribute-specific learnable queries aggregate RGB features across spatial locations and frames into per-attribute visual embeddings. On the language side, diverse descriptions expanded by a large language model are combined with context about relevant body regions to construct textual class prototypes. Beyond attribute recognition, we evaluate the transferability of the attribute-pretrained visual encoder by using it to replace generic upstream large vision models in identityoriented gait recognition pipelines; attribute pre-training uses no identity classification or metric-learning objectives. Experiments demonstrate state-of-the-art attribute recognition performance on RA-GAR and consistent improvements in downstream identity recognition across multiple pipelines and benchmarks, supporting the effectiveness and transferability of attribute-centric gait representation learning.
Saihui Hou, Chenye Wang, Wenpeng Lang +5
School of Artificial Intelligence, Beijing Normal University · The Hong Kong University of Science and Technology · Zhejiang University +3