Subject-Held-Out Evaluation of Deep Ground Reaction Force Estimators Across Two Laboratories: Leakage and Cross-Laboratory Transfer
Authors: Hernan David Santos, Yinong Chen, Yu Ding, Xudong Zhang, Jian Tao
Organizations: Department of Electrical and Computer Engineering and the Texas A&M Institute of Data Science, Texas A&M University, College Station, TX 77843 USA · Department of Environmental and Occupational Health, Texas A&M University, College Station, TX 77843 USA · H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA 30332 USA · Wm Michael Barnes ’64 Department of Industrial and Systems Engineering, Texas A&M University, College Station, TX 77843 USA · College of Performance, Visualization and Fine Arts and the Texas A&M Institute of Data Science, Texas A&M University, College Station, TX 77843 USA
Deep networks that estimate ground reaction forces and centers of pressure from motion-capture markers are validated within one laboratory, in a seldom-stated coordinate frame, and sometimes with trial-level splits that put the same person in the training and the test set. We measured what each practice costs, and whether such models transfer between laboratories. Three sequence models (convolutional, convolutional-recurrent, and convolutional-transformer) were evaluated on two independent datasets, locomotion (6 subjects) and occupational patient handling (10 subjects), under one pipeline, one tuning procedure, three seeds, and four protocols: trial-level split, leave-one-subject-out, pooled two-laboratory training with matched held-out subjects, and cross-laboratory transfer. At a fixed configuration, trial-level splits overstated the subject-held-out coefficient of determination by +0.040 to +0.194; on occupational data the overstatement exceeded every difference between architectures. With subjects held out, the three models reached values between 0.583 and 0.804; the leader differed between datasets, and gaps lay within subject-to-subject variability. Pooling the laboratories changed neither dataset on matched subjects. Transfer between laboratories was below chance in both directions, including the vertical force, which no in-plane rotation can affect. Leaving force targets in the plate frame while inputs were heading-aligned cost 0.57 of locomotion anteroposterior-force explained variance, enough to make that force look hardest; a Newton's-law correlation detected the mismatch before training. Split protocol and target frame changed reported accuracy more than architecture did. Learned force estimators should be validated with subjects held out, checked for frame consistency before training, and tested across laboratories before deployment claims are made.
Figures & tables
Input
Papers
Held out
Not / n.s.
Frame
2nd site
Optical markers
13
6
0 / 7
2
1
Inertial sensors
13
6
6 / 1
3
2
Video
12
8
1 / 3
2
1
Multi-study
2
1
0 / 1
1
1
All
40
21
7 / 12
8
5
TABLE I : Evaluation protocols of forty published learned GRF estimators (each paper, its split, and the source of each classification: Supplementary Table S11). Held out: test subjects absent from training. Not: trial-level split, or a within-subject result quoted as the headline. n.s.: split unit not stated. Frame: target frame stated. 2nd site: tested on another laboratory or dataset with measured forces.
GroundLink
Patient Handling
Task domain
Locomotion, sport
Occupational handling
Subjects used / recorded
6 / 7
10 / 12
Trials used
279
434
Trials per subject
26–57
33–46
Capture system
Qualisys
Vicon
Marker protocol
24 markers
20 markers
TABLE II : Dataset characteristics.
Fig. 1 : Preprocessing pipeline. Kinematic inputs (top) and kinetic targets (bottom) are processed by separate code paths that must share the per-trial heading rotation θ . Applying θ to the markers, the CoP, and the COM features but not to the horizontal force components (red box) leaves the targets in the plate frame; Section IV-B measures the consequence.
Fig. 2 : Trial-level split versus LOSO, same configuration per model; all scores are means over three training seeds. Numbers give the gap between the trial-split score and the LOSO mean, which is positive for every model.
Fig. 3 : Plate-frame versus co-rotated force targets (per-trial standardization). (a) Mean per-trial correlation of summed force with center-of-mass acceleration, computed before training. (b) LOSO R2 , mean of the two plates; top to bottom in each row: ICF, CNN–LSTM, CNN. Only GL AP force moves, in both panels and for every model; red marks a decrease.
Data
Model
LOSO R2
ML
AP
Trial
Δ
GL
CNN–LSTM
0.804 ± 0.003
0.775
0.769
0.894
+0.090
GL
ICF (grid, d =128)
0.763 ± 0.012
0.770
0.729
–
–
GL
CNN
0.760 ± 0.010
0.784
0.769
0.800
+0.040
GL
ICF (search, d =96)
0.727 ± 0.014
0.719
0.667
0.791
+0.064
GL
Ridge (memoryless)
0.59 ± 0.04
–
–
–
–
PH
ICF (search, d =96)
0.648 ± 0.015
0.451
0.485
0.790
+0.142
TABLE III : Subject-held-out R2 , each model tuned with the same procedure. LOSO: mean over held-out subjects, ± the SD across three seeds of the fold mean; ML and AP are the mean of the two plates. Trial: the same configuration under a trial-level 80/10/10 split, mean of three training seeds; Δ is the leakage inflation (trial minus LOSO).
Fig. 4 : Per-subject LOSO R2 (mean of three seeds) for each held-out subject, sorted from hardest to easiest. The shaded column is the proxy subject used as the tuning objective; on both datasets it sits in the harder half.
GL held-out
PH held-out
Model
Pooled
Δ (improved)
Pooled
Δ (improved)
ICF
0.784 ± 0.066
-0.008 (3/6)
0.715 ± 0.058
-0.004 (2/6)
CNN–LSTM
0.841 ± 0.021
-0.001 (3/6)
0.718 ± 0.070
-0.010 (1/6)
CNN
0.758 ± 0.033
+0.001 (2/6)
0.675 ± 0.080
+0.028 (4/6)
TABLE IV : Pooled two-laboratory training. Dual-holdout R2 on the held-out GL and PH subject of each fold (mean ± SD over six folds), and the paired difference Δ against single-laboratory LOSO on the same subjects, with the number of subjects improved by pooling in parentheses.
Direction
Model
Per-trial
Per-fold
Vertical (per-fold)
GL → PH
ICF
-0.45
-3.73
-0.71
GL → PH
CNN–LSTM
-0.78
-1.91
-0.65
GL → PH
CNN
-0.72
<−10
<−10
PH → GL
ICF
-1.40
-1.08
-0.58
PH → GL
CNN–LSTM
-1.49
-0.35
0.12
PH → GL
CNN
-1.10
<−10
<−10
TABLE V : Cross-laboratory transfer R2 (mean over source-laboratory LOSO models, evaluated on every trial of the other laboratory), co-rotated targets, under per-trial and per-fold input standardization. Vertical-force R2 is shown for the per-fold case; it is invariant to any in-plane rotation.
Fig. 5 : Per-channel cross-laboratory R2 (ICF, per-fold standardization), clipped at −3 for display. Vertical force fails with the horizontal channels.
Result
Model
dmodel
L
FFN
Filters ( k )
Params
Tuned per model and dataset: LOSO and trial split (main-text Tables II and III, Figs. 2 and 4)
GL
ICF (search)
96
2
1024
128
0.93M
GL, PH
ICF (grid)
128
2
512
256
1.61M
GL
CNN–LSTM
–
2
512
256 (7)
4.07M
GL
CNN
–
3
–
512 (7)
2.62M
PH
ICF (search)
96
2
512
256
1.42M
TABLE S1 : Configuration behind each reported result. FFN is the feed-forward width for ICF and the LSTM hidden size for CNN–LSTM; k is the convolution kernel; parameter counts are obtained by instantiating each configuration with the 216-feature input.
Dataset
Targets
AP
ML
Vertical
GL (279 trials)
plate frame
-0.11
0.42
0.48
co-rotated
0.29
0.44
0.48
PH (434 trials)
plate frame
0.01
-0.03
0.21
co-rotated
0.04
-0.08
0.21
TABLE S2 : Newton’s-law frame check: mean per-trial Pearson correlation between summed bilateral force and pelvis (COM) acceleration along the same axis, for plate-frame and co-rotated force targets. Vertical is invariant to the rotation and serves as the control.
Data
Targets
Input std.
Mean
Vert.
ML
AP
CoP x
CoP z
ICF
GL
plate
trial
0.590
0.816
0.682
0.108
0.808
0.536
GL
co-rot.
trial
0.713
0.828
0.688
0.675
0.808
0.566
GL
plate
fold
0.608
0.862
0.716
0.057
0.838
0.570
GL
co-rot.
fold
0.732
0.854
0.722
0.700
0.834
0.550
PH
plate
trial
0.588
0.664
0.483
0.397
0.791
0.607
TABLE S3 : Frame study: LOSO R2 (mean over subjects) under the four combinations of target convention and input standardization, for the ICF at one fixed configuration ( dmodel=64 , L=2 ) and for the CNN–LSTM and CNN at their tuned configurations (seed 42). All main-text results use co-rotated targets with per-fold standardization.
TABLE S6 : Fold-mean LOSO R2 per seed on GL, corrected pipeline, all folds, three seeds.
Config
seed 42
seed 43
seed 44
mean
SD
CNN-LSTM-hpo
0.620
0.616
0.629
0.622
0.007
CNN-hpo
0.577
0.579
0.592
0.583
0.008
ICF-grid128
0.645
0.638
0.642
0.642
0.003
ICF-hpo
0.651
0.632
0.662
0.648
0.015
TABLE S7 : Fold-mean LOSO R2 per seed on PH, corrected pipeline, all folds, three seeds.
Model
Fold
GL held out
PH held out
GL R2
PH R2
ICF
1
gl_2
ph_1
0.725
0.714
ICF
2
gl_3
ph_2
0.854
0.633
ICF
3
gl_4
ph_3
0.767
0.706
ICF
4
gl_5
ph_4
0.701
0.810
ICF
5
gl_6
ph_5
0.796
0.692
ICF
6
gl_7
ph_6
0.860
0.733
TABLE S8 : Pooled dual-holdout CV per fold, corrected pipeline.
Direction
Held-out subject
Own R2
Other-lab R2
GL → PH
gl_2
0.759
-3.322
GL → PH
gl_3
0.851
-6.594
GL → PH
gl_4
0.756
-3.851
GL → PH
gl_5
0.722
-1.926
GL → PH
gl_6
0.808
-3.018
GL → PH
gl_7
0.856
-3.648
TABLE S9 : Cross-laboratory transfer per source-laboratory LOSO model (ICF, corrected pipeline, per-fold input standardization). Own = held-out subject of the training laboratory; Other = all trials of the other laboratory.
Arm
GL R2
GL Δ
GL ML
GL AP
PH R2
PH Δ
PH ML
PH AP
full pipeline
0.740 ± 0.014
–
0.739
0.701
0.623 ± 0.003
–
0.405
0.431
no pelvis-relative markers
0.802 ± 0.002
+0.062
0.747
0.728
0.603 ± 0.013
-0.020
0.408
0.360
no heading alignment (targets in plate frame)
0.665 ± 0.013
-0.076
0.727
0.299
0.645 ± 0.009
+0.022
0.546
0.447
no height scaling
0.747 ± 0.014
+0.006
0.739
0.708
0.622 ± 0.000
-0.001
0.416
0.411
none of the three above
0.608 ± 0.011
-0.132
0.693
0.190
0.652 ± 0.007
+0.029
0.464
0.363
no velocity/acceleration features
0.728 ± 0.009
-0.012
0.731
0.649
0.573 ± 0.007
-0.049
0.336
0.371
TABLE S10 : Preprocessing and training-choice ablation: ICF at the fixed configuration ( dmodel=64 , L=2 ), corrected pipeline, full LOSO, one factor removed per row, three training seeds (42, 43, 44). R2 : mean over seeds of the fold-mean LOSO R2± SD across seeds. Δ : paired difference from the full pipeline on the same seeds. ML and AP: mean horizontal-force R2 over plates and seeds. Mass and height inputs, per-fold input standardization, body-weight target normalization, centre-of-mass kinematics, and force co-rotation (wherever a heading exists) are kept in every arm.
Study (venue)
Input
n
Split
Held out
Frame
Second site
Headline result
Optical markers and marker-derived kinematics
Oh 2013 (J Biomech) (low)
Marker-derived segment kinematics (ANN for double support)
not stated
not stated: not stated (10-fold CV; fold unit not g…
not stated
no
no
R 0.991 vertical, 0.985 AP, 0.918 ML
Johnson 2018 (Med Biol Eng Comput) (low)
Optical marker trajectories (PLS)
not stated
not stated: not stated
not stated
no
no
mean r 0.980 GRF / 0.914 GRM (11 PLS variants)
Johnson 2019 (IEEE TBME) (low)
Optical markers encoded as images (CNN)
not stated
not stated: not stated
not stated
no
no
r 0.988 GRF / 0.972 GRM; rRMSE 4.3% / 7.0%
Mundt 2018 † (ISBS Proc)
Marker-derived joint angles (LSTM)
12
subject: subject-level (unknown subjects) + with…
mixed
no
no
qualitative: higher accuracy for known than unknown subjects
Mundt 2019 (Med Biol Eng Comput) (low)
Joint angles during cutting (FFNN); marker-based implied, n…
55
not stated: not stated
not stated
no
no
not retrieved (paywalled)
TABLE S11 : Evaluation-protocol audit of learned ground-reaction-force, moment and center-of-pressure estimators (2013–2026). Split: unit of the headline split (both = within-subject and cross-subject results reported). Held out: subjects absent from training. Frame: coordinate frame of the force targets stated. Second site: test on a second laboratory or dataset with measured forces. † = not a trained estimator or targets other than GRF/CoP; shown for context, excluded from the counts in the main text. Confidence low = classification from an abstract only.
Capital University of Physical Education and Sports. · Beijing Institute of Technology. · 3Beijing Key Laboratory of Interdisciplinary Intelligent Technologies of Sports, Medicine and Engineering.
Department of Engineering, Faculty of Environment, Science and Economy, University of Exeter, Exeter, EX4 4QF, UK · School of Computer and Communication Sciences, École Polytechnique Fédérale de Lausanne (EPFL), CH-1015 Lausanne, Switzerland · Institute for Manufacturing, Department of Engineering, University of Cambridge, Cambridge CB3 0FS, UK +2