Human Pose Estimation in Trampoline Gymnastics: How to Improve Performance on Extreme Poses
Authors: Léa Drolet-Roy, Victor Nogues, Bérenger Chedal-Anglay, Sylvain Gaudet, Eve Charbonneau, Mickaël Begon, Lama Séoud
Organizations: Polytechnique Montreal, Canada · Institut National du Sport du Québec, Canada · University of Sherbrooke, Canada · University of Montreal, Canada
Trampoline gymnastics involves extreme human poses and uncommon viewpoints, on which state-of-the art pose estimation models tend to under-perform. We demonstrate that this problem can be addressed by fine-tuning a pose estimation model on a combination of real extreme poses and domain-specific synthetic poses (STP). We generate STP from motion capture recordings of trampoline routines. We propose a pipeline to fit noisy motion capture data to a parametric human model, then generate multi-view realistic images with high-fidelity keypoint labels. The fine-tuned ViTPose model tested on real multi-view images exhibits accuracy improvements in 2D which translate to improved 3D triangulation. In 2D, we obtain a performance similar to state-of-the-art models on the MS COCO validation set while evaluating on significantly more challenging data, bridging the performance gap between common and extreme poses. In 3D, we reduce the MPJPE by 46.1 mm with our best model, which represents an improvement of 42.7% compared to the pretrained ViTPose model. Our code and data are available at https://github.com/VisionICLab/trampoline_syn_data.
Figures & tables
Figure 1: Qualitative comparison of the ViTPose-S without (Baseline) and with fine-tuning (Ours) in extreme cases, 2 views out of 8 are selected. Illustrated triangulations are obtained from 8 views, using Pose2Sim [ 21 ] weighted triangulation function. Ground-truth poses are shown in gray behind the reconstructed poses.
Figure 2: Markers-to-SMPL fitting pipeline. Illustrated frames are not consecutive.
Error (mm) ↓
Processing time ↓
Subject ID
# frames
SOMA
Ours
SOMA
Ours
1
3347
41.1±54.5
28.1 ± 30.2
42m 27s
15m 10s
2
2412
36.4±25.9
17.3 ± 12.0
37m 01s
10m 56s
3a
5079
41.3±44.7
22.7 ± 23.5
49m 54s
21m 27s
3b
4537
30.9±25.3
16.2 ± 9.3
45m 23s
19m 07s
Table 1: Marker-to-closest-vertex after fitting. SOMA uses SMPLX, whereas our method uses SMPL. Standard deviation is provided for the error. Both methods were run on the same machine equipped with an Intel Core i9-13900 CPU and an NVIDIA GeForce RTX 4090 GPU.
Figure 3: Examples of synthetic images in STP (images are cropped for clarity)
Figure 4: Multi-view acquisition setup. A trampolinist in motion is represented as a skeleton at different positions.
Dataset
Athlete(s) ID
# images
FPS
Annotation tool
SRTa
3
103
30
Label Studio
SRTb
3
746
60
Label Studio
MRTa
4, 5
292
12
Label Studio
MRTb
5
264
12
MVPA
Table 2: Summary of manually annotated datasets details
Model
Fine-tuning data
N images
AP
AP 50
AR
AR 50
RTMPose-S
None ( Baseline )
-
40.1
69.8
47.9
77.4
ViTPose-S
None ( Baseline )
-
50.5
72.9
55.6
77.9
STP
3,624
59.3
84.6
64.0
88.0
LSPe
10,000
64.1
87.4
70.2
90.8
LSPe + STP (Ours)
13,624
68.4
88.0
74.5
91.7
RePoGen *
-
59.1
80.9
70.0
88.7
Table 3: Performance of pose estimators pre-trained on COCO and fine-tuned on COCO (5,000 images) and additional datasets - testing on MRT. The results are the mean performance with 5 different seeds. (*) means that we did not train on RePoGen, but only use the provided weights [ 24 ]
Figure 5: Examples of images acquired from a same camera viewpoint showing the trampolinist from (a) bottom, (b) top and (c) side orientations. Images are cropped for clarity.
Figure 6: MPJPE distribution among camera combinations of N cameras. MPJPE is computed on 3D triangulated poses using baseline 2D predictions (gray) or fine-tuned model 2D predictions (blue) . Outliers are represented as circles.
Figure 7: Percentage of camera combinations leading to triangulation error below MPJPE. Triangulations are done using Baseline predictions (gray) and Fine-tuned model predictions (blue) . The percentage of joints is the average for all combinations of N cameras.