Modeling spinal motion is fundamental to understanding human biomechanics, yet remains underexplored in computer vision due to the spine's complex multi-joint kinematics and the lack of large-scale 3D annotations. We present a biomechanics-aware keypoint simulation framework that augments existing human pose datasets with anatomically consistent 3D spinal keypoints derived from musculoskeletal modeling. Using this framework, we create the first open dataset, named SIMSPINE, which provides sparse vertebra-level 3D spinal annotations for natural full-body motions in indoor multi-camera capture without external restraints. With 2.14 million frames, this enables data-driven learning of vertebral kinematics from subtle posture variations and bridges the gap between musculoskeletal simulation and computer vision. In addition, we release pretrained baselines covering fine-tuned 2D detectors, monocular 3D pose lifting models, and multi-view reconstruction pipelines, establishing a unified benchmark for biomechanically valid spine motion estimation. Specifically, our 2D spine baselines improve the state-of-the-art from 0.63 to 0.80 AUC in controlled environments, and from 0.91 to 0.93 AP for in-the-wild spine tracking. Together, the simulation framework and SIMSPINE dataset advance research in vision-based biomechanics, motion analysis, and digital human modeling by enabling reproducible, anatomically grounded 3D spine estimation under natural conditions.
Figures & tables
Figure 1: SimSpine annotations. Neutral pose of the simulated spine model (left), divided into three anatomical regions—cervical (pink), thoracic (blue), and lumbar (green)—with 15 annotated landmarks: 9 along the vertebral column, 2 on the skull, 2 at the clavicle joints, and 2 on the shoulder blades (last 4 not shown) . Cervical and thoracic motion is limited to transition junctions (indicated by anatomical axis markers), while intermediate vertebrae remain rigidly coupled. The lumbar segment (L1–L5) is fully articulated, with intervertebral rotations simulated for all degrees of freedom (bottom right). Overall spinal curvature is characterized by thoracic kyphosis ( θk ) and lumbar lordosis ( θl ) angles. Motion was generated using Beaucage-Gauvreau et al. ’s musculoskeletal model [ 5 ] as a function of full-body movement in Human3.6M, with 5 training and 2 validation subjects performing 15 actions.
1.5k videos (447k frames) from 1k walking subjects
Scoliosis detection
Evaluation by medical professionals
✓
USTC&SYSU-Scoliosis [ 79 ]
2025
Radiographs, RGB
Regional Cobb’s angles
1898 images / 1067 patients aged 10-18
Scoliosis detection
Multi-expert agreement
†
Table 1: Comparison of existing spine datasets. Summary of public datasets relevant to spinal imaging, scoliosis analysis, and motion estimation. Our proposed dataset uniquely provides vertebra-level 3D kinematics from RGB videos, bridging clinical imaging, biomechanics, and computer vision domains. Availability: ✓ Public, † On request, ✗ Private.
Figure 2: Biomechanics-aware keypoint simulation pipeline. From synchronized multi-view RGB, a 2D detector [ 31 ] predicts spinal landmarks that are robustly triangulated using calibrated cameras to obtain pseudo-3D spinal keypoints. These pseudo labels are temporally aligned and merged with known Human3.6M [ 27 ] 3D markers (GT 3D Pose). OpenSim inverse kinematics (IK) [ 12 ] fits a subject-scaled full-body model [ 53 , 59 , 5 ] to the merged trajectories. We attach virtual markers to vertebral bodies and, using the IK joint angles and subject-specific anthropometrics, generate anatomically consistent spine keypoints via forward kinematics (FK). The pipeline also outputs biomechanical parameters (e.g., per-vertebra rotations).
Figure 3: Thoracolumbar spine in SimSpine . Left: Distributions of thoracolumbar curvature across actions, defined by the Lumbar Lordotic Angle (LLA, θl ) in the lower back and Thoracic Kyphotic Angle (TKA, θk ) in the upper back. LLA and TKA average within 1 SD at 32–41° and 28–39°, respectively, indicating greater curvature in the lower spine but higher variability in the upper. Values fall within reported biomechanical ranges [ 40 , 16 ] , confirming that SimSpine produces anatomically plausible curvatures and captures expected action-specific postural trends [ 4 , 9 , 69 , 66 , 60 ] . Right: Per-vertebra range of motion (ROM) on y-axis for the three lumbar rotational DOFs. Our simulated data (solid) follows similar trends as reported by White and Panjabi (1978) [ 71 ] (dashed).
Figure 4: Cervical spine in SimSpine . The distributions (per action) remain centered near neutral with task-dependent spread, reflecting that our model uses a single 3-DOF aggregate neck joint while keeping the thoracic/cervical bodies rigid beyond the cervicothoracic junction. This is within the neck ROM reported in [ 14 ] with approximately half coverage, which indicates the presence of only small head movements in the dataset.
Dataset
SpineTrack
Ours
Method
Pretrain
Finetune
AP B
AR B
AP S
AR S
AUC
SpinePose-s
SpineTrack
-
0.792
0.821
0.896
0.908
0.611
SpinePose-m
0.840
0.864
0.914
0.926
0.633
SpinePose-l
0.854
0.877
0.910
0.922
0.633
SpinePose-s-ft
SpineTrack
SpineTrack + Ours
0.788
0.815
0.920
0.929
0.790
SpinePose-m-ft
0.821
0.846
0.928
0.937
0.798
Table 2: Baselines for 2D Spine Pose Estimation. Performance of various CNN and transformer architectures on the SpineTrack and SimSpine benchmarks.
Action
SC
ST
SL
S
B
All
Mean (GT 2D)
12.36
8.80
5.09
9.00
7.07
7.85
Directions
33.69
39.86
40.16
37.88
27.63
31.79
Discussion
36.35
40.55
41.44
39.39
28.33
32.81
Eating
38.47
43.04
35.40
39.48
27.44
32.32
Greeting
30.66
35.19
33.73
33.29
25.57
28.70
Phoning
42.41
44.64
39.14
42.43
27.23
33.40
Table 3: Multiview 3D Spine Reconstruction. MPJPE (mm) across actions and spinal regions using linear triangulation. SC : Cervical, ST : Thoracic, SL : Lumbar, S : Full spine, B : Body, and All: Complete skeleton.
Action
SC
ST
SL
S
B
All
Mean (GT 2D)
0.33
0.43
0.06
0.67
1.81
1.79
Mean (Finetuned)
19.68
20.67
12.21
26.67
24.21
29.53
Mean (Zero-Shot)
26.88
23.40
9.92
39.01
42.25
46.48
Table 4: Triangulation Baseline: P-MPJPE (mm). Same setup as Table 3 , evaluated after similarity alignment.
Train Set
2D
Direct
Discuss
Eating
Greet
Phone
Photo
Pose
Purchase
Sit
SitD
Smoke
Wait
WalkD
WalkT
Walk
Avg
Spine Only
Det.
16.58
17.92
17.79
18.32
21.81
19.74
15.69
17.70
19.28
22.32
18.43
18.82
18.91
16.24
15.96
18.58
Full-Body
Det.
15.55
16.40
14.45
15.98
19.84
17.66
14.25
15.68
15.97
19.23
16.38
17.17
17.41
12.24
12.12
16.28
Spine Only
GT
15.66
17.01
16.05
17.43
20.64
18.34
14.87
16.40
17.91
22.10
16.92
17.78
17.96
15.39
15.28
17.52
Full-Body
GT
11.56
13.05
11.36
13.61
18.06
14.61
10.60
12.39
13.64
17.36
13.25
14.47
14.51
9.15
9.98
13.48
Table 5: Simple Baselines for Monocular 3D Spine Pose Lifting. Evaluation of Martinez et al. ’s lifting model trained on spine-only (15 joints) and full-body (37 joints) keypoint sets. Reported metric: Procrustes-aligned MPJPE (P-MPJPE, mm) per action. Training on full-body joints improves spine localization accuracy. Evaluation is on 15 spine joints only.
P-MPJPE
MPJPE
Train Set
2D
SC
ST
SL
S
SC
ST
SL
S
Spine Only
Det.
12.98
14.29
3.32
18.58
96.88
75.96
18.83
67.70
Full-Body
Det.
12.81
10.12
3.51
16.28
100.50
72.61
18.16
67.39
Spine Only
GT
12.03
13.45
3.32
17.52
77.93
58.70
15.54
53.60
Full-Body
GT
10.94
7.98
2.03
13.48
39.24
26.97
7.76
25.94
Table 6: Monocular 3D Spine Pose Lifting (Component-Wise). Same experimental setup as Table 5 , but results are aggregated across actions and broken down by spine components. Columns report P-MPJPE (mm) for SC (cervical), ST (thoracic), SL (lumbar), and S (full spine). Rows compare training on spine-only (15 joints) vs. full-body (37 joints); 2D inputs are either detected (Det.) or ground truth (GT).
Figure 5: Ablation Study: Mixup Composition. We examine how the fraction of SimSpine used in training influences indoor (AUC) and outdoor (AP) performance. Increasing the SimSpine fraction improves indoor performance up to 10%, while outdoor gains saturate by 2–5%. Per-batch (PB) mixup maintains the best balance between indoor and outdoor metrics, whereas per-epoch (PE) alternation favors one domain at the expense of the other. Sampling only 2% of SimSpine achieves near-saturated results on both datasets, indicating diminishing returns from larger fractions.
Reconstructing biomechanically realistic 3D human motion - recovering both kinematics (motion) and kinetics (forces) - is a critical challenge. While marker-based systems are lab-bound and slow, popular monocular methods use oversimplified, anatomically inaccurate models (e.g., SMPL) and ignore physics, fundamentally limiting their biomechanical fidelity. In this work, we introduce MonoMSK, a hybrid framework that bridges data-driven learning and physics-based simulation for biomechanically realistic 3D human motion estimation from monocular video. MonoMSK jointly recovers both kinematics (motions) and kinetics (forces and torques) through an anatomically accurate musculoskeletal model. By integrating transformer-based inverse dynamics with differentiable forward kinematics and dynamics layers governed by ODE-based simulation, MonoMSK establishes a physics-regulated inverse-forward loop that enforces biomechanical causality and physical plausibility. A novel forward-inverse consistency loss further aligns motion reconstruction with the underlying kinetic reasoning. Experiments on BML-MoVi, BEDLAM, and OpenCap show that MonoMSK significantly outperforms state-of-the-art methods in kinematic accuracy, while for the first time enabling precise monocular kinetics estimation.
Understanding human motion beyond surface kinematics is crucial for motion analysis, rehabilitation, and injury risk assessment. However, progress in this domain is limited by the lack of large-scale datasets with biomechanical annotations, and by existing approaches that cannot directly infer internal biomechanical states from visual observations. In this paper, we introduce a simulation-based framework for estimating muscle activations from existing motion capture datasets, resulting in BioHuman10M, a large-scale dataset with synchronized video, motion, and activations. Building on BioHuman10M, we propose BioHuman, an end-to-end model that takes monocular video as input and jointly predicts human motion and muscle activations, effectively bridging visual observations and internal biomechanical states. Extensive experiments demonstrate that BioHuman enables accurate reconstruction of both kinematic motion and muscle activity, and generalizes across diverse subjects and motions. We believe our approach establishes a new benchmark for video-based biomechanical understanding and opens up new possibilities for physically grounded human modeling.
Yujun Huo, He Zhang, Chentao Song +3
Beihang University, China · Tsinghua University, China
This paper presents a biomechanically interpretable framework for gait analysis using 3D human reconstruction from video data. Unlike conventional keypoint based approaches, the proposed method extracts biomechanically meaningful markers analogous to motion capture systems and integrates them within OpenSim for joint kinematic estimation. To evaluate performance, both spatiotemporal and kinematic gait parameters were analysed against reference marker-based data. Results indicate strong agreement with marker-based measurements, with considerable improvements when compared with pose-estimation methods alone. The proposed framework offers a scalable, markerless, and interpretable approach for accurate gait assessment, supporting broader clinical and real world deployment of vision based biomechanics
Akila Pemasiri, Ethan Goan, Glen Lichtwark +3
Queensland University of Technology, Brisbane · Griffith University, Gold Coast