Organizations: University of South China, Hengyang, Hunan, China · Harbin Institute of Technology, Harbin, China · Dalian University of Technology, Dalian, China
Real-time monocular human pose estimation is an important sensing component for human-centered intelligent systems. While recent camera-based methods have achieved impressive accuracy, fundamental challenges persist, particularly when visual observations are incomplete, such as during lower-body occlusion or short out-of-frame intervals. In these challenging but common scenarios, vision-only systems often struggle to reconstruct physically plausible lower-limb motion and reliable foot-ground contact. Due to the highly unobtrusive and wearable nature of smart insoles, this paper investigates whether simple plantar pressure and acceleration sensors can effectively resolve these visually induced limitations. To systematically investigate this, we first collect a synchronized multimodal dataset and then propose a part-decoupled temporal fusion framework. The proposed model fuses foot-sensing features into the lower-body branch to compensate for incomplete visual evidence, while a lightweight coordinator exchanges information between the upper- and lower-body temporal representations to maintain whole-body consistency. The framework is evaluated with three representative real-time visual backbones based on 2D keypoints, 3D landmarks, and SMPL-based human mesh recovery. We evaluate our framework on three representative real-time visual backbones. Quantitative experiments demonstrate that incorporating simple foot sensors consistently improves pose accuracy, motion smoothness, and gait-related measurements over vision-only baselines, with the most significant gains observed under occlusion and short visual-dropout conditions. Ultimately, our results demonstrate that unobtrusive foot sensing provides vital and complementary physical observability when camera observations are temporarily unavailable.
Figures & tables
Fig. 1: Overview of the proposed insole-enhanced real-time monocular human pose estimation framework. The system integrates three representative visual backbones (2D keypoints, 3D landmarks, and SMPL mesh recovery) with a four-point pressure-inertial smart insole via a part-decoupled temporal fusion module, producing robust full-body pose and gait-related outputs under clean, occluded, and short out-of-frame conditions.
Fig. 2: Part-decoupled vision–pressure–inertial fusion network. Visual pose features are separated into upper- and lower-body streams, while plantar-pressure and foot-acceleration features are fused into the lower-body stream. A global coordinator exchanges information between upper and lower temporal memories to maintain whole-body consistency.
jumping out and returning; walking out and returning; leaving the view, sitting, and returning
Visual-dropout analysis
Desk-induced real occlusion
2
15K
upper-body free motion with static lower body; walking with lower body fully occluded by a desk
Real-occlusion stress test
TABLE I: Action composition of the new vision–plantar-pressure–acceleration dataset. Detailed action definitions and preprocessing scripts will be provided in the project repository.
Condition
Evaluation Scope
Backbone
Method
MPJPE (mm) ↓
PA-MPJPE (mm) ↓
Jitter ( m/s3 ) ↓
Clean
Standard Subsets
MediaPipe
Vision Temporal
67.55
45.21
81.40
MediaPipe
Ours
64.15
40.62
65.05
YOLOPose26
Vision Temporal
68.13
45.37
80.28
YOLOPose26
Ours
63.94
41.57
69.35
SAT-HMR
Vision Temporal
63.96
42.64
68.29
SAT-HMR
Ours
58.38
38.06
66.11
TABLE II: Main pose estimation results under clean, lower-body occlusion, and visual dropout conditions. MPJPE, PA-MPJPE, and jitter are lower-is-better.
Condition
Evaluation Scope
Backbone
Method
Stance Acc. (%) ↑
Step Dist. Err. (cm) ↓
Cadence Err. (step/min) ↓
Clean
Walking & Jogging
MediaPipe
Vision Temporal
74.15
8.34
11.42
MediaPipe
Ours
79.78
5.54
6.23
YOLOPose26
Vision Temporal
67.89
6.69
16.21
YOLOPose26
Ours
78.16
4.74
4.15
SAT-HMR
Vision Temporal
66.95
6.91
10.50
SAT-HMR
Ours
78.03
5.56
8.36
TABLE III: Gait-related motion parameters under clean, lower-body occlusion, and visual dropout conditions. Stance accuracy is higher-is-better, while step-distance error and cadence error are lower-is-better.
Fig. 3: Qualitative comparison of full-body pose estimation on diverse highly-dynamic actions. Rows from top to bottom correspond to the input monocular images, poses estimated by our proposed method (Ours), and the Xsens reference ground truth, respectively.
Condition
Method
MPJPE (mm) ↓
PA-MPJPE (mm) ↓
Jitter ( m/s3 ) ↓
Clean
Temporal
73.47
48.86
89.02
Ours
68.06
45.78
75.64
Occlusion
Temporal
88.31
59.20
115.69
Ours
74.48
50.43
83.21
TABLE IV: Cross-subject generalization results on held-out test subjects using MediaPipe. All metrics are lower-is-better.
Method
MPJPE (mm) ↓
PA-MPJPE (mm) ↓
Jitter ( m/s3 ) ↓
MediaPipe Temporal
93.17
65.58
178.25
Ours
62.06
45.79
94.00
TABLE V: Real-occlusion stress test on two held-out desk-induced occlusion sequences using MediaPipe. All metrics are lower-is-better.
Fig. 4: Qualitative behavior under short visual dropout. Rows show three temporal frames in order: an in-frame moment followed by two dropout moments. In each row, the columns from left to right are GT, MediaPipe Temporal, and Ours. During dropout, the vision-only baseline has no new visual observation and therefore degenerates to last-observation holding or temporal-prior extrapolation. The proposed model continues to update the articulated pose using pressure and acceleration cues from the smart insole.
Method
MPJPE (mm) ↓
PA-MPJPE (mm) ↓
Jitter ( m/s3 ) ↓
Vision Temporal
67.55
45.21
81.40
Naive Temporal Fusion
66.90
43.09
77.05
w/o Coordinator
66.25
43.88
80.25
w/o Pressure
65.07
42.20
73.00
w/o Acceleration
66.73
43.55
67.38
Ours
64.15
40.62
65.05
TABLE VI: Ablation study on the MediaPipe clean setting. All metrics are lower-is-better.
Device
SoC / Chipset (Node)
Avg.
p95
p99
FPS
(ms)
(ms)
(ms)
vivo X200 Pro
MediaTek Dimensity 9400 (3 nm)
0.44
0.68
0.80
2264.46
Redmi Note 12
Snapdragon 7+ Gen 2 (4 nm)
1.04
1.15
1.33
960.88
OPPO Reno3
MediaTek Dimensity 1000L (7 nm)
1.32
1.47
2.00
758.59
TABLE VII: Mobile runtime overhead of the proposed fusion module. Latency is measured using ONNX Runtime Android on CPU with synchronized feature streams, batch size 1, streaming inference, and 200 warm-up frames. The ONNX model size is 12.10 MB.