Organizations: University of South China, Hengyang, Hunan, China · Harbin Institute of Technology, Harbin, China · Dalian University of Technology, Dalian, China
Real-time monocular human pose estimation is an important sensing component for human-centered intelligent systems. While recent camera-based methods have achieved impressive accuracy, fundamental challenges persist, particularly when visual observations are incomplete, such as during lower-body occlusion or short out-of-frame intervals. In these challenging but common scenarios, vision-only systems often struggle to reconstruct physically plausible lower-limb motion and reliable foot-ground contact. Due to the highly unobtrusive and wearable nature of smart insoles, this paper investigates whether simple plantar pressure and acceleration sensors can effectively resolve these visually induced limitations. To systematically investigate this, we first collect a synchronized multimodal dataset and then propose a part-decoupled temporal fusion framework. The proposed model fuses foot-sensing features into the lower-body branch to compensate for incomplete visual evidence, while a lightweight coordinator exchanges information between the upper- and lower-body temporal representations to maintain whole-body consistency. The framework is evaluated with three representative real-time visual backbones based on 2D keypoints, 3D landmarks, and SMPL-based human mesh recovery. We evaluate our framework on three representative real-time visual backbones. Quantitative experiments demonstrate that incorporating simple foot sensors consistently improves pose accuracy, motion smoothness, and gait-related measurements over vision-only baselines, with the most significant gains observed under occlusion and short visual-dropout conditions. Ultimately, our results demonstrate that unobtrusive foot sensing provides vital and complementary physical observability when camera observations are temporarily unavailable.
Figures & tables
Fig. 1: Overview of the proposed insole-enhanced real-time monocular human pose estimation framework. The system integrates three representative visual backbones (2D keypoints, 3D landmarks, and SMPL mesh recovery) with a four-point pressure-inertial smart insole via a part-decoupled temporal fusion module, producing robust full-body pose and gait-related outputs under clean, occluded, and short out-of-frame conditions.
Fig. 2: Part-decoupled vision–pressure–inertial fusion network. Visual pose features are separated into upper- and lower-body streams, while plantar-pressure and foot-acceleration features are fused into the lower-body stream. A global coordinator exchanges information between upper and lower temporal memories to maintain whole-body consistency.
jumping out and returning; walking out and returning; leaving the view, sitting, and returning
Visual-dropout analysis
Desk-induced real occlusion
2
15K
upper-body free motion with static lower body; walking with lower body fully occluded by a desk
Real-occlusion stress test
TABLE I: Action composition of the new vision–plantar-pressure–acceleration dataset. Detailed action definitions and preprocessing scripts will be provided in the project repository.
Condition
Evaluation Scope
Backbone
Method
MPJPE (mm) ↓
PA-MPJPE (mm) ↓
Jitter ( m/s3 ) ↓
Clean
Standard Subsets
MediaPipe
Vision Temporal
67.55
45.21
81.40
MediaPipe
Ours
64.15
40.62
65.05
YOLOPose26
Vision Temporal
68.13
45.37
80.28
YOLOPose26
Ours
63.94
41.57
69.35
SAT-HMR
Vision Temporal
63.96
42.64
68.29
SAT-HMR
Ours
58.38
38.06
66.11
TABLE II: Main pose estimation results under clean, lower-body occlusion, and visual dropout conditions. MPJPE, PA-MPJPE, and jitter are lower-is-better.
Condition
Evaluation Scope
Backbone
Method
Stance Acc. (%) ↑
Step Dist. Err. (cm) ↓
Cadence Err. (step/min) ↓
Clean
Walking & Jogging
MediaPipe
Vision Temporal
74.15
8.34
11.42
MediaPipe
Ours
79.78
5.54
6.23
YOLOPose26
Vision Temporal
67.89
6.69
16.21
YOLOPose26
Ours
78.16
4.74
4.15
SAT-HMR
Vision Temporal
66.95
6.91
10.50
SAT-HMR
Ours
78.03
5.56
8.36
TABLE III: Gait-related motion parameters under clean, lower-body occlusion, and visual dropout conditions. Stance accuracy is higher-is-better, while step-distance error and cadence error are lower-is-better.
Fig. 3: Qualitative comparison of full-body pose estimation on diverse highly-dynamic actions. Rows from top to bottom correspond to the input monocular images, poses estimated by our proposed method (Ours), and the Xsens reference ground truth, respectively.
Condition
Method
MPJPE (mm) ↓
PA-MPJPE (mm) ↓
Jitter ( m/s3 ) ↓
Clean
Temporal
73.47
48.86
89.02
Ours
68.06
45.78
75.64
Occlusion
Temporal
88.31
59.20
115.69
Ours
74.48
50.43
83.21
TABLE IV: Cross-subject generalization results on held-out test subjects using MediaPipe. All metrics are lower-is-better.
Method
MPJPE (mm) ↓
PA-MPJPE (mm) ↓
Jitter ( m/s3 ) ↓
MediaPipe Temporal
93.17
65.58
178.25
Ours
62.06
45.79
94.00
TABLE V: Real-occlusion stress test on two held-out desk-induced occlusion sequences using MediaPipe. All metrics are lower-is-better.
Fig. 4: Qualitative behavior under short visual dropout. Rows show three temporal frames in order: an in-frame moment followed by two dropout moments. In each row, the columns from left to right are GT, MediaPipe Temporal, and Ours. During dropout, the vision-only baseline has no new visual observation and therefore degenerates to last-observation holding or temporal-prior extrapolation. The proposed model continues to update the articulated pose using pressure and acceleration cues from the smart insole.
Method
MPJPE (mm) ↓
PA-MPJPE (mm) ↓
Jitter ( m/s3 ) ↓
Vision Temporal
67.55
45.21
81.40
Naive Temporal Fusion
66.90
43.09
77.05
w/o Coordinator
66.25
43.88
80.25
w/o Pressure
65.07
42.20
73.00
w/o Acceleration
66.73
43.55
67.38
Ours
64.15
40.62
65.05
TABLE VI: Ablation study on the MediaPipe clean setting. All metrics are lower-is-better.
Device
SoC / Chipset (Node)
Avg.
p95
p99
FPS
(ms)
(ms)
(ms)
vivo X200 Pro
MediaTek Dimensity 9400 (3 nm)
0.44
0.68
0.80
2264.46
Redmi Note 12
Snapdragon 7+ Gen 2 (4 nm)
1.04
1.15
1.33
960.88
OPPO Reno3
MediaTek Dimensity 1000L (7 nm)
1.32
1.47
2.00
758.59
TABLE VII: Mobile runtime overhead of the proposed fusion module. Latency is measured using ONNX Runtime Android on CPU with synchronized feature streams, batch size 1, streaming inference, and 200 warm-up frames. The ONNX model size is 12.10 MB.
Monocular egocentric human pose estimation is essential for ubiquitous activity monitoring. However, understanding the user's absolute location within the environment remains a challenge. Existing methods primarily focus on relative motion from an initial position, and tend not to account for the wearer's absolute location within an environment. Furthermore, inherent scale ambiguity in monocular vision leads to severe translational drift, limiting long-term tracking without specialized multi-sensor hardware. To address this, we propose MapMonoEgo, a novel framework achieving globally consistent human pose estimation solely from a monocular camera by leveraging a pre-scanned 3D point cloud. We also introduce AIST-Living dataset, a new dataset pairing egocentric video with ground-truth motion in a scanned environment. Experiments demonstrate that our approach significantly outperforms the state-of-the-art baseline, proving its utility for practical monitoring tasks without specialized hardware.
Hiroyuki Deguchi, Ryosuke Hori, Kotaro Amaya +3
1Keio University · 2National Institute of Advanced Industrial Science and Technology
We propose Ground Reaction Inertial Poser (GRIP), a method that reconstructs physically plausible human motion using four wearable devices. Unlike conventional IMU-only approaches, GRIP combines IMU signals with foot pressure data to capture both body dynamics and ground interactions. Furthermore, rather than relying solely on kinematic estimation, GRIP uses a digital twin of a person, in the form of a synthetic humanoid in a physics simulator, to reconstruct realistic and physically plausible motion. At its core, GRIP consists of two modules: KinematicsNet, which estimates body poses and velocities from sensor data, and DynamicsNet, which controls the humanoid in the simulator using the residual between the KinematicsNet prediction and the simulated humanoid state. To enable robust training and fair evaluation, we introduce a large-scale dataset, Pressure and Inertial Sensing for Human Motion and Interaction (PRISM), that captures diverse human motions with synchronized IMUs and insole pressure sensors. Experimental results show that GRIP outperforms existing IMU-only and IMU-pressure fusion methods across all evaluated datasets, achieving higher global pose accuracy and improved physical consistency.
Ryosuke Hori, Jyun-Ting Song, Zhengyi Luo +4
Carnegie Mellon University · Keio University · Keio AI Research Center
Recent advancements in visual-inertial motion capture systems have demonstrated the potential of combining monocular cameras with sparse inertial measurement units (IMUs) as cost-effective solutions, which effectively mitigate occlusion and drift issues inherent in single-modality systems. However, they are still limited by metric inaccuracies in global translations stemming from monocular depth ambiguity, and shape-agnostic local motion estimations that ignore anthropometric variations. We present Stereo-Inertial Poser, a real-time motion capture system that leverages a single stereo camera and six IMUs to estimate metric-accurate and shape-aware 3D human motion. By replacing the monocular RGB with stereo vision, our system resolves depth ambiguity through calibrated baseline geometry, enabling direct 3D keypoint extraction and body shape parameter estimation. IMU data and visual cues are fused for predicting drift-compensated joint positions and root movements, while a novel shape-aware fusion module dynamically harmonizes anthropometry variations with global translations. Our end-to-end pipeline achieves over 200 FPS without optimization-based post-processing, enabling real-time deployment. Quantitative evaluations across various datasets demonstrate state-of-the-art performance. Qualitative results show our method produces drift-free global translation under a long recording time and reduces foot-skating effects.
Tutian Tang, Xingyu Ji, Yutong Li +3
School of Computer Science, Shanghai Jiao Tong University · Meta Robotics Institute, SJTU · Shanghai Innovation Institute