cs.CVSep 16, 2026

Pose2Muscle: Structured Spatio-Temporal Decoding for Discrete Muscle Activity Estimation from Human Pose

Authors: Yuepeng Chen, Jiehong Shi, Kaili Zheng, Boyi Zhang, Chenyi Guo, Ji Wu, Xiangling Fu

Organizations: School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications, Beijing, China · Key Laboratory of Trustworthy Distributed Computing and Service (BUPT), Ministry of Education, Beijing, China · Department of Rehabilitation Medicine, Peking Union Medical College Hospital, Beijing, China · Department of Electronic Engineering, Tsinghua University, Beijing, China · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Muscle activity is fundamental to human movement, and understanding its patterns is critical for injury prevention and rehabilitation. Conventional muscle activity monitoring relies on specialized sensors such as surface electromyography, which limits its practicality for long-term real-world use. Existing studies suggest that muscle-related information can be inferred from human pose. However, the substantial gap between externally observable pose and internal muscle activation, limits the accuracy and generalization of current approaches. In this study, we propose Pose2Muscle, a pose-driven framework for discrete muscle activity estimation without requiring sEMG signals at inference time. Instead of directly regressing continuous sEMG signals, Pose2Muscle reformulates muscle estimation as a structured prediction problem over discrete muscle activity states, yielding a more stable and interpretable target space. The framework combines multi-scale spatio-temporal attention to capture motion patterns at complementary spatial and temporal scales with a directed acyclic graph-based decoder that maintains multiple candidate muscle-state hypotheses and performs structured trajectory inference over time. To support this task, we construct PoseEMG-43, a synchronized pose-sEMG dataset containing 2,992 movement instances from 43 daily-life actions performed by 14 participants. Experiments show that Pose2Muscle consistently outperforms representative retrieval- and pose-based baselines. It achieves an Adjacent-level Accuracy of 86.36% and a Pearson correlation coefficient of 0.8821 under the Random Split, and 63.97% and 0.6795, respectively, under the Subject-Level Split. These results demonstrate the feasibility of inferring structured muscle-state patterns from human pose and suggest the potential of Pose2Muscle for muscle-aware movement analysis when direct physiological sensing is impractical

Figures & tables

Figure 1: Key limitations of pose-based muscle activity estimation.
Figure 2: Overview of Pose2Muscle. Dynamic estimation of multi-muscle activity in home and gym scenarios based on easily acquired pose information. At each time step, muscle activities are categorized into 13 discrete levels (A–M), with contraction intensity decreasing accordingly.
Figure 3: The framework of Pose2Muscle. We propose an MS-ST attention module in the motion encoder to learn motion features relevant to muscle activity estimation. The encoded features are fed into the muscle decoder, where a directed acyclic graph is constructed to mitigate uncertainty in muscle activity estimation.
Figure 4: Temporal Sampling Alignment between Pose and Muscle. Point-to-window temporal alignment between muscle-state and pose sequences, where each discrete muscle-state sample is associated with a corresponding local pose window.
Figure 5: Multi-Scale Spatio-Temporal Attention Module. The temporal branch models short- and long-range motion dependencies through intra-window and inter-window attention, respectively, while the spatial branch captures local and global body interactions through intra-limb and inter-limb attention over predefined skeleton partitions.
IDAction NameIDAction Name
0Jogging22Plank with Alternating Arm Raise
1High Knees23Standing Head Turn to Look at Object
2Frog Jumps24Maximal Forward Reach with One Arm Fist Clench
3High Kicks25Lunge Jumps
4Squats26Single-Leg Stand
5Walking with Turning27Biceps Curl
Table 1: List of Actions in the PoseEMG-43 Dataset
Figure 6: Overview of the Pose and sEMG in PoseEMG-43. Raw motion capture data are collected using the OptiTrack system (a-1) and processed using Mosh++ ( Mahmood et al., 2019 ) to obtain a 25-joint skeleton representation (a-2), which is formatted consistently with NTU RGB+D, enabling methodological extensibility and broader applicability. For the sEMG modality, sensors are placed over major muscle groups spanning the upper and lower body, as shown in the right panel.
Figure 7: Distribution of movement cycles across action categories and subjects in the PoseEMG-43 dataset.
StrategyMethodTop-1 ACC ↑\uparrowAdj-ACC ↑\uparrowMAE ↓\downarrowPCC ↑\uparrow
Random SplitRetrieval33.78 ±0.0033.78\,{\scriptstyle\pm\,0.00}74.85 ±0.0074.85\,{\scriptstyle\pm\,0.00}1.0853 ±0.00001.0853\,{\scriptstyle\pm\,0.0000}0.7757 ±0.00000.7757\,{\scriptstyle\pm\,0.0000}
ST-GCN32.23 ±0.2532.23\,{\scriptstyle\pm\,0.25}71.44 ±0.3371.44\,{\scriptstyle\pm\,0.33}1.1770 ±0.01031.1770\,{\scriptstyle\pm\,0.0103}0.7496 ±0.00380.7496\,{\scriptstyle\pm\,0.0038}
PoseFormer38.15 ±0.1738.15\,{\scriptstyle\pm\,0.17}79.26 ±0.2679.26\,{\scriptstyle\pm\,0.26}0.9508 ±0.00770.9508\,{\scriptstyle\pm\,0.0077}0.8174 ±0.00280.8174\,{\scriptstyle\pm\,0.0028}
MotionBERT40.70 ±0.0740.70\,{\scriptstyle\pm\,0.07}82.44 ±0.0582.44\,{\scriptstyle\pm\,0.05}0.8567 ±0.00160.8567\,{\scriptstyle\pm\,0.0016}0.8553 ±0.00060.8553\,{\scriptstyle\pm\,0.0006}
SkateFormer41.89 ±0.0741.89\,{\scriptstyle\pm\,0.07}83.83 ±0.1783.83\,{\scriptstyle\pm\,0.17}0.8173 ±0.00400.8173\,{\scriptstyle\pm\,0.0040}0.8687 ±0.00140.8687\,{\scriptstyle\pm\,0.0014}
MIA-Model40.46 ±0.1140.46\,{\scriptstyle\pm\,0.11}82.63 ±0.1082.63\,{\scriptstyle\pm\,0.10}0.8567 ±0.00220.8567\,{\scriptstyle\pm\,0.0022}0.8508 ±0.00090.8508\,{\scriptstyle\pm\,0.0009}
Table 2: Comparison of different methods on PoseEMG-43. Results under the Random Split are reported as mean ±\pm standard deviation over 10 random seeds, while results under the Subject-Level Split are reported as mean ±\pm standard deviation over five-fold cross-validation. Bold values indicate the best results, and dashed lines separate different categories of methods.
StrategyMethodTop-1 ACC ↑\uparrowAdj-ACC ↑\uparrowMAE ↓\downarrowPCC ↑\uparrow
Random SplitBase (w/o MS-ST & DAG)38.10 ±0.4638.10\,{\scriptstyle\pm\,0.46}78.19 ±0.8078.19\,{\scriptstyle\pm\,0.80}0.9846 ±0.02410.9846\,{\scriptstyle\pm\,0.0241}0.7994 ±0.00940.7994\,{\scriptstyle\pm\,0.0094}
w/o MS-ST40.76 ±0.2840.76\,{\scriptstyle\pm\,0.28}83.22 ±0.3683.22\,{\scriptstyle\pm\,0.36}0.8403 ±0.01050.8403\,{\scriptstyle\pm\,0.0105}0.8561 ±0.00390.8561\,{\scriptstyle\pm\,0.0039}
w/o DAG41.96 ±0.4341.96\,{\scriptstyle\pm\,0.43}84.47 ±0.5084.47\,{\scriptstyle\pm\,0.50}0.8076 ±0.01370.8076\,{\scriptstyle\pm\,0.0137}0.8649 ±0.00440.8649\,{\scriptstyle\pm\,0.0044}
Full Model43.54 ±0.20\mathbf{43.54}\,{\scriptstyle\boldsymbol{\pm}\,\mathbf{0.20}}86.36 ±0.24\mathbf{86.36}\,{\scriptstyle\boldsymbol{\pm}\,\mathbf{0.24}}0.7541 ±0.0067\mathbf{0.7541}\,{\scriptstyle\boldsymbol{\pm}\,\mathbf{0.0067}}0.8821 ±0.0021\mathbf{0.8821}\,{\scriptstyle\boldsymbol{\pm}\,\mathbf{0.0021}}
Subject-Level SplitBase (w/o MS-ST & DAG)22.25 ±0.9222.25\,{\scriptstyle\pm\,0.92}56.50 ±1.3756.50\,{\scriptstyle\pm\,1.37}1.6365 ±0.06291.6365\,{\scriptstyle\pm\,0.0629}0.5429 ±0.02240.5429\,{\scriptstyle\pm\,0.0224}
w/o MS-ST24.55 ±0.9124.55\,{\scriptstyle\pm\,0.91}61.87 ±1.6761.87\,{\scriptstyle\pm\,1.67}1.4381 ±0.06691.4381\,{\scriptstyle\pm\,0.0669}0.6436 ±0.01770.6436\,{\scriptstyle\pm\,0.0177}
Table 3: Ablation study results on the key components. Results under the Random Split are reported as mean ±\pm standard deviation over 10 random seeds, while results under the Subject-Level Split are reported as mean ±\pm standard deviation over five-fold cross-validation. w/o indicates that the corresponding component is removed from the model.
Figure 8: Confusion matrices of muscle activity predictions at each time step. In each matrix, the horizontal axis represents the ground truth labels and the vertical axis represents the predicted labels.
Figure 9: Prediction error distributions under different data-splitting strategies.
Figure 10: Sensitivity analysis of the number of candidate states λ\lambda in Pose2Muscle.
Figure 11: Prediction results for the biceps curl exercise. The masked regions highlight pose discrepancies, which may contribute to differences in muscle activation estimation.
MethodTop-1 ACC ↑\uparrowAdj-ACC ↑\uparrowMAE ↓\downarrowPCC ↑\uparrow
Retrieval (Random)15.59 ±0.0015.59\,{\scriptstyle\pm\,0.00}43.30 ±0.0043.30\,{\scriptstyle\pm\,0.00}2.2145 ±0.00002.2145\,{\scriptstyle\pm\,0.0000}0.3469 ±0.00000.3469\,{\scriptstyle\pm\,0.0000}
Retrieval (Subject-Level)16.28 ±0.9416.28\,{\scriptstyle\pm\,0.94}44.53 ±2.0544.53\,{\scriptstyle\pm\,2.05}2.1168 ±0.09422.1168\,{\scriptstyle\pm\,0.0942}0.3817 ±0.04150.3817\,{\scriptstyle\pm\,0.0415}
Ours (Random)22.72 ±0.90\mathbf{22.72}\,{\scriptstyle\boldsymbol{\pm}\,\mathbf{0.90}}59.74 ±1.67\mathbf{59.74}\,{\scriptstyle\boldsymbol{\pm}\,\mathbf{1.67}}1.4558 ±0.0504\mathbf{1.4558}\,{\scriptstyle\boldsymbol{\pm}\,\mathbf{0.0504}}0.6479 ±0.0161\mathbf{0.6479}\,{\scriptstyle\boldsymbol{\pm}\,\mathbf{0.0161}}
Ours (Subject-Level)21.39 ±0.8321.39\,{\scriptstyle\pm\,0.83}55.67 ±1.6155.67\,{\scriptstyle\pm\,1.61}1.6012 ±0.05911.6012\,{\scriptstyle\pm\,0.0591}0.6032 ±0.03190.6032\,{\scriptstyle\pm\,0.0319}
Table 4: External inference performance of Pose2Muscle on the MEQAD dataset.
Figure 12: Qualitative results of Pose2Muscle on Frog Jumps. Each subplot visualizes sixteen muscles, where the horizontal axis corresponds to time steps and the vertical axis represents muscle activity types. • and • represent the ground truth and predicted labels, respectively.
Figure 13: Qualitative results of Pose2Muscle on Calf Raises. Each subplot visualizes sixteen muscles, where the horizontal axis corresponds to time steps and the vertical axis represents muscle activity types. • and • represent the ground truth and predicted labels, respectively.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Action NameCorrect Action DefinitionFaulty Action DescriptionPrimary Muscle Activation Differences
High KneesPerformed with forefoot landing , an upright and stable torso, and consistent rhythmic stepping throughout the movement.Whole-foot contact occurs at landing, indicating reduced lower-limb control and insufficient propulsion.Reduced activation of the gastrocnemius due to the transition from forefoot to flat-foot landing.
SquatsModerate forward trunk inclination with coordinated hip and knee flexion, heels firmly grounded, and posteriorly maintained center of gravity.Limited ankle dorsiflexion causes heel lift, shifting the center of mass anteriorly and reducing postural stability.Compensatory overactivation of the gastrocnemius , reflecting impaired ankle mobility control.
Biceps CurlElbows remain close to the torso, with elbow flexion driven primarily by the biceps brachii .Excessive shoulder flexion introduces forward displacement of the upper arm, disrupting the intended motion trajectory.Increased activation of the anterior deltoid , indicating compensatory recruitment of non-target muscles.
Appendix
Table 5: Movement Quality Assessment Dataset (MEQAD)
More in this topic

Spatio-Temporal Attention

See all 11 papers

Explore similar work

CardsList
Jun 5, 2026

Physiologically Constrained Musculoskeletal Neural Network for Multi-DoF Joint Kinematics Estimation from Partially Observed sEMG

This paper investigates multi-degrees of freedom (DoF) joint kinematics estimation under partially observed surface electromyography (sEMG), where only a subset of task-relevant muscles can be measured due to anatomical inaccessibility or sensor constraints. A novel musculoskeletal neural network (MSK-NN) is proposed to estimate multi-DoF joint angles while simultaneously inferring activations for both measured and unmeasured muscles. MSK-NN consists of a CNN-based muscle activation estimator and an embedded MSK forward dynamics module, forming a fully differentiable architecture. Unlike existing hybrid neural frameworks that require additional biomechanical labels (e.g., muscle-tendon forces, joint torques), MSK-NN is trained without direct supervision of internal biomechanical variables. A composite physics-physiology loss is designed by incorporating a joint kinematics loss, a data-driven muscle synergy loss, and an anatomy-guided trend loss. The proposed method is evaluated on two-DoF wrist kinematics estimation across three rhythmic motions with unconstrained speed and amplitude, and one random motion. Compared with CNN, Bi-LSTM, CNN-LSTM, and PET baselines, MSK-NN achieves lower normalized root mean square error (NRMSE) and higher coefficient of determination (R2), especially for the random motion. More importantly, the optimized MSK parameters remain within physiological limits, and the estimated activation of an input-excluded muscle exhibits strong temporal agreement with its recorded sEMG envelope, demonstrating the capability of musculoskeletal (MSK)-NN to recover physiologically plausible activations.

Wending Heng, Mingming Zhang, Glen Cooper +1

University of Manchester · 2Southern University of Science and Technology

Surface ElectromyographyPhysics-Informed MLMusculoskeletal Modeling
AskPDF
Sep 30, 2026

EFormer: Temporally Aligned Local Correction for Continuous sEMG-Based Hand Pose Tracking

Surface electromyography (sEMG) provides a wearable, camera-free signal for continuous hand-motion inference. Mapping muscle activity to joint kinematics remains challenging because the recorded waveforms are indirect measurements, their relationship with motion changes over time, and individual anatomy and sensor placement alter the signal distribution. This paper presents EFormer, a residual feature-correction network built on a frozen tracking backbone. EFormer combines a high-rate event branch, temporally aligned local cross-attention, two causal rotary position embedding (RoPE) temporal layers, and a bounded, dynamically gated residual. EFormer receives 16-channel sEMG sampled at 2 kHz and fuses a 64-channel tracking representation at 25 Hz with a 128-channel event representation at 200 Hz. Cross-attention uses a nominal delay of 100 ms, a 300 ms history parameter, and a 50 ms tolerance; its causal mask restricts each query to events occurring 50-400 ms earlier. The correction scale is 0.15. The evaluated continuation-training configuration contains 585,376 trainable parameters and 5,974,508 frozen parameters. On the test set, EFormer achieves an MAE of 0.1546634 rad, an RMSE of 0.24063 rad, and an R^2 of 0.74801, compared with 0.1745326 rad, 0.2715448 rad, and 0.6791103 for the official tracking baseline. EFormer reduces MAE by 11.38% relative to the baseline. The results show that temporally aligned event-feature correction can reduce continuous hand-pose tracking error.

JiaCheng Ge, SiYu Zhang

3D Hand Pose EstimationSurface Electromyography
AskPDF
May 14, 2026

BioHuman: Learning Biomechanical Human Representations from Video

Understanding human motion beyond surface kinematics is crucial for motion analysis, rehabilitation, and injury risk assessment. However, progress in this domain is limited by the lack of large-scale datasets with biomechanical annotations, and by existing approaches that cannot directly infer internal biomechanical states from visual observations. In this paper, we introduce a simulation-based framework for estimating muscle activations from existing motion capture datasets, resulting in BioHuman10M, a large-scale dataset with synchronized video, motion, and activations. Building on BioHuman10M, we propose BioHuman, an end-to-end model that takes monocular video as input and jointly predicts human motion and muscle activations, effectively bridging visual observations and internal biomechanical states. Extensive experiments demonstrate that BioHuman enables accurate reconstruction of both kinematic motion and muscle activity, and generalizes across diverse subjects and motions. We believe our approach establishes a new benchmark for video-based biomechanical understanding and opens up new possibilities for physically grounded human modeling.

Yujun Huo, He Zhang, Chentao Song +3

Beihang University, China · Tsinghua University, China

Human Pose Estimation
AskPDF