Pose2Muscle: Structured Spatio-Temporal Decoding for Discrete Muscle Activity Estimation from Human Pose

Yuepeng Chen; Jiehong Shi; Kaili Zheng; Boyi Zhang; Chenyi Guo; Ji Wu; Xiangling Fu

cs.CVSep 16, 2026

Pose2Muscle: Structured Spatio-Temporal Decoding for Discrete Muscle Activity Estimation from Human Pose

Authors: Yuepeng Chen, Jiehong Shi, Kaili Zheng, Boyi Zhang, Chenyi Guo, Ji Wu, Xiangling Fu

Organizations: School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications, Beijing, China · Key Laboratory of Trustworthy Distributed Computing and Service (BUPT), Ministry of Education, Beijing, China · Department of Rehabilitation Medicine, Peking Union Medical College Hospital, Beijing, China · Department of Electronic Engineering, Tsinghua University, Beijing, China ·

Abstract

Muscle activity is fundamental to human movement, and understanding its patterns is critical for injury prevention and rehabilitation. Conventional muscle activity monitoring relies on specialized sensors such as surface electromyography, which limits its practicality for long-term real-world use. Existing studies suggest that muscle-related information can be inferred from human pose. However, the substantial gap between externally observable pose and internal muscle activation, limits the accuracy and generalization of current approaches. In this study, we propose Pose2Muscle, a pose-driven framework for discrete muscle activity estimation without requiring sEMG signals at inference time. Instead of directly regressing continuous sEMG signals, Pose2Muscle reformulates muscle estimation as a structured prediction problem over discrete muscle activity states, yielding a more stable and interpretable target space. The framework combines multi-scale spatio-temporal attention to capture motion patterns at complementary spatial and temporal scales with a directed acyclic graph-based decoder that maintains multiple candidate muscle-state hypotheses and performs structured trajectory inference over time. To support this task, we construct PoseEMG-43, a synchronized pose-sEMG dataset containing 2,992 movement instances from 43 daily-life actions performed by 14 participants. Experiments show that Pose2Muscle consistently outperforms representative retrieval- and pose-based baselines. It achieves an Adjacent-level Accuracy of 86.36% and a Pearson correlation coefficient of 0.8821 under the Random Split, and 63.97% and 0.6795, respectively, under the Subject-Level Split. These results demonstrate the feasibility of inferring structured muscle-state patterns from human pose and suggest the potential of Pose2Muscle for muscle-aware movement analysis when direct physiological sensing is impractical

Figures & tables

Figure 1: Key limitations of pose-based muscle activity estimation.

Figure 2: Overview of Pose2Muscle. Dynamic estimation of multi-muscle activity in home and gym scenarios based on easily acquired pose information. At each time step, muscle activities are categorized into 13 discrete levels (A–M), with contraction intensity decreasing accordingly.

Figure 3: The framework of Pose2Muscle. We propose an MS-ST attention module in the motion encoder to learn motion features relevant to muscle activity estimation. The encoded features are fed into the muscle decoder, where a directed acyclic graph is constructed to mitigate uncertainty in muscle activity estimation.

Figure 4: Temporal Sampling Alignment between Pose and Muscle. Point-to-window temporal alignment between muscle-state and pose sequences, where each discrete muscle-state sample is associated with a corresponding local pose window.

Figure 5: Multi-Scale Spatio-Temporal Attention Module. The temporal branch models short- and long-range motion dependencies through intra-window and inter-window attention, respectively, while the spatial branch captures local and global body interactions through intra-limb and inter-limb attention over predefined skeleton partitions.

ID	Action Name	ID	Action Name
0	Jogging	22	Plank with Alternating Arm Raise
1	High Knees	23	Standing Head Turn to Look at Object
2	Frog Jumps	24	Maximal Forward Reach with One Arm Fist Clench
3	High Kicks	25	Lunge Jumps
4	Squats	26	Single-Leg Stand
5	Walking with Turning	27	Biceps Curl

Table 1: List of Actions in the PoseEMG-43 Dataset

Figure 6: Overview of the Pose and sEMG in PoseEMG-43. Raw motion capture data are collected using the OptiTrack system (a-1) and processed using Mosh++ ( Mahmood et al., 2019 ) to obtain a 25-joint skeleton representation (a-2), which is formatted consistently with NTU RGB+D, enabling methodological extensibility and broader applicability. For the sEMG modality, sensors are placed over major muscle groups spanning the upper and lower body, as shown in the right panel.

Figure 7: Distribution of movement cycles across action categories and subjects in the PoseEMG-43 dataset.

Strategy	Method	Top-1 ACC $\uparrow$	Adj-ACC $\uparrow$	MAE $\downarrow$	PCC $\uparrow$
Random Split	Retrieval	$33.78\,{\scriptstyle\pm\,0.00}$	$74.85\,{\scriptstyle\pm\,0.00}$	$1.0853\,{\scriptstyle\pm\,0.0000}$	$0.7757\,{\scriptstyle\pm\,0.0000}$
	ST-GCN	$32.23\,{\scriptstyle\pm\,0.25}$	$71.44\,{\scriptstyle\pm\,0.33}$	$1.1770\,{\scriptstyle\pm\,0.0103}$	$0.7496\,{\scriptstyle\pm\,0.0038}$
	PoseFormer	$38.15\,{\scriptstyle\pm\,0.17}$	$79.26\,{\scriptstyle\pm\,0.26}$	$0.9508\,{\scriptstyle\pm\,0.0077}$	$0.8174\,{\scriptstyle\pm\,0.0028}$
	MotionBERT	$40.70\,{\scriptstyle\pm\,0.07}$	$82.44\,{\scriptstyle\pm\,0.05}$	$0.8567\,{\scriptstyle\pm\,0.0016}$	$0.8553\,{\scriptstyle\pm\,0.0006}$
	SkateFormer	$41.89\,{\scriptstyle\pm\,0.07}$	$83.83\,{\scriptstyle\pm\,0.17}$	$0.8173\,{\scriptstyle\pm\,0.0040}$	$0.8687\,{\scriptstyle\pm\,0.0014}$
	MIA-Model	$40.46\,{\scriptstyle\pm\,0.11}$	$82.63\,{\scriptstyle\pm\,0.10}$	$0.8567\,{\scriptstyle\pm\,0.0022}$	$0.8508\,{\scriptstyle\pm\,0.0009}$

Table 2: Comparison of different methods on PoseEMG-43. Results under the Random Split are reported as mean

\pm

standard deviation over 10 random seeds, while results under the Subject-Level Split are reported as mean

\pm

standard deviation over five-fold cross-validation. Bold values indicate the best results, and dashed lines separate different categories of methods.

Strategy	Method	Top-1 ACC $\uparrow$	Adj-ACC $\uparrow$	MAE $\downarrow$	PCC $\uparrow$
Random Split	Base (w/o MS-ST & DAG)	$38.10\,{\scriptstyle\pm\,0.46}$	$78.19\,{\scriptstyle\pm\,0.80}$	$0.9846\,{\scriptstyle\pm\,0.0241}$	$0.7994\,{\scriptstyle\pm\,0.0094}$
	w/o MS-ST	$40.76\,{\scriptstyle\pm\,0.28}$	$83.22\,{\scriptstyle\pm\,0.36}$	$0.8403\,{\scriptstyle\pm\,0.0105}$	$0.8561\,{\scriptstyle\pm\,0.0039}$
	w/o DAG	$41.96\,{\scriptstyle\pm\,0.43}$	$84.47\,{\scriptstyle\pm\,0.50}$	$0.8076\,{\scriptstyle\pm\,0.0137}$	$0.8649\,{\scriptstyle\pm\,0.0044}$
	Full Model	$\mathbf{43.54}\,{\scriptstyle\boldsymbol{\pm}\,\mathbf{0.20}}$	$\mathbf{86.36}\,{\scriptstyle\boldsymbol{\pm}\,\mathbf{0.24}}$	$\mathbf{0.7541}\,{\scriptstyle\boldsymbol{\pm}\,\mathbf{0.0067}}$	$\mathbf{0.8821}\,{\scriptstyle\boldsymbol{\pm}\,\mathbf{0.0021}}$
Subject-Level Split	Base (w/o MS-ST & DAG)	$22.25\,{\scriptstyle\pm\,0.92}$	$56.50\,{\scriptstyle\pm\,1.37}$	$1.6365\,{\scriptstyle\pm\,0.0629}$	$0.5429\,{\scriptstyle\pm\,0.0224}$
	w/o MS-ST	$24.55\,{\scriptstyle\pm\,0.91}$	$61.87\,{\scriptstyle\pm\,1.67}$	$1.4381\,{\scriptstyle\pm\,0.0669}$	$0.6436\,{\scriptstyle\pm\,0.0177}$

Table 3: Ablation study results on the key components. Results under the Random Split are reported as mean

\pm

standard deviation over 10 random seeds, while results under the Subject-Level Split are reported as mean

\pm

standard deviation over five-fold cross-validation. w/o indicates that the corresponding component is removed from the model.

Figure 8: Confusion matrices of muscle activity predictions at each time step. In each matrix, the horizontal axis represents the ground truth labels and the vertical axis represents the predicted labels.

Figure 9: Prediction error distributions under different data-splitting strategies.

Figure 10: Sensitivity analysis of the number of candidate states $\lambda$ in Pose2Muscle.

Figure 11: Prediction results for the biceps curl exercise. The masked regions highlight pose discrepancies, which may contribute to differences in muscle activation estimation.

Method	Top-1 ACC $\uparrow$	Adj-ACC $\uparrow$	MAE $\downarrow$	PCC $\uparrow$
Retrieval (Random)	$15.59\,{\scriptstyle\pm\,0.00}$	$43.30\,{\scriptstyle\pm\,0.00}$	$2.2145\,{\scriptstyle\pm\,0.0000}$	$0.3469\,{\scriptstyle\pm\,0.0000}$
Retrieval (Subject-Level)	$16.28\,{\scriptstyle\pm\,0.94}$	$44.53\,{\scriptstyle\pm\,2.05}$	$2.1168\,{\scriptstyle\pm\,0.0942}$	$0.3817\,{\scriptstyle\pm\,0.0415}$
Ours (Random)	$\mathbf{22.72}\,{\scriptstyle\boldsymbol{\pm}\,\mathbf{0.90}}$	$\mathbf{59.74}\,{\scriptstyle\boldsymbol{\pm}\,\mathbf{1.67}}$	$\mathbf{1.4558}\,{\scriptstyle\boldsymbol{\pm}\,\mathbf{0.0504}}$	$\mathbf{0.6479}\,{\scriptstyle\boldsymbol{\pm}\,\mathbf{0.0161}}$
Ours (Subject-Level)	$21.39\,{\scriptstyle\pm\,0.83}$	$55.67\,{\scriptstyle\pm\,1.61}$	$1.6012\,{\scriptstyle\pm\,0.0591}$	$0.6032\,{\scriptstyle\pm\,0.0319}$

Table 4: External inference performance of Pose2Muscle on the MEQAD dataset.

Figure 12: Qualitative results of Pose2Muscle on Frog Jumps. Each subplot visualizes sixteen muscles, where the horizontal axis corresponds to time steps and the vertical axis represents muscle activity types. • and • represent the ground truth and predicted labels, respectively.

Figure 13: Qualitative results of Pose2Muscle on Calf Raises. Each subplot visualizes sixteen muscles, where the horizontal axis corresponds to time steps and the vertical axis represents muscle activity types. • and • represent the ground truth and predicted labels, respectively.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Action Name	Correct Action Definition	Faulty Action Description	Primary Muscle Activation Differences
High Knees	Performed with forefoot landing , an upright and stable torso, and consistent rhythmic stepping throughout the movement.	Whole-foot contact occurs at landing, indicating reduced lower-limb control and insufficient propulsion.	Reduced activation of the gastrocnemius due to the transition from forefoot to flat-foot landing.
Squats	Moderate forward trunk inclination with coordinated hip and knee flexion, heels firmly grounded, and posteriorly maintained center of gravity.	Limited ankle dorsiflexion causes heel lift, shifting the center of mass anteriorly and reducing postural stability.	Compensatory overactivation of the gastrocnemius , reflecting impaired ankle mobility control.
Biceps Curl	Elbows remain close to the torso, with elbow flexion driven primarily by the biceps brachii .	Excessive shoulder flexion introduces forward displacement of the upper arm, disrupting the intended motion trajectory.	Increased activation of the anterior deltoid , indicating compensatory recruitment of non-target muscles.

Appendix

Table 5: Movement Quality Assessment Dataset (MEQAD)

Spatio-Temporal Attention

See all 11 papers

Physiologically Constrained Musculoskeletal Neural Network for Multi-DoF Joint Kinematics Estimation from Partially Observed sEMG
Jun 5, 2026Wending Heng, Mingming Zhang, Glen Cooper +1Surface Electromyography Physics-Informed ML
University of Manchester · 2Southern University of Science and Technology
EFormer: Temporally Aligned Local Correction for Continuous sEMG-Based Hand Pose Tracking
Sep 30, 2026JiaCheng Ge, SiYu Zhang3D Hand Pose Estimation Surface Electromyography
BioHuman: Learning Biomechanical Human Representations from Video
May 14, 2026Yujun Huo, He Zhang, Chentao Song +3Human Pose Estimation
Beihang University, China · Tsinghua University, China

Jun 5, 2026J/K move · Enter open · S save

Physiologically Constrained Musculoskeletal Neural Network for Multi-DoF Joint Kinematics Estimation from Partially Observed sEMG

Wending Heng, Mingming Zhang, Glen Cooper, Zhenhong Li

University of Manchester · 2Southern University of Science and Technology

Surface Electromyography Physics-Informed ML Musculoskeletal Modeling

Open paper AskPDF arXiv

This paper investigates multi-degrees of freedom (DoF) joint kinematics estimation under partially observed surface electromyography (sEMG), where only a subset of task-relevant muscles can be measured due to anatomical inaccessibility or sensor constraints. A novel musculoskeletal neural network (MSK-NN) is proposed to estimate multi-DoF joint angles while simultaneously inferring activations for both measured and unmeasured muscles. MSK-NN consists of a CNN-based muscle activation estimator and an embedded MSK forward dynamics module, forming a fully differentiable architecture. Unlike existing hybrid neural frameworks that require additional biomechanical labels (e.g., muscle-tendon forces, joint torques), MSK-NN is trained without direct supervision of internal biomechanical variables. A composite physics-physiology loss is designed by incorporating a joint kinematics loss, a data-driven muscle synergy loss, and an anatomy-guided trend loss. The proposed method is evaluated on two-DoF wrist kinematics estimation across three rhythmic motions with unconstrained speed and amplitude, and one random motion. Compared with CNN, Bi-LSTM, CNN-LSTM, and PET baselines, MSK-NN achieves lower normalized root mean square error (NRMSE) and higher coefficient of determination (R2), especially for the random motion. More importantly, the optimized MSK parameters remain within physiological limits, and the estimated activation of an input-excluded muscle exhibits strong temporal agreement with its recorded sEMG envelope, demonstrating the capability of musculoskeletal (MSK)-NN to recover physiologically plausible activations.

Abstract

Spatio-Temporal Attention

Explore similar work

Physiologically Constrained Musculoskeletal Neural Network for Multi-DoF Joint Kinematics Estimation from Partially Observed sEMG

EFormer: Temporally Aligned Local Correction for Continuous sEMG-Based Hand Pose Tracking

BioHuman: Learning Biomechanical Human Representations from Video