Authors: Ilseung Park, Eunsik Choi, Jangwhan Ahn, Jooeun Ahn
Organizations: Carnegie Mellon University · Department of Physical Education, Seoul National University, Seoul, Republic of Korea · Seoul National University · UNC-Chapel Hill and NC State University · Lampe Joint Department of Biomedical Engineering, UNC-Chapel Hill and NC State University, Raleigh, NC 27695, USA
Human locomotion emerges from high-dimensional neuromuscular control, making predictive musculoskeletal simulation challenging. We present a physiology-informed reinforcement-learning framework that constrains control using muscle synergies. We extracted a low-dimensional synergy basis from inverse musculoskeletal analyses of a small set of overground walking trials and used it as the action space for a muscle-driven three-dimensional model trained across variable speeds, slopes and uneven terrain. The resulting controller generated stable gait from 0.7-1.8 m/s and on ± 6∘ grades and reproduced condition-dependent modulation of joint angles, joint moments and ground reaction forces. Compared with an unconstrained controller, synergy-constrained control reduced non-physiological knee kinematics and kept knee moment profiles within the experimental envelope. Across conditions, simulated vertical ground reaction forces correlated strongly with human measurements, and muscle-activation timing largely fell within inter-subject variability. These results show that embedding neurophysiological structure into reinforcement learning can improve biomechanical fidelity and generalization in predictive human locomotion simulation with limited experimental data.
Figures & tables
Figure 1: Comparison of reinforcement learning performance for independent and synergistic control strategies. Mean episode reward is shown as a function of total training timesteps (×10 7 ), averaged over five training runs with different random seeds. Solid lines indicate a 10-point moving average applied to the mean rewards, and shaded regions represent the standard deviation across runs. Both control strategies exhibit stable learning behavior throughout training. The independent controller (orange) shows an earlier rise in reward and converges to a higher final performance level than the synergistic controller (blue).
Figure 2: Similarity between predicted muscle activations and experimental EMG envelopes across eight lower-limb muscles. Top box plots show Pearson correlation coefficients between predicted muscle activations and experimental EMG envelopes during level treadmill walking at 1.2 m/s for soleus (SOL), gastrocnemius medialis (GAS), tibialis anterior (TA), semitendinosus (ST), biceps femoris (BF), rectus femoris (RF), vastus medialis (VM), and vastus lateralis (VL). Gray box plots represent the cross-human experimental benchmark derived from 22 participants of the open-source dataset, and the surrounding gray shaded regions indicate the corresponding envelope of cross-human variability. Orange and blue box plots correspond to independent and synergistic control conditions, respectively. Round dots denote individual correlation coefficients (n = 231 for cross-human pairwise comparisons; n = 22 per controller for correlations between simulated and experimental activations). Bottom line plots show representative activation profiles over the normalized gait cycle, where gray curves indicate experimental EMG envelopes across participants, and colored curves represent predicted activations from the independent (orange) and synergistic (blue) controllers. Muscle activations were normalized to reference voluntary contraction (RVC): for each muscle, predicted activations were scaled to the maximum amplitude observed within each controller, whereas experimental EMG envelopes were normalized to the subject-specific maximum amplitude.
Figure 3: Comparison of experimental and simulated kinematics, kinetics, and ground reaction forces across walking speeds. Sagittal-plane joint angles and joint moments are shown for the hip, knee, and ankle over the gait cycle, alongside anterior–posterior (AP) and vertical GRFs. Joint moments and GRFs were normalized by body mass. Experimental measurements are shown in grayscale, independent control in orange, and synergistic control in blue. For each condition, lighter colors indicate slower walking speeds (0.7 m/s) and darker colors indicate faster walking speeds (1.8 m/s) in increments of 0.1 m/s. Arrows highlight gait phases exhibiting clear speed-dependent magnitude modulation. Red arrows denote trends observed in the experimental data and similarly captured by the synergistic controller, whereas green arrows indicate regions where the independent controller deviates from the experimentally observed pattern.
Figure 4: Comparison of experimental and simulated kinematics, kinetics, and ground reaction forces across terrain slope conditions. Sagittal-plane joint angles and joint moments are shown for the hip, knee, and ankle over the gait cycle, alongside AP and vertical GRFs. Joint moments and GRFs were normalized by body mass. Experimental measurements are shown in grayscale, independent control in orange, and synergistic control in blue. For each condition, darker colors represent downhill walking (−5°), intermediate colors represent level walking (0°), and lighter colors represent uphill walking (+5°).
Figure 5: Phase-dependent root-mean-square error (RMSE) ratios of simulated kinematics, kinetics, and GRFs relative to experimental data. Heatmaps summarize the averaged RMSE ratios across gait phases (loading response, mid-stance, pre-swing, and swing) for independent control (orange) and synergistic control (blue) under both speed (left) and slope (right) conditions. RMSE values for joint angles, joint moments, and GRFs were normalized by the corresponding cross-human RMSE; ratios close to 1.0 indicate the errors comparable to cross-human variability.
Muscle-driven locomotion provides a physically grounded approach to generating realistic human movement. However, achieving both physiological plausibility and adaptability to changes in musculoskeletal capacity and external disturbances remains a fundamental challenge. To address this limitation, we propose a Reflex-Informed Neuromuscular Reinforcement Learning framework for muscle-driven locomotion. Within this framework, a fixed phase-dependent reflex controller serves as the underlying neuromuscular control mechanism, while the reinforcement learning policy produces four biomechanically meaningful residual parameters to modulate key reflex gains and thresholds associated with hip swing, knee support, and ankle propulsion according to the current state. Experimental results demonstrate that the proposed framework generates physiologically plausible locomotion with improved kinematic accuracy and dynamic consistency, as well as better bilateral symmetry and stride-to-stride consistency under nominal walking conditions. The learned policy remains robust under muscle weakness and external perturbations without retraining.
Data-driven joint-moment predictors offer a scalable alternative to laboratory-based inverse-dynamics pipelines for biomechanics estimation and exoskeleton control. Meanwhile, physics-based reinforcement learning (RL) enables simulation-trained controllers to learn dynamics-aware assistance strategies without extensive human experimentation. However, quantitative verification of simulation-trained exoskeleton torque predictors, and their impact on human joint power injection, remains limited. This paper presents (1) an RL framework to learn exoskeleton assistance policies that reduce biological joint moments, and (2) a validation pipeline that verifies the trained control networks using an open-source gait dataset through inference and comparison with biological joint moments. Simulation-trained multilayer perceptron (MLP) controllers are developed for level-ground and ramp walking, mapping short-horizon histories of bilateral hip and knee kinematics to normalized assistance torques. Results show that predicted assistance preserves task-intensity trends across speeds and inclines. Agreement is particularly strong at the hip, with cross-correlation coefficients reaching 0.94 at 1.8 m/s and 0.98 during 5° decline walking, demonstrating near-matched temporal structure. Discrepancies increase at higher speeds and steeper inclines, especially at the knee, and are more pronounced in joint power comparisons. Delay tuning biases assistance toward greater positive power injection; modest timing shifts increase positive power and improve agreement in specific gait intervals. Together, these results establish a quantitative validation framework for simulation-trained exoskeleton controllers, demonstrate strong sim-to-data consistency at the torque level, and highlight both the promise and the remaining challenges for sim-to-real transfer.
Zihang You, Xianlian Zhou
Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USA
Developing exoskeleton controllers that generalize across diverse locomotor conditions typically requires extensive motion-capture data and biomechanical labeling, limiting scalability beyond instrumented laboratory settings. Here, we present a physics-based neuromusculoskeletal learning framework that trains a hip-exoskeleton control policy entirely in simulation, without motion-capture demonstrations, and deploys it on hardware via policy distillation. A reinforcement learning teacher policy is trained using a muscle-synergy action prior over a wide range of walking speeds and slopes through a two-stage curriculum, enabling direct comparison between assisted and no-exoskeleton conditions. In simulation, exoskeleton assistance reduces mean muscle activation by up to 3.4% and mean positive joint power by up to 7.0% on level ground and ramp ascent, with benefits increasing systematically with walking speed. On hardware, the assistance profiles learned in simulation are preserved across matched speed-slope conditions (r: 0.82, RMSE: 0.03 Nm/kg), providing quantitative evidence of sim-to-real transfer without additional hardware tuning. These results demonstrate that physics-based neuromusculoskeletal simulation can serve as a practical and scalable foundation for exoskeleton controller development, substantially reducing experimental burden during the design phase.
Ilseung Park, Changseob Song, Inseung Kang
Department of Mechanical Engineering, Carnegie Mellon University, Pittsburgh, PA, 15213 USA.