Lambda-Hold Control: Human-Like Movement Emerges from a Minimal Task Reward in Predictive Musculoskeletal Simulation
Authors: Jun Hyuk Lee, Chihyeong Lee, Jooeun Ahn
Organizations: Department of Physical Education, Seoul National University, Seoul, Republic of Korea · Institute of Sport Science, Seoul National University, Seoul, Republic of Korea · SNU Robotics Institute, Seoul National University, Seoul, Republic of Korea
The massive overactuation in the human musculoskeletal system makes it challenging to train musculoskeletal models to generate human-like motion via reinforcement learning, primarily because exploration in the resulting high-dimensional and redundant action space is extremely inefficient. To address this problem, we propose the λ-hold controller, inspired by the equilibrium-point (EP) hypothesis, which has been widely supported by extensive evidence from human motor control studies. The policy's control variable is the per-muscle EP threshold length λ, from which a stretch-reflex recruitment law computes the muscle excitations automatically. Holding each λ over an interval of the gait phase also sharply reduces the frequency at which the policy must be queried. Consequently, the controller, to our knowledge for the first time, enables a muscle-actuated skeletal model to learn human-like sprinting using only a minimal reward within an hour of training. The efficient exploration through the proposed λ-hold controller is not merely an engineering trick but an approach grounded in physiology, bringing together the EP hypothesis, intermittent control, and optimal feedback control. Beyond encapsulating human-like behavior in predictive simulation, this achievement contributes to developing a learnable model of the human motor controller.
Figures & tables
Figure 1: Overview of the λ -hold controller. The policy commands per-muscle threshold lengths λ only at intermittent decision points. Between them λ is held while a stretch-reflex recruitment law converts it, together with the evolving muscle length and velocity, into muscle excitation. Trained by reinforcement learning on a minimal reward of forward velocity alone, this controller learns human-like sprinting within about an hour of training.
Figure 2: Decision timing during a high-speed λ -hold sprint. A new λ is issued at each ground-reaction-force event, both foot strikes and toe-offs, with one additional decision point inserted between consecutive events. Filled curves are the per-foot vertical ground-reaction force (left foot up, right foot down). Vertical lines mark the λ resample points: a foot strike (rising contact edge, red), a toe-off (falling edge, orange), and the interposed sub-point (blue). The policy is re-queried only at these instants. Between them, the held λ and the stretch-reflex law generate the excitations.
Figure 3: Sprint learning curves for λ -hold and the four baselines, trained under an identical simulation-step budget. Each solid line is the mean over five random seeds and the shaded band shows ± standard deviation. The return axis is logarithmic, so the plotted gap understates how far λ -hold outperforms the baselines. The deep, regular downward spikes in the λ -hold curve, visible on both axes, are the late-training oscillation analyzed in Appendix B . (a) Return against simulation steps, the axis on which all methods receive equal environment experience. (b) Return against decision steps, the number of policy queries.
Figure 4: How widely each controller explores joint-velocity space over training. (a) The entropy of the visited joint-velocity distribution, a measure of how varied the movements are, estimated with a Kozachenko–Leonenko k -nearest-neighbor estimator at an equal sample size ( n=150 ) across methods. Excitation-hold and Synergy have no estimate at the earliest points because they had not yet accumulated that sample. (b) The coverage of joint-velocity space, a measure of how much of the movement space is reached, taken as the cumulative number of distinct cells visited on a grid shared across methods, with 4 bins along each of the 15 joint-velocity dimensions.
Figure 5: Human-likeness of the emergent sprint, speed-matched to human treadmill running [ 45 ] . Simulated gait cycles are grouped by their mean COM speed, keeping those within ±0.35 m/s of each reference speed ( 2.5 , 3.5 , and 4.5 m/s). Both bands are mean ± standard deviation. Each simulated band pools 32 , 54 , and 218 gait cycles at 2.5 , 3.5 , and 4.5 m/s, and each measured band pools 31 , 39 , and 31 runners.
Figure 6: Model muscle activation against measured surface EMG. Model activations (blue) pool all 129 gait cycles whose mean COM speed exceeds 4.3 m/s, aligned at right initial contact. Measured EMG (gray) is the mean ± standard deviation of 19 subjects running at 5.0 m/s [ 46 ] . Both are peak-normalized over the gait cycle.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
SAC-based
DEP-RL
Algorithm
SAC
DEP + MPO
Actor / critic network
MLP, two hidden layers of 256 units
MLP, two hidden layers of 1024 units
Exploration
gSDE, resampled every 4 decision steps
DEP (defaults, rescaled to 0.01 s)
Learning rate
3×10−4
actor 3.53×10−5 , critic 6.08×10−5 , dual 2.13×10−3
Discount γ / target smoothing τ
0.99 / 0.005
γ1/2.5≈0.996 / —
Batch size / replay buffer
256 / 5×105
256 / 1×106
Appendix
Table 1: Training hyperparameters. The SAC-based column applies to the λ -hold controller and to the Plain SAC, Excitation-hold, and Synergy baselines. The DEP-RL column gives the DEP + MPO settings. DEP’s step-counted quantities are rescaled from the reference 0.025 s step to the shared 0.01 s step (Section 2.6 ).
Figure 7: The effect of reference-state initialization (RSI) and the maximum-shortening-velocity curriculum. Three conditions are compared: λ -hold alone (blue), λ -hold with RSI (green), and λ -hold with RSI and the shortening-velocity curriculum (purple). (a) Return against simulation steps and (b) against decision steps. All three use five seeds, with the solid line the mean and the shaded band showing ± standard deviation.
Figure 8: Hip-flexor force and velocity of the right psoas major and iliacus at the top-speed λ -hold episode over the last few gait cycles. (a) Muscle activation, which saturates at 1 during the concentric swing drive. (b) Normalized velocity over the same window, reaching −2 to −3 (negative is shortening) just as activation saturates. (c) Normalized force, strongly modulated despite the near-constant activation. (d) Fully activated samples only (activation >0.9 ), plotted against shortening speed. Averaged over these instants, force is 0.95 (psoas major) and 0.94 (iliacus) near isometric but only 0.26 and 0.33 at the fastest shortening.
Human locomotion emerges from high-dimensional neuromuscular control, making predictive musculoskeletal simulation challenging. We present a physiology-informed reinforcement-learning framework that constrains control using muscle synergies. We extracted a low-dimensional synergy basis from inverse musculoskeletal analyses of a small set of overground walking trials and used it as the action space for a muscle-driven three-dimensional model trained across variable speeds, slopes and uneven terrain. The resulting controller generated stable gait from 0.7-1.8 m/s and on ± 6∘ grades and reproduced condition-dependent modulation of joint angles, joint moments and ground reaction forces. Compared with an unconstrained controller, synergy-constrained control reduced non-physiological knee kinematics and kept knee moment profiles within the experimental envelope. Across conditions, simulated vertical ground reaction forces correlated strongly with human measurements, and muscle-activation timing largely fell within inter-subject variability. These results show that embedding neurophysiological structure into reinforcement learning can improve biomechanical fidelity and generalization in predictive human locomotion simulation with limited experimental data.
Ilseung Park, Eunsik Choi, Jangwhan Ahn +1
Carnegie Mellon University · Department of Physical Education, Seoul National University, Seoul, Republic of Korea · Seoul National University +2
Developing exoskeleton controllers that generalize across diverse locomotor conditions typically requires extensive motion-capture data and biomechanical labeling, limiting scalability beyond instrumented laboratory settings. Here, we present a physics-based neuromusculoskeletal learning framework that trains a hip-exoskeleton control policy entirely in simulation, without motion-capture demonstrations, and deploys it on hardware via policy distillation. A reinforcement learning teacher policy is trained using a muscle-synergy action prior over a wide range of walking speeds and slopes through a two-stage curriculum, enabling direct comparison between assisted and no-exoskeleton conditions. In simulation, exoskeleton assistance reduces mean muscle activation by up to 3.4% and mean positive joint power by up to 7.0% on level ground and ramp ascent, with benefits increasing systematically with walking speed. On hardware, the assistance profiles learned in simulation are preserved across matched speed-slope conditions (r: 0.82, RMSE: 0.03 Nm/kg), providing quantitative evidence of sim-to-real transfer without additional hardware tuning. These results demonstrate that physics-based neuromusculoskeletal simulation can serve as a practical and scalable foundation for exoskeleton controller development, substantially reducing experimental burden during the design phase.
Ilseung Park, Changseob Song, Inseung Kang
Department of Mechanical Engineering, Carnegie Mellon University, Pittsburgh, PA, 15213 USA.
The embodied learning of human motor control requires whole-body neuro-actuated musculoskeletal dynamics, while the internal muscle-driven processes underlying movement remain inaccessible to direct measurement. Computational modeling offers an alternative, but inverse dynamics methods struggled to resolve redundant control from observed kinematics in the high-dimensional, over-actuated system. Forward imitation approaches based on deep reinforcement learning exhibited inadequate tracking performance due to the curse of dimensionality in both control and reward design. Here we introduce a large-scale parallel musculoskeletal computation framework for biomechanically grounded whole-body motion reproduction. By integrating large-scale parallel GPU simulation with adversarial reward aggregation and value-guided flow exploration, the MS-Emulator framework overcomes key optimization bottlenecks in high-dimensional reinforcement learning for musculoskeletal control, which accurately reproduces a broad repertoire of motions in a whole-body human musculoskeletal system actuated by approximately 700 muscles. It achieved high joint angle accuracy and body position alignment for highly dynamic tasks such as dance, cartwheel, and backflip. The framework was also used to explore the musculoskeletal control solution space, identifying distinct musculoskeletal control policies that converge to nearly identical external kinematic and mechanical measurements. This work establishes a tractable computational route to analyzing the specificity and diversity underlying human embodied control of movement. Project page: https://lnsgroup.cc/research/MS-Emulator.