cs.ROAug 17, 2026

Lambda-Hold Control: Human-Like Movement Emerges from a Minimal Task Reward in Predictive Musculoskeletal Simulation

Authors: Jun Hyuk Lee, Chihyeong Lee, Jooeun Ahn

Organizations: Department of Physical Education, Seoul National University, Seoul, Republic of Korea · Institute of Sport Science, Seoul National University, Seoul, Republic of Korea · SNU Robotics Institute, Seoul National University, Seoul, Republic of Korea

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The massive overactuation in the human musculoskeletal system makes it challenging to train musculoskeletal models to generate human-like motion via reinforcement learning, primarily because exploration in the resulting high-dimensional and redundant action space is extremely inefficient. To address this problem, we propose the λλ-hold controller, inspired by the equilibrium-point (EP) hypothesis, which has been widely supported by extensive evidence from human motor control studies. The policy's control variable is the per-muscle EP threshold length λλ, from which a stretch-reflex recruitment law computes the muscle excitations automatically. Holding each λλ over an interval of the gait phase also sharply reduces the frequency at which the policy must be queried. Consequently, the controller, to our knowledge for the first time, enables a muscle-actuated skeletal model to learn human-like sprinting using only a minimal reward within an hour of training. The efficient exploration through the proposed λλ-hold controller is not merely an engineering trick but an approach grounded in physiology, bringing together the EP hypothesis, intermittent control, and optimal feedback control. Beyond encapsulating human-like behavior in predictive simulation, this achievement contributes to developing a learnable model of the human motor controller.

Figures & tables

Figure 1: Overview of the λ\lambda -hold controller. The policy commands per-muscle threshold lengths λ\lambda only at intermittent decision points. Between them λ\lambda is held while a stretch-reflex recruitment law converts it, together with the evolving muscle length and velocity, into muscle excitation. Trained by reinforcement learning on a minimal reward of forward velocity alone, this controller learns human-like sprinting within about an hour of training.
Figure 2: Decision timing during a high-speed λ\lambda -hold sprint. A new λ\lambda is issued at each ground-reaction-force event, both foot strikes and toe-offs, with one additional decision point inserted between consecutive events. Filled curves are the per-foot vertical ground-reaction force (left foot up, right foot down). Vertical lines mark the λ\lambda resample points: a foot strike (rising contact edge, red), a toe-off (falling edge, orange), and the interposed sub-point (blue). The policy is re-queried only at these instants. Between them, the held λ\lambda and the stretch-reflex law generate the excitations.
Figure 3: Sprint learning curves for λ\lambda -hold and the four baselines, trained under an identical simulation-step budget. Each solid line is the mean over five random seeds and the shaded band shows ±\pm standard deviation. The return axis is logarithmic, so the plotted gap understates how far λ\lambda -hold outperforms the baselines. The deep, regular downward spikes in the λ\lambda -hold curve, visible on both axes, are the late-training oscillation analyzed in Appendix B . (a) Return against simulation steps, the axis on which all methods receive equal environment experience. (b) Return against decision steps, the number of policy queries.
Figure 4: How widely each controller explores joint-velocity space over training. (a) The entropy of the visited joint-velocity distribution, a measure of how varied the movements are, estimated with a Kozachenko–Leonenko kk -nearest-neighbor estimator at an equal sample size ( n=150n=150 ) across methods. Excitation-hold and Synergy have no estimate at the earliest points because they had not yet accumulated that sample. (b) The coverage of joint-velocity space, a measure of how much of the movement space is reached, taken as the cumulative number of distinct cells visited on a grid shared across methods, with 4 bins along each of the 15 joint-velocity dimensions.
Figure 5: Human-likeness of the emergent sprint, speed-matched to human treadmill running [ 45 ] . Simulated gait cycles are grouped by their mean COM speed, keeping those within ±0.35 \pm 0.35\, m/s of each reference speed ( 2.52.5 , 3.53.5 , and 4.5 4.5\, m/s). Both bands are mean ±\pm standard deviation. Each simulated band pools 3232 , 5454 , and 218218 gait cycles at 2.52.5 , 3.53.5 , and 4.5 4.5\, m/s, and each measured band pools 3131 , 3939 , and 3131 runners.
Figure 6: Model muscle activation against measured surface EMG. Model activations (blue) pool all 129129 gait cycles whose mean COM speed exceeds 4.3 4.3\, m/s, aligned at right initial contact. Measured EMG (gray) is the mean ±\pm standard deviation of 1919 subjects running at 5.0 5.0\, m/s [ 46 ] . Both are peak-normalized over the gait cycle.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

SettingSAC-basedDEP-RL
AlgorithmSACDEP + MPO
Actor / critic networkMLP, two hidden layers of 256 unitsMLP, two hidden layers of 1024 units
ExplorationgSDE, resampled every 4 decision stepsDEP (defaults, rescaled to 0.01 0.01\, s)
Learning rate3×10−43\times 10^{-4}actor 3.53×10−53.53\times 10^{-5} , critic 6.08×10−56.08\times 10^{-5} , dual 2.13×10−32.13\times 10^{-3}
Discount γ\gamma / target smoothing τ\tau0.990.99 / 0.0050.005γ1/2.5 ⁣≈ ⁣0.996\gamma^{1/2.5}\!\approx\!0.996 / —
Batch size / replay buffer256256 / 5×1055\times 10^{5}256256 / 1×1061\times 10^{6}
Appendix
Table 1: Training hyperparameters. The SAC-based column applies to the λ\lambda -hold controller and to the Plain SAC, Excitation-hold, and Synergy baselines. The DEP-RL column gives the DEP + MPO settings. DEP’s step-counted quantities are rescaled from the reference 0.025 0.025\, s step to the shared 0.01 0.01\, s step (Section 2.6 ).
Figure 7: The effect of reference-state initialization (RSI) and the maximum-shortening-velocity curriculum. Three conditions are compared: λ\lambda -hold alone (blue), λ\lambda -hold with RSI (green), and λ\lambda -hold with RSI and the shortening-velocity curriculum (purple). (a) Return against simulation steps and (b) against decision steps. All three use five seeds, with the solid line the mean and the shaded band showing ±\pm standard deviation.
Figure 8: Hip-flexor force and velocity of the right psoas major and iliacus at the top-speed λ\lambda -hold episode over the last few gait cycles. (a) Muscle activation, which saturates at 11 during the concentric swing drive. (b) Normalized velocity over the same window, reaching −2-2 to −3-3 (negative is shortening) just as activation saturates. (c) Normalized force, strongly modulated despite the near-constant activation. (d) Fully activated samples only (activation >0.9{>}0.9 ), plotted against shortening speed. Averaged over these instants, force is 0.950.95 (psoas major) and 0.940.94 (iliacus) near isometric but only 0.260.26 and 0.330.33 at the fastest shortening.
More in this topic

RL for Legged Locomotion

Reinforcement learning for legged locomotion trains walking robots to move across varied terrain, adapt to disturbances, and carry out tasks.

See all 158 papers

Explore similar work

CardsList

  1. Muscle Synergy Priors Enhance Biomechanical Fidelity in Predictive Musculoskeletal Locomotion Simulation

    Mar 11, 2026Ilseung Park, Eunsik Choi, Jangwhan Ahn +1Musculoskeletal ModelingRL Control

    Carnegie Mellon University · Department of Physical Education, Seoul National University, Seoul, Republic of Korea · Seoul National University +2

  2. Learning Hip Exoskeleton Control Policy via Predictive Neuromusculoskeletal Simulation

    Mar 4, 2026Ilseung Park, Changseob Song, Inseung KangHip Exoskeleton ControlRobotic Control

    Department of Mechanical Engineering, Carnegie Mellon University, Pittsburgh, PA, 15213 USA.

  3. Scaling Whole-Body Human Musculoskeletal Behavior Emulation for Specificity and Diversity

    Mar 31, 2026Yunyue Wei, Chenhui Zuo, Shanning Zhuang +3Whole-Body ControlHuman Motion Generation

    Tsinghua University.