Multi-Agent Reinforcement Learning of Co-Adaptive Hip Exoskeleton Control (MARL-Exo) for Metabolic Energy Reduction

Yifei Yuan; Jakob Wolf; Ghaith Androwis; Xianlian Zhou

cs.ROOct 27, 2026

Multi-Agent Reinforcement Learning of Co-Adaptive Hip Exoskeleton Control (MARL-Exo) for Metabolic Energy Reduction

Authors: Yifei Yuan, Jakob Wolf, Ghaith Androwis, Xianlian Zhou

Organizations: Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USA · Kessler Foundation, West Orange, NJ 07052, USA

Abstract

Objective: This study aims to determine whether a hip-exoskeleton policy trained in simulation, by co-adapting with a musculoskeletal human model under a metabolic-energy reward, reduces the metabolic cost of walking in new users without per-user tuning, and how mechanical-power shaping affects the learned assistance. Methods: We developed MARL-Exo, a multi-agent reinforcement learning framework in which a full-body human model (304 muscle actuators; randomized anatomy, gait, and speed) and a bilateral hip-exoskeleton policy are trained concurrently in two stages. The exoskeleton observes only delayed hip kinematics and its past commands; its reward includes muscle metabolic power from a physiologically based energetics model, alone (Metabolic) or with mechanical-power shaping (Metabolic + Power). The Metabolic policy was deployed and tested without retraining in eight healthy adults walking at 1.25 m/s, using indirect calorimetry. Results: In simulation, the Metabolic policy reduced modeled muscle metabolic power by 13.8% relative to no-device walking, versus 8.8% for Metabolic + Power. On hardware, MARL-Exo reduced net metabolic rate by 18.3 $\pm$ 6.6% relative to Passive in all eight participants. Relative to No-Exo, the mean reduction was 7.4 $\pm$ 10.0%; six participants more than offset the 13.1% device-wearing penalty, walking up to 24.2% below their No-Exo cost. Conclusion: A metabolically informed policy co-adapted in simulation transferred to users without per-user tuning and consistently reduced the metabolic cost of walking with the device, while explicit power shaping reduced the simulated benefit. Significance: Unlike per-user human-in-the-loop optimization, this approach yields a single population-level controller and allows assistance objectives to be screened in simulation before human testing.

Figures & tables

Fig. 1: Overview of MARL-Exo. (a) Training in simulation: a human policy $\pi_{H}$ and an exoskeleton policy $\pi_{E}$ act on a shared physics simulation; human actions pass through stable proportional-derivative (SPD) control and an auxiliary muscle network $\Phi^{\psi}$ , and the exoskeleton reward is computed by a muscle metabolic energy model. Stage 1 trains $\pi_{H}$ alone, and Stage 2 adds the exoskeleton and co-adapts both policies. Multi-agent PPO updates both policies from a shared rollout batch $\mathcal{D}$ , while $\Phi^{\psi}$ is trained by supervised regression to reproduce the desired joint torques $\tau_{\mathrm{des}}$ through the muscle moment-arm mapping $J^{\top}A$ (line 22 of Algorithm 1). (b) The trained $\pi_{E}$ is deployed on a Raspberry Pi 5 without subject-specific tuning and validated in eight adults.

	SMAT	MARL-Exo
Human model	26 muscles (13 per side), imitation reward	50-DOF body, 304 Hill-type muscle actuators, gait reward
Human action	Muscle excitations	Residual joint targets on a reference trajectory $\to$ PD torque $\to$ learned muscle network
Randomization	None	Anatomy, strength, gait, speed in observation
Curriculum	Four stages	Two stages
Critic	Shared by both actors	One per agent
Exo reward	Power sign and smoothness	Smoothness, magnitude, hip-muscle metabolic power, and weak human-reward coupling (deployed policy)

TABLE I: Main differences between MARL-Exo (this work) and SMAT [ 18 , 19 ] .

Policy optimization (PPO, per agent)
Parameter	Value
Learning rate	$1{\times}10^{-4}$
PPO clip range $\epsilon$	0.2
Value-function clip	100
Value loss coefficient	1.0
Discount $\gamma$	0.99

TABLE II: PPO training hyperparameters. The same configuration is shared by the human policy

\pi_{H}

and the exoskeleton policy

\pi_{E}

; the auxiliary muscle network is trained by supervised regression.

Fig. 2: Experimental protocol. Each participant completed standing rest, no-exoskeleton walking (No-Exo), and passive walking (Passive), then the two active controllers (MARL-Exo and SMAT) in randomized order, followed by the same reference conditions in reverse order. Each walking trial lasted approximately 6 min on a treadmill at 1.25 m/s, with metabolic rate measured by indirect calorimetry throughout.

Fig. 3: Hip joint angles, muscle-contributed joint moment (mass-normalized), assistance torque, and assistance power over the gait cycle for the simulated conditions (PowerExo: Metabolic + Power; EnergyExo: Metabolic) and an independent biological reference (Experiment, teal dashed, mean

\pm

1 SD) [ 31 ] . Muscle-contributed moments are not identical to inverse-dynamics net moments.

Muscle group	NoExo	PowerExo	EnergyExo	NoExo	PowerExo $-$ NoExo	EnergyExo $-$ NoExo
	W	W	W	W/kg	$\Delta$ (%)	$\Delta$ (%)
Hip Extension	15.47	12.50	9.99	0.212	$-$ 2.97 ( $-$ 19.2%)	$-$ 5.48 ( $-$ 35.4%)
Hip Flexion	14.84	12.97	11.27	0.204	$-$ 1.87 ( $-$ 12.6%)	$-$ 3.57 ( $-$ 24.0%)
Knee Flexion	56.07	52.39	53.64	0.769	$-$ 3.68 ( $-$ 6.6%)	$-$ 2.43 ( $-$ 4.3%)
Knee Extension	20.32	14.94	12.97	0.279	$-$ 5.38 ( $-$ 26.5%)	$-$ 7.35 ( $-$ 36.2%)
Ankle PlantarFlex	69.60	70.02	68.40	0.955	$+$ 0.42 ( $+$ 0.6%)	$-$ 1.20 ( $-$ 1.7%)

TABLE III: Cycle-averaged metabolic power by functional group (sum across member muscles) and total across all involved muscle actuators, all values per leg (left–right averaged; a partial-muscle sum, not whole-body metabolism). PowerExo: Metabolic + Power reward; EnergyExo: Metabolic reward (deployed).

\Delta

in Watts; percentages signed vs. NoExo. Groups overlap through biarticular muscles, so the six group totals exceed the total; the TOTAL row is the true non-overlapping sum. The Hip, Knee, and Ankle rows sum both directions at each joint (no muscle belongs to both directions of the same joint); biarticular muscles still appear in more than one joint row. Group membership is given in Supplementary Table S2. W/kg is the per-leg power divided by body mass (72.9 kg).

Fig. 4: Muscle activation (top row) and metabolic energy rate (bottom row) over the gait cycle for the three functional groups where the controllers produce their largest reductions: Hip Extension, Hip Flexion, and Knee Extension (group mean activation; group summed energy; mean

\pm

1 SD). Hip-flexor activation rises during stance phase but falls during swing phase while hip-flexor energy mostly falls under both controllers.

Fig. 5: Net metabolic rate (W kg -1 ) across conditions (

n=8

). Half-violin distributions, box plots, and individual subjects (connected lines) for No-Exo, Passive, and MARL-Exo. The No-Exo and Passive data are shared with [ 19 ] . Brackets show the group-mean percentage change between conditions (

+13.1\,\%

-18.3\,\%

-7.4\,\%

). Paired

t

-test with Bonferroni correction:

{}^{*}p<0.05

{}^{**}p<0.01

{}^{***}p<0.001

; n.s.

p\geq 0.05

. All eight subjects reduced net cost under assistance relative to the passive condition.

	Net metabolic rate (W/kg)				MARL-Exo reduction (%)		SMAT reduction (%)
Subject	No-Exo	Passive	MARL-Exo	SMAT	vs Passive	vs No-Exo	vs Passive	vs No-Exo
S1	3.008	3.326	2.405	2.738	27.7	20.1	17.7	9.0
S2	3.575	4.081	3.376	3.221	17.3	5.6	21.1	9.9
S3	2.941	3.051	2.228	2.368	27.0	24.2	22.4	19.5
S4	2.794	3.243	2.747	2.711	15.3	1.7	16.4	3.0
S5	3.734	4.222	3.916	3.571	7.2	$-$ 4.9	15.4	4.4

TABLE IV: Per-subject net metabolic rate and reductions under MARL-Exo and SMAT (

n=8

Fig. 6: Treadmill walking at 1.25 m/s under No-Exo (black), Passive (blue), and MARL-Exo (green) (

n=8

; per-subject means of both hips). Left: gait-cycle hip angle (flexion positive) with peak flexion and extension; labels give the change in ROM. Middle: cadence and stride length. Right: hip angular velocity together with the exoskeleton torque and mechanical power delivered by MARL-Exo. Bands and error bars show mean

\pm

1 SD across subjects. The No-Exo and Passive data are shared with [ 19 ] .

Subject	$\tau_{\text{RMS}}$ (N $\cdot$ m)	$\tau_{\text{MAX}}^{\text{flex}}$ (N $\cdot$ m)	$\tau_{\text{MAX}}^{\text{ext}}$ (N $\cdot$ m)	MPP (W)	MNP (W)	$R_{\text{pos}}$
S1	9.62	11.94	$-$ 11.97	24.04	$-$ 0.54	0.978
S2	9.76	11.97	$-$ 11.98	21.02	$-$ 0.24	0.989
S3	9.36	11.92	$-$ 11.94	16.79	$-$ 0.20	0.988
S4	10.10	11.99	$-$ 11.99	21.53	$-$ 0.33	0.985
S5	9.42	11.97	$-$ 11.97	18.40	$-$ 0.39	0.979
S6	9.68	11.80	$-$ 11.96	18.26	$-$ 0.05	0.997

TABLE V: Per-subject exoskeleton mechanical output, MARL-Exo condition (

n=8

, both hips averaged).

Term	Symbol	Value	Description
$r_{\text{lim}}$	$k_{\text{lim}}$	100	penalty gain
	$\theta^{\min}$	$-0.1$ rad	knee lower (extension) limit
	$\delta$	0.1 rad	soft-limit margin
$r_{\tau}$	$c$	$10^{-3}$ (N $\cdot$ m) -2	exponential scale
$r_{\text{clr}}$	$k_{\text{clr}}$	100 s m -3	penalty gain
	$h_{c}$	0.03 m	clearance height

TABLE S1: Constants of the auxiliary human reward terms.

		Hip		Knee		Ankle
Anatomical muscle	Actuators	HE	HF	KF	KE	PF	DF
Biceps femoris, long head	1	$\bullet$		$\bullet$
Biceps femoris, short head	2			$\bullet$
Gastrocnemius, lateral head	1			$\bullet$		$\bullet$
Gastrocnemius, medial head	1			$\bullet$		$\bullet$
Gluteus maximus	5	$\bullet$

TABLE S2: Membership of the analyzed muscle actuators (per leg) in the functional and joint groups.

\bullet

marks membership. The bottom rows give the number of actuators in each group.

Hip Exoskeleton Control

See all 12 papers

Staged Multi-Agent Training for Human-Exoskeleton Co-Adaptation: Simulation-Based Learning and Real-World Metabolic and Biomechanical Validation
Aug 1, 2026Yifei Yuan, Jakob Wolf, Ghaith Androwis +1Robotic Control Hip Exoskeleton Control
Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USA · Kessler Foundation, West Orange, NJ 07052, USA
Learning Hip Exoskeleton Control Policy via Predictive Neuromusculoskeletal Simulation
Mar 4, 2026Ilseung Park, Changseob Song, Inseung KangHip Exoskeleton Control Robotic Control
Department of Mechanical Engineering, Carnegie Mellon University, Pittsburgh, PA, 15213 USA.
Learning a Speed-adaptive Hip Exoskeleton Control Policy Via Sim-to-real Reinforcement Learning
Sep 23, 2026Bin Li, Zhimin Hou, Jiacheng Hou +4Hip Exoskeleton Control Assistive Robotics
Division of Industrial Data Science (DIDS), School of Data Science, Lingnan University, Hong Kong · Department of Mechanical and Energy Engineering, Southern University of Science and Technology, Shenzhen, 518055, China

Aug 1, 2026J/K move · Enter open · S save

Staged Multi-Agent Training for Human-Exoskeleton Co-Adaptation: Simulation-Based Learning and Real-World Metabolic and Biomechanical Validation

Yifei Yuan, Jakob Wolf, Ghaith Androwis, Xianlian Zhou

Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USA · Kessler Foundation, West Orange, NJ 07052, USA

Robotic Control Hip Exoskeleton Control Assistive Robotics Curriculum Learning Cooperative MARL

Open paper AskPDF arXiv

Learning-based controllers can enable exoskeleton assistance to be trained entirely in physics-based simulation, but achieving effective assistance on real users remains challenging because the human and device continuously adapt to one another. We investigate a Staged Multi-Agent Training (SMAT) approach that jointly trains a musculoskeletal human actor and a bilateral hip-exoskeleton actor (control policy) to learn coordinated assistance strategies while accounting for human-device co-adaptation. The resulting policy was transferred to a physical hip exoskeleton and evaluated in eight healthy adults across users and locomotion conditions without subject-specific retraining. Indirect calorimetry was used to compare no-exoskeleton, passive, and active conditions. Active assistance reduced net metabolic rate by 19.7% relative to the passive device (p < 0.001). The policy produced predominantly positive hip mechanical power across subjects, with a positive-power ratio of 0.984, and maintained effective assistance across walking speeds and terrains beyond the training conditions. These results demonstrate that simulation-based multi-agent co-adaptation can produce a single transferable exoskeleton policy that elicits meaningful physiological benefits in real users without subject-specific retraining, providing a framework for reducing the experimental burden of human-in-the-loop controller development.

Abstract

Hip Exoskeleton Control

Explore similar work

Staged Multi-Agent Training for Human-Exoskeleton Co-Adaptation: Simulation-Based Learning and Real-World Metabolic and Biomechanical Validation

Learning Hip Exoskeleton Control Policy via Predictive Neuromusculoskeletal Simulation

Learning a Speed-adaptive Hip Exoskeleton Control Policy Via Sim-to-real Reinforcement Learning