PROMO: Preference-conditioned Multi-Objective Reinforcement Learning for Quadrupedal Robots

Amr Mousa; Rifny Rachman; Neil Karavis; Michele Caprio; Richard Allmendinger

cs.ROOct 1, 2026

PROMO: Preference-conditioned Multi-Objective Reinforcement Learning for Quadrupedal Robots

Authors: Amr Mousa, Rifny Rachman, Neil Karavis, Michele Caprio, Richard Allmendinger

Organizations: The University of Manchester, United Kingdom · BAE Systems, United Kingdom · University of Warwick, United Kingdom

Abstract

Quadrupedal locomotion requires balancing conflicting objectives such as command tracking, stability, and energy efficiency, yet conventional reinforcement learning (RL) hardcodes these priorities into a fixed scalar reward at training time. We present PROMO (Preference-Conditioned Multi-Objective Reinforcement Learning), a semantic multi-objective approach that makes this trade-off an explicit runtime input to a single locomotion policy. PROMO conditions the policy on deployment facing preferences while keeping embodiment-specific locomotion priors fixed, thereby separating operator intent from reward shaping terms required for viable gait generation. Compared with fixed-objective controllers, multi-objective baselines, and independently trained specialists, PROMO achieves objective specialization and robustness from a single deployable policy. Across 100 sampled preferences in simulation, 67 behaviors are non-dominated under exact Pareto dominance, with a mean preference-objective correlation of 0.843, demonstrating broad Pareto coverage and predictable preference response. The same policy transfers zero-shot to a Unitree Go2, where preference changes alone reduce specific energy by up to 30.4%, position error by 38.7%, and peak body-attitude deviation by 59.0% relative to the balanced preference. These results establish preference-conditioned multi-objective RL as a practical runtime interface for adaptive legged locomotion, extending its role beyond offline Pareto-set construction. Open-source code and videos are available at https://amrmousa.com/promo/.

Figures & tables

Panel of Figure 1 — Fig. 3: Training optimization over three seeds (mean, $\pm 1$ s.d.). The top row reports mean reward and the bottom row reports mean episode length for locomotion baselines (a,d), morl baselines (b,e), and independently trained specialists (c,f). promo converges to a higher reward than the shared-scale locomotion and morl references while matching the episode-length convergence of the balanced specialist, learning the preference space at roughly the optimization cost of a single controller. Specialist rewards are shown for completeness but use differently scaled scalar objectives, so episode length is the more informative cross-specialist comparison.

Method	Tracking $\uparrow$	Stability $\uparrow$	Efficiency $\uparrow$	Fall $\downarrow$	cvar tilt $\downarrow$
tar	$14.921$	$\mathbf{-2.218}$	$-3.643$	$14.5\%$	$0.116$
rma	$13.742$	$-2.920$	$\mathbf{-3.423}$	$18.2\%$	$0.125$
promo - sorl	$16.225$	$-2.804$	$-4.157$	$11.0\%$	$0.109$
promo (ours)	$\mathbf{18.703}$	$-2.855$	$-5.366$	$\mathbf{7.4\%}$	$\mathbf{0.105}$

TABLE I: Common balanced locomotion evaluation. Higher is better for semantic rewards; lower is better for fall and cvar tilt. Fall values are percentages. Bold denotes the best mean in each column.

Method	Tracking $T\uparrow$	Stability $S\uparrow$	Efficiency $E\uparrow$	Fall $\downarrow$	Scalarized return $\uparrow$	4-anchor Ctrl. $\uparrow$
dpmorl	$14.430$	$-2.352$	$\mathbf{-2.133}$	$26.6\%$	$2.986$	$--$
moppo	$15.326$	$-2.305$	$-2.794$	$19.4\%$	$2.212$	$0.492$
Specialists	$17.833$	$\mathbf{-1.916}$	$-2.942$	$12.6\%$	$3.255$	$--$
promo	$\mathbf{20.099}$	$-2.245$	$-2.189$	$\mathbf{8.4\%}$	$\mathbf{3.964}$	$\mathbf{0.662}$

TABLE II: Four-anchor semantic evaluation against morl and specialist baselines. Bold denotes the best mean in each column.

Panel of Figure 4 — Fig. 4: Standalone simulation results for promo policy evaluated at 100 sampled preference vectors. Each sampled preference produces one policy evaluation; panels (a)–(c) show different 2-D projections of these same 100 evaluations, with Pareto-nondominated and dominated samples distinguished visually. Tracking and efficiency form the clearest conflict, while stability and efficiency are largely complementary. Panels (d)–(f) show marginal per-objective preference-to-behavior mappings: each realized objective return rises near-monotonically with its corresponding preference weight, while the spread reflects variation in the other two simplex weights.

Panel of Figure 5 — Fig. 5: Preference-induced gait adaptation on hardware. Chronophotographic overlays with 2-s tracked foot trajectories for tracking-, stability-, and efficiency-heavy preferences (left to right) using the same deployed promo policy. Tracking produces larger foot excursions, efficiency produces compact low-clearance trajectories, and stability exhibits an intermediate gait with increased clearance in this sequence.

Preference	Vel. RMSE [m/s]	Pos. MAE [m]	Energy [J/m]	cot [-]	Peak att. [rad]	Act. rate [rad/s]	Trq. rate [N m/s]
Replay-round
Balanced	$0.444$	$0.534$	$124.6$	$1.058$	$0.196$	$8.45$	$240.7$
Efficiency	$0.444$	$0.624$	$\mathbf{102.3}$	$\mathbf{0.869}$	$0.187$	$\mathbf{5.42}$	$\mathbf{153.4}$
Stability	$\mathbf{0.428}$	$0.538$	$132.4$	$1.125$	$\mathbf{0.158}$	$9.26$	$263.0$
Tracking	$0.493$	$\mathbf{0.405}$	$169.6$	$1.441$	$0.281$	$11.98$	$342.9$
Move-fast

TABLE III: Real-robot evaluation of a single promo policy under four semantic preferences and three command regimes on the Unitree Go2. Values are means over

n=5

trials per condition; bold denotes the preferred value within each regime.

Fig. 6: Preference-conditioned hardware behavior. Tracking mae , peak attitude deviation, and specific energy for the same deployed promo policy under four semantic preferences across replay, fast, and slow regimes. Bars show means over five hardware trials and error bars denote one standard deviation. Lower is better for all metrics. The corresponding preference generally shifts behavior toward its intended semantic objective, while cross-objective differences expose the resulting physical trade-offs.

Fig. 7: Disturbance recovery on a low-traction surface. Representative sequences using the balanced preference on a low-traction surface. Top is whole-body recovery following a disturbance at the head. Middle is recovery from a rear-left hip disturbance inducing approximately $230^{\circ}$ of yaw rotation. Bottom is recovery from direct disturbance of the front-right leg, requiring adjustment of the support configuration.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Term	Group	Coefficient	Physical interpretation
track_lin_vel_xy_exp	Tracking	$1.5$	Rewards planar velocity tracking with kernel standard deviation $0.5$
track_ang_vel_z_exp	Tracking	$0.75$	Rewards yaw-rate tracking with kernel standard deviation $0.5$
lin_vel_z_l2	Stability	$-2.0$	Penalizes vertical base motion
ang_vel_xy_l2	Stability	$-0.05$	Penalizes roll and pitch angular velocity
flat_orientation_l2	Stability	$-2.5$	Penalizes body tilt away from upright posture
joint_torques_l2	Efficiency	$-2.0\times 10^{-4}$	Penalizes squared applied torque

Appendix

TABLE A.1: Semantic objectives and fixed locomotion priors used by promo .

Protocol	Preference specification	Purpose
Training distribution	$w_{i}=0.1+0.7x_{i}$ , $x\sim\mathrm{Dirichlet}(1,1,1)$	Uniform truncated-simplex coverage without zeroed objectives
Balanced	$(1/3,1/3,1/3)$	Nominal semantic trade-off
Tracking-heavy	$(0.8,0.1,0.1)$	Tracking-focused anchor
Stability-heavy	$(0.1,0.8,0.1)$	Stability-focused anchor
Efficiency-heavy	$(0.1,0.1,0.8)$	Efficiency-focused anchor
Two-objective mixtures	$(0.45,0.45,0.1)$ , $(0.45,0.1,0.45)$ , $(0.1,0.45,0.45)$	Representative simplex edges

Appendix

TABLE A.2: Preference configurations used for training and evaluation.

Item	Value
Robot	Unitree Go2, $12$ - dof quadruped
Simulator	Isaac Sim / IsaacLab
Actor hidden sizes	$[512,256,128]$
Critic shared trunk	$[512,256,128]$
Semantic critic heads	$K=3$ scalar heads
Privileged encoder	MLP over privileged state and preference

Appendix

TABLE B.1: promo network, simulation, and control configuration.

Parameter	Range / setting
Static and dynamic friction	$[0.6,\,1.2]$
Restitution coefficient	$[0,\,1]$
Base mass offset	$[0,\,2]$ kg
Force perturbation	$\pm 10$ N
Torque perturbation	$\pm 5$ N m
Velocity perturbation	$\pm 2$ m/s every $2$ – $4$ s

Appendix

TABLE B.2: Domain randomization and perturbation ranges used during training.

Parameter	Value
Parallel environments	$4096$
Rollout horizon	$50$ policy steps per env
Discount factor $\gamma$	$0.99$
gae parameter $\lambda_{\mathrm{GAE}}$	$0.95$
ppo clipping $\epsilon$	$0.12$
Entropy coefficient	$0.005$

Appendix

TABLE B.3: Training and optimization configuration.

Metric	Value
Mean utility per step	$0.0149$
cvar 10 utility	$0.0120$
Fall rate	$7.50\%$
Preference–objective correlation	$0.8434$
tracking $\rho_{T}$	$0.7748$
stability $\rho_{S}$	$0.8289$

Appendix

TABLE C.1: Key interface metrics for promo over

100

preferences.

$\varepsilon$ threshold	$0$	$0.005$	$0.01$	$0.02$	$0.03$
Non-dominated solutions	$67$	$50$	$37$	$25$	$14$
Non-dominated set retained (%)	$100$	$74.6$	$55.2$	$37.3$	$20.9$

Appendix

TABLE C.2:

\varepsilon

-dominance pruning of the 100-preference evaluation.

Fig. C.2: Semantic preference simplex for promo . The simplex defines the deployment-facing trade-off among tracking, stability, and efficiency; the marked anchors are used for representative simulation and hardware evaluations, while the dense evaluation samples the interior.

Regime	Command protocol	Evaluation focus
Replay	Recorded, identical	Command-matched comparison
Fast	Manual joystick	Dynamic locomotion
Slow	Manual joystick	Near-standstill locomotion

Appendix

TABLE C.3: Real-robot evaluation regimes.

Variant	Reward $\uparrow$	Ep. len. $\uparrow$	Ctrl. $\uparrow$	Fall $\downarrow$	hv $\uparrow$
Fixed priors (ours)	$\mathbf{27.06}$	$\mathbf{997.56}$	$\mathbf{0.741}$	$\mathbf{0.060}$	$0.693$
Prior removed	$25.63$	$956.54$	$0.638$	$0.131$	$\mathbf{0.906}$
Priors inside objectives	$26.72$	$996.60$	$0.699$	$0.165$	$0.626$
Prior as 4th objective	$26.86$	$996.72$	$0.447$	$0.218$	$0.512$

Appendix

TABLE D.1: Effect of objective/prior factorization on comparable training and interface metrics.

Panel of Figure 19 — Fig. D.1: Training behavior under alternative objective/prior formulations. Keeping the locomotion priors as fixed auxiliary shaping reaches stable locomotion and high reward fastest; embedding the priors in the semantic objectives or exposing them as a fourth objective delays convergence.

Type	$\lambda$	Reward $\uparrow$	Ep. len. $\uparrow$	Ctrl. $\uparrow$	cvar 10 $\uparrow$	Fall $\downarrow$	hv $\uparrow$
None	–	26.8	997.3	0.80	0.012	0.10	0.56
Action	$10^{-6}$	26.8	996.9	0.88	0.012	0.07	0.55
	$3{\times}10^{-6}$	26.6	994.9	0.78	0.012	0.07	0.74
	$\mathbf{10^{-5}}$	26.9	997.8	0.79	0.012	0.08	0.66
	$3{\times}10^{-5}$	26.8	996.4	0.86	0.011	0.11	0.59
	$10^{-4}$	26.5	996.7	0.48	0.009	0.13	0.85

Appendix

TABLE D.2: Diversity-regularization ablation. Training metrics are final-window means; evaluation metrics are averaged over three seeds.

Fig. E.1: Replay command grid. Identical recorded command sequence is replayed for all preference anchors, isolating preference-induced differences in trajectory tracking, energetic cost, and body regulation under matched command conditions.

Explore similar work

MAPL: Multi-Objective Preference Learning for Robot Locomotion
Jun 24, 2026Xiyue Chen, Muhan Lin, Shuyang Shi +1Multi-Objective Reinforcement Learning Preference Learning
Department of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47906, USA · Department of Computer Science, Purdue University, West Lafayette, IN 47906, USA · independent researcher
Learning Gait-Aware Quadruped Locomotion with Temporal Logic Specifications
Jul 1, 2026Merve Atasever, Cagan Bakirci, Alfredo Reina Corona +2Locomotion Gait Dynamics
University of Southern California, Los Angeles, California, USA
LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback
Jul 31, 2026Manith Adikari, Bei Peng, Samuele Vinanzi +1Multi-Objective Reinforcement Learning Reinforcement Learning From Human Feedback
Department of Computer Science, University of Manchester · Centre for Robotics & AI, University of Manchester · School of Computer Science, University of Sheffield +1

Jun 24, 2026, cs.ROJ/K move · Enter open · S save

MAPL: Multi-Objective Preference Learning for Robot Locomotion

Xiyue Chen, Muhan Lin, Shuyang Shi, Joseph Campbell

Department of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47906, USA · Department of Computer Science, Purdue University, West Lafayette, IN 47906, USA · independent researcher

Multi-Objective Reinforcement Learning Preference Learning Reward Functions

Open paper AskPDF arXiv

Reward design remains a major bottleneck in reinforcement learning for robot locomotion, where successful policies often depend on carefully tuned, task-specific reward functions. Preference-based reinforcement learning offers an alternative, but existing LLM-based methods typically ask for a single overall judgment between behaviors, making it difficult to capture the multiple competing objectives that underlie high-quality locomotion. We present Multi-Objective AI-Informed Preference Learning (MAPL), a framework that learns locomotion rewards from high-level natural language objectives rather than manually engineered reward equations. MAPL prompts a large language model to compare trajectories independently along semantically meaningful criteria, using generic language descriptions that are terrain-invariant and require little domain expertise. These objective-wise preferences are used to train a multi-head preference scoring model, whose outputs are aggregated to form a scalar reward for policy optimization. Across four quadruped locomotion environments, MAPL trains policies using only LLM-generated preferences and achieves performance comparable to or better than expert-designed rewards, while eliminating task-specific reward engineering.

Abstract

Explore similar work

MAPL: Multi-Objective Preference Learning for Robot Locomotion

Learning Gait-Aware Quadruped Locomotion with Temporal Logic Specifications

LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback