PROMO: Preference-conditioned Multi-Objective Reinforcement Learning for Quadrupedal Robots

Amr Mousa; Rifny Rachman; Neil Karavis; Michele Caprio; Richard Allmendinger

cs.ROOct 1, 2026

PROMO: Preference-conditioned Multi-Objective Reinforcement Learning for Quadrupedal Robots

Authors: Amr Mousa, Rifny Rachman, Neil Karavis, Michele Caprio, Richard Allmendinger

Organizations: The University of Manchester, United Kingdom · BAE Systems, United Kingdom · University of Warwick, United Kingdom

Abstract

Quadrupedal locomotion requires balancing conflicting objectives such as command tracking, stability, and energy efficiency, yet conventional reinforcement learning (RL) hardcodes these priorities into a fixed scalar reward at training time. We present PROMO (Preference-Conditioned Multi-Objective Reinforcement Learning), a semantic multi-objective approach that makes this trade-off an explicit runtime input to a single locomotion policy. PROMO conditions the policy on deployment facing preferences while keeping embodiment-specific locomotion priors fixed, thereby separating operator intent from reward shaping terms required for viable gait generation. Compared with fixed-objective controllers, multi-objective baselines, and independently trained specialists, PROMO achieves objective specialization and robustness from a single deployable policy. Across 100 sampled preferences in simulation, 67 behaviors are non-dominated under exact Pareto dominance, with a mean preference-objective correlation of 0.843, demonstrating broad Pareto coverage and predictable preference response. The same policy transfers zero-shot to a Unitree Go2, where preference changes alone reduce specific energy by up to 30.4%, position error by 38.7%, and peak body-attitude deviation by 59.0% relative to the balanced preference. These results establish preference-conditioned multi-objective RL as a practical runtime interface for adaptive legged locomotion, extending its role beyond offline Pareto-set construction. Open-source code and videos are available at https://amrmousa.com/promo/.

Figures & tables

Panel of Figure 1 — Fig. 3: Training optimization over three seeds (mean, $\pm 1$ s.d.). The top row reports mean reward and the bottom row reports mean episode length for locomotion baselines (a,d), morl baselines (b,e), and independently trained specialists (c,f). promo converges to a higher reward than the shared-scale locomotion and morl references while matching the episode-length convergence of the balanced specialist, learning the preference space at roughly the optimization cost of a single controller. Specialist rewards are shown for completeness but use differently scaled scalar objectives, so episode length is the more informative cross-specialist comparison.

Method	Tracking $\uparrow$	Stability $\uparrow$	Efficiency $\uparrow$	Fall $\downarrow$	cvar tilt $\downarrow$
tar	$14.921$	$\mathbf{-2.218}$	$-3.643$	$14.5\%$	$0.116$
rma	$13.742$	$-2.920$	$\mathbf{-3.423}$	$18.2\%$	$0.125$
promo - sorl	$16.225$	$-2.804$	$-4.157$	$11.0\%$	$0.109$
promo (ours)	$\mathbf{18.703}$	$-2.855$	$-5.366$	$\mathbf{7.4\%}$	$\mathbf{0.105}$

TABLE I: Common balanced locomotion evaluation. Higher is better for semantic rewards; lower is better for fall and cvar tilt. Fall values are percentages. Bold denotes the best mean in each column.

Method	Tracking $T\uparrow$	Stability $S\uparrow$	Efficiency $E\uparrow$	Fall $\downarrow$	Scalarized return $\uparrow$	4-anchor Ctrl. $\uparrow$
dpmorl	$14.430$	$-2.352$	$\mathbf{-2.133}$	$26.6\%$	$2.986$	$--$
moppo	$15.326$	$-2.305$	$-2.794$	$19.4\%$	$2.212$	$0.492$
Specialists	$17.833$	$\mathbf{-1.916}$	$-2.942$	$12.6\%$	$3.255$	$--$
promo	$\mathbf{20.099}$	$-2.245$	$-2.189$	$\mathbf{8.4\%}$	$\mathbf{3.964}$	$\mathbf{0.662}$

TABLE II: Four-anchor semantic evaluation against morl and specialist baselines. Bold denotes the best mean in each column.

Panel of Figure 4 — Fig. 4: Standalone simulation results for promo policy evaluated at 100 sampled preference vectors. Each sampled preference produces one policy evaluation; panels (a)–(c) show different 2-D projections of these same 100 evaluations, with Pareto-nondominated and dominated samples distinguished visually. Tracking and efficiency form the clearest conflict, while stability and efficiency are largely complementary. Panels (d)–(f) show marginal per-objective preference-to-behavior mappings: each realized objective return rises near-monotonically with its corresponding preference weight, while the spread reflects variation in the other two simplex weights.

Panel of Figure 5 — Fig. 5: Preference-induced gait adaptation on hardware. Chronophotographic overlays with 2-s tracked foot trajectories for tracking-, stability-, and efficiency-heavy preferences (left to right) using the same deployed promo policy. Tracking produces larger foot excursions, efficiency produces compact low-clearance trajectories, and stability exhibits an intermediate gait with increased clearance in this sequence.

Preference	Vel. RMSE [m/s]	Pos. MAE [m]	Energy [J/m]	cot [-]	Peak att. [rad]	Act. rate [rad/s]	Trq. rate [N m/s]
Replay-round
Balanced	$0.444$	$0.534$	$124.6$	$1.058$	$0.196$	$8.45$	$240.7$
Efficiency	$0.444$	$0.624$	$\mathbf{102.3}$	$\mathbf{0.869}$	$0.187$	$\mathbf{5.42}$	$\mathbf{153.4}$
Stability	$\mathbf{0.428}$	$0.538$	$132.4$	$1.125$	$\mathbf{0.158}$	$9.26$	$263.0$
Tracking	$0.493$	$\mathbf{0.405}$	$169.6$	$1.441$	$0.281$	$11.98$	$342.9$
Move-fast

TABLE III: Real-robot evaluation of a single promo policy under four semantic preferences and three command regimes on the Unitree Go2. Values are means over

n=5

trials per condition; bold denotes the preferred value within each regime.

Fig. 6: Preference-conditioned hardware behavior. Tracking mae , peak attitude deviation, and specific energy for the same deployed promo policy under four semantic preferences across replay, fast, and slow regimes. Bars show means over five hardware trials and error bars denote one standard deviation. Lower is better for all metrics. The corresponding preference generally shifts behavior toward its intended semantic objective, while cross-objective differences expose the resulting physical trade-offs.

Fig. 7: Disturbance recovery on a low-traction surface. Representative sequences using the balanced preference on a low-traction surface. Top is whole-body recovery following a disturbance at the head. Middle is recovery from a rear-left hip disturbance inducing approximately $230^{\circ}$ of yaw rotation. Bottom is recovery from direct disturbance of the front-right leg, requiring adjustment of the support configuration.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Term	Group	Coefficient	Physical interpretation
track_lin_vel_xy_exp	Tracking	$1.5$	Rewards planar velocity tracking with kernel standard deviation $0.5$
track_ang_vel_z_exp	Tracking	$0.75$	Rewards yaw-rate tracking with kernel standard deviation $0.5$
lin_vel_z_l2	Stability	$-2.0$	Penalizes vertical base motion
ang_vel_xy_l2	Stability	$-0.05$	Penalizes roll and pitch angular velocity
flat_orientation_l2	Stability	$-2.5$	Penalizes body tilt away from upright posture
joint_torques_l2	Efficiency	$-2.0\times 10^{-4}$	Penalizes squared applied torque

Appendix

TABLE A.1: Semantic objectives and fixed locomotion priors used by promo .

Protocol	Preference specification	Purpose
Training distribution	$w_{i}=0.1+0.7x_{i}$ , $x\sim\mathrm{Dirichlet}(1,1,1)$	Uniform truncated-simplex coverage without zeroed objectives
Balanced	$(1/3,1/3,1/3)$	Nominal semantic trade-off
Tracking-heavy	$(0.8,0.1,0.1)$	Tracking-focused anchor
Stability-heavy	$(0.1,0.8,0.1)$	Stability-focused anchor
Efficiency-heavy	$(0.1,0.1,0.8)$	Efficiency-focused anchor
Two-objective mixtures	$(0.45,0.45,0.1)$ , $(0.45,0.1,0.45)$ , $(0.1,0.45,0.45)$	Representative simplex edges

Appendix

TABLE A.2: Preference configurations used for training and evaluation.

Item	Value
Robot	Unitree Go2, $12$ - dof quadruped
Simulator	Isaac Sim / IsaacLab
Actor hidden sizes	$[512,256,128]$
Critic shared trunk	$[512,256,128]$
Semantic critic heads	$K=3$ scalar heads
Privileged encoder	MLP over privileged state and preference

Appendix

TABLE B.1: promo network, simulation, and control configuration.

Parameter	Range / setting
Static and dynamic friction	$[0.6,\,1.2]$
Restitution coefficient	$[0,\,1]$
Base mass offset	$[0,\,2]$ kg
Force perturbation	$\pm 10$ N
Torque perturbation	$\pm 5$ N m
Velocity perturbation	$\pm 2$ m/s every $2$ – $4$ s

Appendix

TABLE B.2: Domain randomization and perturbation ranges used during training.

Parameter	Value
Parallel environments	$4096$
Rollout horizon	$50$ policy steps per env
Discount factor $\gamma$	$0.99$
gae parameter $\lambda_{\mathrm{GAE}}$	$0.95$
ppo clipping $\epsilon$	$0.12$
Entropy coefficient	$0.005$

Appendix

TABLE B.3: Training and optimization configuration.

Metric	Value
Mean utility per step	$0.0149$
cvar 10 utility	$0.0120$
Fall rate	$7.50\%$
Preference–objective correlation	$0.8434$
tracking $\rho_{T}$	$0.7748$
stability $\rho_{S}$	$0.8289$

Appendix

TABLE C.1: Key interface metrics for promo over

100

preferences.

$\varepsilon$ threshold	$0$	$0.005$	$0.01$	$0.02$	$0.03$
Non-dominated solutions	$67$	$50$	$37$	$25$	$14$
Non-dominated set retained (%)	$100$	$74.6$	$55.2$	$37.3$	$20.9$

Appendix

TABLE C.2:

\varepsilon

-dominance pruning of the 100-preference evaluation.

Fig. C.2: Semantic preference simplex for promo . The simplex defines the deployment-facing trade-off among tracking, stability, and efficiency; the marked anchors are used for representative simulation and hardware evaluations, while the dense evaluation samples the interior.

Regime	Command protocol	Evaluation focus
Replay	Recorded, identical	Command-matched comparison
Fast	Manual joystick	Dynamic locomotion
Slow	Manual joystick	Near-standstill locomotion

Appendix

TABLE C.3: Real-robot evaluation regimes.

Variant	Reward $\uparrow$	Ep. len. $\uparrow$	Ctrl. $\uparrow$	Fall $\downarrow$	hv $\uparrow$
Fixed priors (ours)	$\mathbf{27.06}$	$\mathbf{997.56}$	$\mathbf{0.741}$	$\mathbf{0.060}$	$0.693$
Prior removed	$25.63$	$956.54$	$0.638$	$0.131$	$\mathbf{0.906}$
Priors inside objectives	$26.72$	$996.60$	$0.699$	$0.165$	$0.626$
Prior as 4th objective	$26.86$	$996.72$	$0.447$	$0.218$	$0.512$

Appendix

TABLE D.1: Effect of objective/prior factorization on comparable training and interface metrics.

Panel of Figure 19 — Fig. D.1: Training behavior under alternative objective/prior formulations. Keeping the locomotion priors as fixed auxiliary shaping reaches stable locomotion and high reward fastest; embedding the priors in the semantic objectives or exposing them as a fourth objective delays convergence.

Type	$\lambda$	Reward $\uparrow$	Ep. len. $\uparrow$	Ctrl. $\uparrow$	cvar 10 $\uparrow$	Fall $\downarrow$	hv $\uparrow$
None	–	26.8	997.3	0.80	0.012	0.10	0.56
Action	$10^{-6}$	26.8	996.9	0.88	0.012	0.07	0.55
	$3{\times}10^{-6}$	26.6	994.9	0.78	0.012	0.07	0.74
	$\mathbf{10^{-5}}$	26.9	997.8	0.79	0.012	0.08	0.66
	$3{\times}10^{-5}$	26.8	996.4	0.86	0.011	0.11	0.59
	$10^{-4}$	26.5	996.7	0.48	0.009	0.13	0.85

Appendix

TABLE D.2: Diversity-regularization ablation. Training metrics are final-window means; evaluation metrics are averaged over three seeds.

Fig. E.1: Replay command grid. Identical recorded command sequence is replayed for all preference anchors, isolating preference-induced differences in trajectory tracking, energetic cost, and body regulation under matched command conditions.

Explore similar work

Jun 24, 2026cs.RO

MAPL: Multi-Objective Preference Learning for Robot Locomotion

Reward design remains a major bottleneck in reinforcement learning for robot locomotion, where successful policies often depend on carefully tuned, task-specific reward functions. Preference-based reinforcement learning offers an alternative, but existing LLM-based methods typically ask for a single overall judgment between behaviors, making it difficult to capture the multiple competing objectives that underlie high-quality locomotion. We present Multi-Objective AI-Informed Preference Learning (MAPL), a framework that learns locomotion rewards from high-level natural language objectives rather than manually engineered reward equations. MAPL prompts a large language model to compare trajectories independently along semantically meaningful criteria, using generic language descriptions that are terrain-invariant and require little domain expertise. These objective-wise preferences are used to train a multi-head preference scoring model, whose outputs are aggregated to form a scalar reward for policy optimization. Across four quadruped locomotion environments, MAPL trains policies using only LLM-generated preferences and achieves performance comparable to or better than expert-designed rewards, while eliminating task-specific reward engineering.

Xiyue Chen, Muhan Lin, Shuyang Shi +1

Department of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47906, USA · Department of Computer Science, Purdue University, West Lafayette, IN 47906, USA · independent researcher

Multi-Objective Reinforcement Learning Preference Learning Reward Functions

Ask PDF

Jul 1, 2026cs.RO

Learning Gait-Aware Quadruped Locomotion with Temporal Logic Specifications

Reinforcement learning (RL) for quadruped locomotion commonly depends on fixed, hand-crafted, and Markovian reward functions that limit both interpretability of learned policies and lack explicit control over gait behaviors. We introduce a framework where distinct gaits are specified using parameterized constraints expressed in Signal Temporal Logic (STL). These include safety bounds, gait synchronization constraints, command tracking, and actuation bounds. From these specifications, we develop a reward shaping mechanism that provides learning agents a dense, continuous reward landscape that encodes desired behavior. We define parametric STL templates for three speed regimes (walking-trot, trot, bound), calibrate their parameters from reference rollouts, and compute rewards from using smooth approximations of STL robustness over the rollouts. The generated rewards can be used to provide shaped gradients compatible with Proximal Policy Optimization (PPO). We instantiate the approach on Google's Barkour quadruped robot in MuJoCo XLA (MJX). We use parallelization within the simulator to improve training speeds and use domain randomization to robustify learned policies. We show that compared to a baseline of hand-crafted rewards, the STL-shaped rewards yield tighter velocity tracking and more stable training. Videos can be found on our project website: https://stl-locomotion.github.io/.

Merve Atasever, Cagan Bakirci, Alfredo Reina Corona +2

University of Southern California, Los Angeles, California, USA

Locomotion Gait Dynamics Signal Temporal Logic

Ask PDF

Jul 31, 2026cs.AI

LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback

Reinforcement Learning (RL) systems are typically trained using a single, well-specified scalar reward function. However, real-world decision-making tasks often involve multiple, competing objectives, such as performance versus efficiency, where ground-truth reward functions are difficult to specify or inaccessible. While Multi-Objective RL (MORL) addresses such trade-offs by modeling rewards as vectors, existing approaches typically assume access to a well-specified reward function for each objective, inheriting the same challenges faced by single-objective RL. Meanwhile, Preference-based RL (PbRL) has shown great potential in solving complex tasks without access to a pre-defined reward function through reward learning from human feedback, yet has largely been studied in single-objective settings. In this work, we bridge this gap with LEMUR: Learning to Align with Multi-Objective Reinforcement Learning with Preference feedback, a novel framework where an agent interactively learns from the preferences of multiple humans to learn optimal multi-objective policies. Our approach jointly learns policies and multiple objective-specific reward models from human feedback, enabling agents to effectively balance competing objectives during learning. We evaluate LEMUR on a variety of benchmark multi-objective tasks, and empirical results demonstrate its superior performance over baseline methods. Our method presents a promising direction for solving multi-objective decision-making tasks without pre-defined reward functions.

Manith Adikari, Bei Peng, Samuele Vinanzi +1

Department of Computer Science, University of Manchester · Centre for Robotics & AI, University of Manchester · School of Computer Science, University of Sheffield +1

Multi-Objective Reinforcement Learning Reinforcement Learning From Human Feedback Reward Functions Bird

Ask PDF