MO-Playground: Massively Parallelized Multi-Objective Reinforcement Learning for Robotics

Neil Janwani; Ellen Novoseller; Vernon J. Lawhern; Maegan Tucker

cs.ROMar 10, 2026

MO-Playground: Massively Parallelized Multi-Objective Reinforcement Learning for Robotics

Authors: Neil Janwani, Ellen Novoseller, Vernon J. Lawhern, Maegan Tucker

Organizations: Georgia Institute of Technology, Atlanta, GA USA · DEVCOM Army Research Laboratory, MD, USA

Abstract

Multi-objective reinforcement learning (MORL) is a powerful tool to learn Pareto-optimal policy families across conflicting objectives. However, unlike traditional RL algorithms, existing MORL algorithms do not effectively leverage large-scale parallelization to concurrently simulate thousands of environments, resulting in vastly increased computation time. Ultimately, this has limited MORL's application towards complex multi-objective robotics problems. To address these challenges, we present 1) MORLAX, a new GPU-native, fast MORL algorithm, and 2) MO-Playground, a pip-installable playground of GPU-accelerated multi-objective environments. Together, MORLAX and MO-Playground approximate Pareto sets within minutes, offering 25-270x speed-ups compared to legacy CPU-based approaches whilst achieving superior Pareto front hypervolumes. We demonstrate the versatility of our approach by implementing a custom BRUCE humanoid robot environment using MO-Playground and learning Pareto-optimal locomotion policies across 6 realistic objectives for BRUCE, such as smoothness, efficiency and arm swinging.

Figures & tables

Framework	Real Robots	GPU	Multi-Obj.	Ref.
Gymnasium	✗	✗	✗	[ 25 , 26 ]
MO-Gymnasium	✗	✗	✓	[ 12 ]
MuJoCo Playground	✓	✓	✗	[ 19 ]
MO-Playground	✓	✓	✓	Ours

TABLE I : A comparison of open-source RL toolboxes for robotics.

Fig. 2 : MORLAX Architecture. (a) showcases evaluating the policy hypernetwork $\mathcal{H}_{\pi}$ by picking a trade-off $\bm{w}$ , evaluating $\mathcal{H}_{\pi}$ , and rolling out the extracted policy in the environment. (b) showcases how MORLAX trains $\mathcal{H}_{\pi}$ via a parallelized sample - rollout - update procedure.

Fig. 3 : MORLaX finds Pareto sets that achieve better-performing, diverse behavior compared to the baseline, HYPER-MORL [ 16 ] , but significantly (21-270x) faster. To visualize each environment’s objectives, Pareto fronts along with composite figures of a single MORLAX policy hypernetwork are shown across extreme input trade-offs (e.g. maximum speed vs. maximum energy efficiency in MOCheetah). The displayed trajectories are generated via policies highlighted on the Pareto Fronts.

Environment	MORLAX	HYPER-MORL	Improvement
Maximum Hypervolume Possible			Hypervol. Increase
Cheetah	$\bm{5.78\cdot 10^{6}}$	$5.58\cdot 10^{6}$	1.03x
Walker	$\bm{6.48\cdot 10^{6}}$	$5.28\cdot 10^{6}$	1.23x
Ant	$\bm{2.81\cdot 10^{7}}$	$2.70\cdot 10^{7}$	1.04x
Hopper	$\bm{1.48\cdot 10^{7}}$	$0.60\cdot 10^{7}$	2.48x
Humanoid	$\bm{5.45\cdot 10^{6}}$	$0.74\cdot 10^{5}$	7.33x

TABLE II : This table presents the comparative results between MORLAX and an existing MORL algorithm, HYPER-MORL [ 16 ] . Top : the maximum hypervolume each algorithm attained when run until convergence (5-45 min for MORLAX ). Bottom : the time taken for each algorithm to reach the lower of the two maximum hypervolumes in Top . Across all environments, MORLAX achieved larger hypervolumes faster than HYPER-MORL.

Fig. 4 : We project the learned six-dimensional Pareto front representing 30,720 policies tracking a moderate, forward velocity onto two 3D plots, each with different objectives. The colors represent trade-off vectors, and are blended according to the color-coded objectives along each axis. $\pi_{1}$ represents a policy with arm swinging, $\pi_{2}$ represents a policy with rigid arms, and $\pi_{3}$ represents a policy with maximum smoothness. One interesting finding is that $\pi_{1}$ walks faster and has higher efficiency, likely an emergent benefit of arm-swing.

Explore similar work

PRISM: Parallel Reward Integration with Symmetry for MORL
Feb 20, 2026Finn van der Knaap, Kejiang Qian, Zheng Xu +1Reinforcement Learning Multi-Objective Reinforcement Learning
University of Edinburgh · Meta Superintelligence Labs.
PA2D-MORL: Pareto Ascent Directional Decomposition based Multi-Objective Reinforcement Learning
Mar 20, 2026Tianmeng Hu, Biao LuoMulti-Objective Reinforcement Learning Robotic RL
Central South University
A Reward-Free Viewpoint on Multi-Objective Reinforcement Learning
Apr 27, 2026Ying-Tu Chen, Wei Hung, Bing-Shu Wu +2Multi-Objective Reinforcement Learning Reinforcement Learning
National Yang Ming Chiao Tung University, Hsinchu, Taiwan · Massachusetts Institute of Technology

Feb 20, 2026J/K move · Enter open · S save

PRISM: Parallel Reward Integration with Symmetry for MORL

Finn van der Knaap, Kejiang Qian, Zheng Xu, Fengxiang He

University of Edinburgh · Meta Superintelligence Labs.

Reinforcement Learning Multi-Objective Reinforcement Learning Reward Design for RL Sparse-Reward RL Dense Reward RL

Open paper AskPDF arXiv

This work studies heterogeneous Multi-Objective Reinforcement Learning (MORL), where objectives can differ sharply in temporal frequency. Such heterogeneity allows dense objectives to dominate learning, while sparse long-horizon rewards receive weak credit assignment, leading to poor sample efficiency. We propose a Parallel Reward Integration with Symmetry (PRISM) algorithm that enforces reflectional symmetry as an inductive bias in aligning reward channels. PRISM introduces ReSymNet, a theory-motivated model that reconciles temporal-frequency mismatches across objectives, using residual blocks to learn a scaled opportunity value that accelerates exploration while preserving the optimal policy. We also propose SymReg, a reflectional equivariance regulariser that enforces agent mirroring and constrains policy search to a reflection-equivariant subspace. This restriction provably reduces hypothesis complexity and improves generalisation. Across MuJoCo benchmarks, PRISM consistently outperforms both a sparse-reward baseline and an oracle trained with full dense rewards, improving Pareto coverage and distributional balance: it achieves hypervolume gains exceeding 100% over the baseline and up to 32% over the oracle. The code is at https://github.com/EVIEHub/PRISM.

Abstract

Explore similar work

PRISM: Parallel Reward Integration with Symmetry for MORL

PA2D-MORL: Pareto Ascent Directional Decomposition based Multi-Objective Reinforcement Learning

A Reward-Free Viewpoint on Multi-Objective Reinforcement Learning