Multi-objective reinforcement learning (MORL) is a powerful tool to learn Pareto-optimal policy families across conflicting objectives. However, unlike traditional RL algorithms, existing MORL algorithms do not effectively leverage large-scale parallelization to concurrently simulate thousands of environments, resulting in vastly increased computation time. Ultimately, this has limited MORL's application towards complex multi-objective robotics problems. To address these challenges, we present 1) MORLAX, a new GPU-native, fast MORL algorithm, and 2) MO-Playground, a pip-installable playground of GPU-accelerated multi-objective environments. Together, MORLAX and MO-Playground approximate Pareto sets within minutes, offering 25-270x speed-ups compared to legacy CPU-based approaches whilst achieving superior Pareto front hypervolumes. We demonstrate the versatility of our approach by implementing a custom BRUCE humanoid robot environment using MO-Playground and learning Pareto-optimal locomotion policies across 6 realistic objectives for BRUCE, such as smoothness, efficiency and arm swinging.
Figures & tables
Framework
Real Robots
GPU
Multi-Obj.
Ref.
Gymnasium
✗
✗
✗
[ 25 , 26 ]
MO-Gymnasium
✗
✗
✓
[ 12 ]
MuJoCo Playground
✓
✓
✗
[ 19 ]
MO-Playground
✓
✓
✓
Ours
TABLE I : A comparison of open-source RL toolboxes for robotics.
Fig. 2 : MORLAX Architecture. (a) showcases evaluating the policy hypernetwork Hπ by picking a trade-off w , evaluating Hπ , and rolling out the extracted policy in the environment. (b) showcases how MORLAX trains Hπ via a parallelized sample - rollout - update procedure.
Fig. 3 : MORLaX finds Pareto sets that achieve better-performing, diverse behavior compared to the baseline, HYPER-MORL [ 16 ] , but significantly (21-270x) faster. To visualize each environment’s objectives, Pareto fronts along with composite figures of a single MORLAX policy hypernetwork are shown across extreme input trade-offs (e.g. maximum speed vs. maximum energy efficiency in MOCheetah). The displayed trajectories are generated via policies highlighted on the Pareto Fronts.
Environment
MORLAX
HYPER-MORL
Improvement
Maximum Hypervolume Possible
Hypervol. Increase
Cheetah
5.78⋅106
5.58⋅106
1.03x
Walker
6.48⋅106
5.28⋅106
1.23x
Ant
2.81⋅107
2.70⋅107
1.04x
Hopper
1.48⋅107
0.60⋅107
2.48x
Humanoid
5.45⋅106
0.74⋅105
7.33x
TABLE II : This table presents the comparative results between MORLAX and an existing MORL algorithm, HYPER-MORL [ 16 ] . Top : the maximum hypervolume each algorithm attained when run until convergence (5-45 min for MORLAX ). Bottom : the time taken for each algorithm to reach the lower of the two maximum hypervolumes in Top . Across all environments, MORLAX achieved larger hypervolumes faster than HYPER-MORL.
Fig. 4 : We project the learned six-dimensional Pareto front representing 30,720 policies tracking a moderate, forward velocity onto two 3D plots, each with different objectives. The colors represent trade-off vectors, and are blended according to the color-coded objectives along each axis. π1 represents a policy with arm swinging, π2 represents a policy with rigid arms, and π3 represents a policy with maximum smoothness. One interesting finding is that π1 walks faster and has higher efficiency, likely an emergent benefit of arm-swing.
This work studies heterogeneous Multi-Objective Reinforcement Learning (MORL), where objectives can differ sharply in temporal frequency. Such heterogeneity allows dense objectives to dominate learning, while sparse long-horizon rewards receive weak credit assignment, leading to poor sample efficiency. We propose a Parallel Reward Integration with Symmetry (PRISM) algorithm that enforces reflectional symmetry as an inductive bias in aligning reward channels. PRISM introduces ReSymNet, a theory-motivated model that reconciles temporal-frequency mismatches across objectives, using residual blocks to learn a scaled opportunity value that accelerates exploration while preserving the optimal policy. We also propose SymReg, a reflectional equivariance regulariser that enforces agent mirroring and constrains policy search to a reflection-equivariant subspace. This restriction provably reduces hypothesis complexity and improves generalisation. Across MuJoCo benchmarks, PRISM consistently outperforms both a sparse-reward baseline and an oracle trained with full dense rewards, improving Pareto coverage and distributional balance: it achieves hypervolume gains exceeding 100% over the baseline and up to 32% over the oracle. The code is at https://github.com/EVIEHub/PRISM.
Finn van der Knaap, Kejiang Qian, Zheng Xu +1
University of Edinburgh · Meta Superintelligence Labs.
Multi-objective reinforcement learning (MORL) provides an effective solution for decision-making problems involving conflicting objectives. However, achieving high-quality approximations to the Pareto policy set remains challenging, especially in complex tasks with continuous or high-dimensional state-action space. In this paper, we propose the Pareto Ascent Directional Decomposition based Multi-Objective Reinforcement Learning (PA2D-MORL) method, which constructs an efficient scheme for multi-objective problem decomposition and policy improvement, leading to a superior approximation of Pareto policy set. The proposed method leverages Pareto ascent direction to select the scalarization weights and computes the multi-objective policy gradient, which determines the policy optimization direction and ensures joint improvement on all objectives. Meanwhile, multiple policies are selectively optimized under an evolutionary framework to approximate the Pareto frontier from different directions. Additionally, a Pareto adaptive fine-tuning approach is applied to enhance the density and spread of the Pareto frontier approximation. Experiments on various multi-objective robot control tasks show that the proposed method clearly outperforms the current state-of-the-art algorithm in terms of both quality and stability of the outcomes.
Many sequential decision-making tasks involve optimizing multiple conflicting objectives, requiring policies that adapt to different user preferences. In multi-objective reinforcement learning (MORL), one widely studied approach} addresses this by training a single policy network conditioned on preference-weighted rewards. In this paper, we explore a novel algorithmic perspective: leveraging reward-free reinforcement learning (RFRL) for MORL. While RFRL has historically been studied independently of MORL, it learns optimal policies for any possible reward function, making it a natural fit for MORL's challenge of handling unknown user preferences. We propose using the RFRL's training objective as an auxiliary task to enhance MORL, enabling more effective knowledge sharing beyond the multi-objective reward function given at training time. To this end, we adapt a state-of-the-art RFRL algorithm to the MORL setting and introduce a preference-guided exploration strategy that focuses learning on relevant parts of the environment. Through extensive experiments and ablation studies, we demonstrate that our approach significantly outperforms the state-of-the-art MORL methods across diverse MO-Gymnasium tasks, achieving superior performance and data efficiency. This work provides the first systematic adaptation of RFRL to MORL, demonstrating its potential as a scalable and empirically effective solution to multi-objective policy learning.
Ying-Tu Chen, Wei Hung, Bing-Shu Wu +2
National Yang Ming Chiao Tung University, Hsinchu, Taiwan · Massachusetts Institute of Technology