PA2D-MORL: Pareto Ascent Directional Decomposition based Multi-Objective Reinforcement Learning

Tianmeng Hu; Biao Luo

cs.AIMar 20, 2026

PA2D-MORL: Pareto Ascent Directional Decomposition based Multi-Objective Reinforcement Learning

Authors: Tianmeng Hu, Biao Luo

Organizations: Central South University

Abstract

Multi-objective reinforcement learning (MORL) provides an effective solution for decision-making problems involving conflicting objectives. However, achieving high-quality approximations to the Pareto policy set remains challenging, especially in complex tasks with continuous or high-dimensional state-action space. In this paper, we propose the Pareto Ascent Directional Decomposition based Multi-Objective Reinforcement Learning (PA2D-MORL) method, which constructs an efficient scheme for multi-objective problem decomposition and policy improvement, leading to a superior approximation of Pareto policy set. The proposed method leverages Pareto ascent direction to select the scalarization weights and computes the multi-objective policy gradient, which determines the policy optimization direction and ensures joint improvement on all objectives. Meanwhile, multiple policies are selectively optimized under an evolutionary framework to approximate the Pareto frontier from different directions. Additionally, a Pareto adaptive fine-tuning approach is applied to enhance the density and spread of the Pareto frontier approximation. Experiments on various multi-objective robot control tasks show that the proposed method clearly outperforms the current state-of-the-art algorithm in terms of both quality and stability of the outcomes.

Figures & tables

Figure 1: Illustration of the PGR and PA-FT methods. (a) A portion of the better performing policies in each partition is selected as candidate policies, and then one is randomly selected from the candidates. (b) The larger missing regions in the current Pareto frontier approximation are identified. Policies around the missing region are selected and fine-tuned.

		PA2D-MORL	PA2D-ablated	PGMORL	PFA	MOEA/D
Walker2d	HV $\uparrow(\times 10^{6})$	$\mathbf{5.743\pm 0.121}$	$5.320\pm 0.186$	$4.849\pm 0.558$	$4.329\pm 0.553$	$4.612\pm 0.545$
Walker2d	SP $\downarrow(\times 10^{4})$	$\mathbf{0.014\pm 0.006}$	$0.180\pm 0.096$	$0.021\pm 0.018$	$0.309\pm 0.225$	$0.710\pm 0.285$
Humanoid	HV $\uparrow(\times 10^{6})$	$\mathbf{51.23\pm 2.66}$	$42.93\pm 4.14$	$44.75\pm 5.81$	$40.55\pm 5.02$	$46.35\pm 7.33$
Humanoid	SP $\downarrow(\times 10^{4})$	$\mathbf{0.133\pm 0.031}$	$0.274\pm 0.177$	$0.255\pm 0.121$	$0.715\pm 0.516$	$2.871\pm 1.342$
HalfCheetah	HV $\uparrow(\times 10^{6})$	$\mathbf{5.787\pm 0.020}$	$5.741\pm 0.053$	$5.782\pm 0.018$	$5.765\pm 0.081$	$5.739\pm 0.075$
HalfCheetah	SP $\downarrow(\times 10^{4})$	$0.026\pm 0.013$	$0.106\pm 0.035$	$\mathbf{0.022\pm 0.015}$	$0.548\pm 0.209$	$0.679\pm 0.295$

Table 1: Evaluation results for all algorithms. The experiments are based on 7 MuJoCo environments. Both the average and standard deviation of the HV and SP metrics are reported. The modified PFA is difficult to apply to the 3-objective task and is not tested on Hopper-3. All data are based on 6 independent runs.

Figure 2: Hypervolume and sparsity curves on the Walker environment. The light-colored parts show the standard deviation. Data are based on 6 independent runs.

Figure 3: Comparison of Pareto frontier approximations. Results of PA2D-MORL, PGMORL, and MOEA/D are shown. PA2D-MORL achieves a higher-quality policy set.

Figure 4: Comparison of Pareto frontier approximations. Results of PA2D-MORL and PA2D-ablated are shown. PA2D-MORL achieves a denser policy set.

$p$	metrics	Walker2d	Humanoid
$6$	HV $\uparrow(\times 10^{6})$	$5.652\pm 0.193$	$50.89\pm 3.07$
$6$	SP $\downarrow(\times 10^{4})$	$0.017\pm 0.009$	$0.130\pm 0.053$
$8$	HV $\uparrow(\times 10^{6})$	$5.743\pm 0.121$	$51.23\pm 2.66$
$8$	SP $\downarrow(\times 10^{4})$	$0.014\pm 0.006$	$0.133\pm 0.031$
$10$	HV $\uparrow(\times 10^{6})$	$5.774\pm 0.140$	$52.37\pm 2.31$
$10$	SP $\downarrow(\times 10^{4})$	$0.010\pm 0.003$	$0.105\pm 0.044$

Table 2: Comparison of PA2D-MORL with different

p

$M_{ft}$	metrics	Walker2d	Humanoid
$\frac{1}{3}M$	HV $\uparrow(\times 10^{6})$	$5.743\pm 0.121$	$51.23\pm 2.66$
$\frac{1}{3}M$	SP $\downarrow(\times 10^{4})$	$0.014\pm 0.006$	$0.133\pm 0.031$
$\frac{1}{2}M$	HV $\uparrow(\times 10^{6})$	$5.761\pm 0.317$	$51.10\pm 2.87$
$\frac{1}{2}M$	SP $\downarrow(\times 10^{4})$	$0.022\pm 0.013$	$0.138\pm 0.028$
$\frac{4}{5}M$	HV $\uparrow(\times 10^{6})$	$5.437\pm 0.259$	$44.86\pm 4.83$
$\frac{4}{5}M$	SP $\downarrow(\times 10^{4})$	$0.131\pm 0.090$	$0.271\pm 0.109$

Table 3: Comparison of PA2D-MORL with different

M_{ft}

Multi-Objective Reinforcement Learning

Multi-objective reinforcement learning studies how agents can balance competing goals and learn policies for different trade-offs.

See all 133 papers

A Reward-Free Viewpoint on Multi-Objective Reinforcement Learning
Apr 27, 2026Ying-Tu Chen, Wei Hung, Bing-Shu Wu +2Multi-Objective Reinforcement Learning Reinforcement Learning
National Yang Ming Chiao Tung University, Hsinchu, Taiwan · Massachusetts Institute of Technology
Deterministic Pareto-Optimal Policy Synthesis for Multi-Objective Reinforcement Learning
Jun 24, 2026Aniruddha Joshi, Niklas Lauffer, Sanjit SeshiaMulti-Objective Reinforcement Learning Multi-Objective Optimization
University of California Berkeley, USA.
Post Hoc Extraction of Pareto Fronts for Continuous Control
Mar 3, 2026Raghav Thakar, Gaurav Dixit, Kagan TumerMulti-Objective Reinforcement Learning Pareto Optimization
The Collaborative Robotics and Intelligent Systems (CoRIS) Institute Oregon State University, Corvallis, Oregon, USA

Apr 27, 2026J/K move · Enter open · S save

A Reward-Free Viewpoint on Multi-Objective Reinforcement Learning

Ying-Tu Chen, Wei Hung, Bing-Shu Wu, Zhang-Wei Hong+1

National Yang Ming Chiao Tung University, Hsinchu, Taiwan · Massachusetts Institute of Technology

Multi-Objective Reinforcement Learning Reinforcement Learning Reward-Free RL

Open paper AskPDF arXiv

Many sequential decision-making tasks involve optimizing multiple conflicting objectives, requiring policies that adapt to different user preferences. In multi-objective reinforcement learning (MORL), one widely studied approach} addresses this by training a single policy network conditioned on preference-weighted rewards. In this paper, we explore a novel algorithmic perspective: leveraging reward-free reinforcement learning (RFRL) for MORL. While RFRL has historically been studied independently of MORL, it learns optimal policies for any possible reward function, making it a natural fit for MORL's challenge of handling unknown user preferences. We propose using the RFRL's training objective as an auxiliary task to enhance MORL, enabling more effective knowledge sharing beyond the multi-objective reward function given at training time. To this end, we adapt a state-of-the-art RFRL algorithm to the MORL setting and introduce a preference-guided exploration strategy that focuses learning on relevant parts of the environment. Through extensive experiments and ablation studies, we demonstrate that our approach significantly outperforms the state-of-the-art MORL methods across diverse MO-Gymnasium tasks, achieving superior performance and data efficiency. This work provides the first systematic adaptation of RFRL to MORL, demonstrating its potential as a scalable and empirically effective solution to multi-objective policy learning.

Abstract

Multi-Objective Reinforcement Learning

Explore similar work

A Reward-Free Viewpoint on Multi-Objective Reinforcement Learning

Deterministic Pareto-Optimal Policy Synthesis for Multi-Objective Reinforcement Learning

Post Hoc Extraction of Pareto Fronts for Continuous Control