Post Hoc Extraction of Pareto Fronts for Continuous Control

Raghav Thakar; Gaurav Dixit; Kagan Tumer

cs.LGMar 3, 2026

Post Hoc Extraction of Pareto Fronts for Continuous Control

Authors: Raghav Thakar, Gaurav Dixit, Kagan Tumer

Organizations: The Collaborative Robotics and Intelligent Systems (CoRIS) Institute Oregon State University, Corvallis, Oregon, USA

Abstract

Agents in the real world must often balance multiple objectives, such as speed, stability, and energy efficiency in continuous control. To account for changing conditions and preferences, an agent must ideally learn a Pareto frontier of policies representing multiple optimal trade-offs. Recent advances in multi-policy multi-objective reinforcement learning (MORL) enable learning a Pareto front directly, but require full multi-objective consideration from the start of training. In practice, multi-objective preferences often arise after a policy has already been trained on a single specialised objective. Existing MORL methods cannot leverage these pre-trained `specialists' to learn Pareto fronts and avoid incurring the sample costs of retraining. We introduce Mixed Advantage Pareto Extraction (MAPEX), an offline MORL method that constructs a frontier of policies by reusing pre-trained specialist policies, critics, and replay buffers. MAPEX combines evaluations from specialist critics into a mixed advantage signal, and weights a behaviour cloning loss with it to train new policies that balance multiple objectives. MAPEX's post hoc Pareto front extraction preserves the simplicity of single-objective off-policy RL, and avoids retrofitting these algorithms into complex MORL frameworks. We formally describe the MAPEX procedure and evaluate MAPEX on five multi-objective MuJoCo environments. Given the same starting policies, MAPEX produces comparable fronts at $0.001\%$ the sample cost of established baselines.

Figures & tables

Figure 1: Sample efficiency comparison on MO-Ant-v5. (Top) Mean hypervolume

\pm

SEM vs. cumulative environment samples. MAPEX and MAPEX-PostHoc achieve high hypervolume almost instantaneously, while MOPDERL requires significantly more interaction. (Bottom) Evolution of the Pareto front approximation. MAPEX/MAPEX-PostHoc fill the front immediately, whereas MOPDERL gradually expands coverage over 300,000+ environment samples.

Figure 2: Samples required to attain target hypervolume thresholds. Comparison on MO-Hopper-v5 (top) and MO-Walker2d-v5 (bottom). Note the logarithmic scale on the y-axis. MAPEX and MAPEX-PostHoc require up to three orders of magnitude fewer samples (

10^{2}

10^{5}

) than MOPDERL to reach identical performance levels.

Figure 3: Robustness of MAPEX to specialist type and critic training. Mean hypervolume (± SEM) over generations on MO-Walker2d-v5 and MO-Ant-v5. The similar performance of standard MAPEX, MAPEX-PostHoc (offline critics), and MAPEX-TD3 (off-policy specialists) demonstrates the method’s flexibility in effectively extracting fronts from decoupled pre-trained sources.

	Hypervolume ( $\uparrow$ )			Sparsity ( $\downarrow$ )
Environment	MAPX	MOPDERL	MORL/D	MAPEX	MOPDERL	MORL/D
Ant-2obj	$1.19\text{e}7\pm 1.2\text{e}6$	$1.46\text{e}7\pm 8.2\text{e}5$	$1.41\text{e}7\pm 1.9\text{e}4$	$315.5\pm 36.1$	$\mathbf{201.4\pm 13.8}$	$289.8\pm 8.3$
Hopper-2obj	$3.34\text{e}6\pm 2.9\text{e}5$	$3.17\text{e}6\pm 2.8\text{e}5$	$\mathbf{4.31\text{e}6\pm 1.7\text{e}5}$	$209.6\pm 39.8$	$\mathbf{64.2\pm 10.3}$	$115.5\pm 17.7$
Swimmer	$1.78\text{e}5\pm 1.6\text{e}4$	$\mathbf{2.41\text{e}5\pm 2.9\text{e}4}$	$9.29\text{e}4\pm 5.9\text{e}2$	$49.5\pm 9.2$	$30.0\pm 3.2$	$\mathbf{2.9\pm 0.3}$
Walker2d	$3.09\text{e}6\pm 2.2\text{e}5$	$3.47\text{e}6\pm 3.5\text{e}5$	$\mathbf{6.52\text{e}6\pm 4.2\text{e}5}$	$157.5\pm 7.9$	$103.2\pm 16.0$	$96.9\pm 11.2$
HalfCheetah	$1.10\text{e}7\pm 5.3\text{e}5$	$1.55\text{e}7\pm 6.8\text{e}5$	$\mathbf{1.88\text{e}7\pm 5.4\text{e}4}$	$337.5\pm 26.7$	$\mathbf{101.7\pm 5.9}$	$136.4\pm 19.6$

Table 1: Performance metrics (Hypervolume and Sparsity) across v5 MO-Gymnasium MuJoCo environments. Results are Mean

\pm

SEM. Hypervolume is the space between the Pareto front and a dominated reference point and sparsity is the average euclidean distance between neighbouring points.

Figure 4: Final Pareto fronts across five MO-MuJoCo benchmarks. Comparison of fronts extracted by MAPEX against fully trained MOPDERL and MORL/D baselines. Despite relying purely on single-objective training data, MAPEX recovers fronts that are dense and competitive with baselines trained from scratch.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Env.	$d_{a}$	$d_{o}$	Objective 1	Objective 2
Ant-2obj	8	105	x-vel	y-vel
Hopper-2obj	3	11	x-vel & survival	Jump (w/ cost)
Swimmer	2	8	x-vel	Energy eff.
Walker2d	6	17	x-vel	Energy eff.
HalfCheetah	6	17	x-vel	Energy eff.

Appendix

Table 2: The v5 MO-Gymnasium MuJoCo environments. Objectives include velocity, energy efficiency, jump height, and stability and must all be maximised . The action dimensions are

d_{a}

, and

d_{o}

the observation (state) dimensions.

PDERL
Parameter	Value
Population Size	10
Mini Buffer Size	50,000
TD3 / General
Start Timesteps	$25,000$
Discount Factor ( $\gamma$ )	0.99

Appendix

Table 3: PDERL and TD3 hyperparameters (Shared across environments).

Hyperparameter	Value
Total iterations	1,200
Child buffer size	200,000
Warm-up steps / epoch	1,000
Warm-up learning rate	$3\times 10^{-4}$
Warm-up batch size	256
Evaluation episodes / actor	5

Appendix

Table 4: MAPEX hyperparameters shared across environments.

Parameter	Environment
	Ant-2obj	Hopper-2obj	Swimmer	Walker2d	HalfCheetah
MAPEX epochs	20	10	10	20	20
AWR $\beta$	0.5	0.1	1.0	0.5	1.0
AWR clip ( $\omega_{\max}$ )	1.0	20.0	20.0	20.0	1.0

Appendix

Table 5: Environment-specific MAPEX hyperparameters across five v5 MO-Gymnasium MuJoCo environments.

Environment	MOPDERL				MAPEX (specialists)		MORL/D Total
Environment	Warm up	Warm up / obj.	Stage 2	Total	Specialist / obj.	Total	MORL/D Total
MO-Ant-2obj-v5	2.0M	1.0M	2.0M	4.0M	2.0M	4.0M	4.0M
MO-Hopper-2obj-v5	2.25M	1.125M	1.75M	4.0M	2.0M	4.0M	4.0M
MO-Swimmer-v5	1.0M	0.5M	1.0M	2.0M	1.0M	2.0M	2.0M
MO-Walker2d-v5	2.0M	1.0M	2.0M	4.0M	2.0M	4.0M	4.0M
MO-HalfCheetah-v5	2.0M	1.0M	2.0M	4.0M	2.0M	4.0M	4.0M

Appendix

Table 6: Environment interaction budgets (frames) per method and environment. For bi-objective tasks (

N{=}2

), the “/ obj.” column reports frames per objective-specific training process (PDERL for MOPDERL; PDERL / TD3 for MAPEX). MAPEX extraction uses 0 additional environment interaction.

Explore similar work

Mar 20, 2026

PA2D-MORL: Pareto Ascent Directional Decomposition based Multi-Objective Reinforcement Learning

Multi-objective reinforcement learning (MORL) provides an effective solution for decision-making problems involving conflicting objectives. However, achieving high-quality approximations to the Pareto policy set remains challenging, especially in complex tasks with continuous or high-dimensional state-action space. In this paper, we propose the Pareto Ascent Directional Decomposition based Multi-Objective Reinforcement Learning (PA2D-MORL) method, which constructs an efficient scheme for multi-objective problem decomposition and policy improvement, leading to a superior approximation of Pareto policy set. The proposed method leverages Pareto ascent direction to select the scalarization weights and computes the multi-objective policy gradient, which determines the policy optimization direction and ensures joint improvement on all objectives. Meanwhile, multiple policies are selectively optimized under an evolutionary framework to approximate the Pareto frontier from different directions. Additionally, a Pareto adaptive fine-tuning approach is applied to enhance the density and spread of the Pareto frontier approximation. Experiments on various multi-objective robot control tasks show that the proposed method clearly outperforms the current state-of-the-art algorithm in terms of both quality and stability of the outcomes.

Tianmeng Hu, Biao Luo

Central South University

Multi-Objective Reinforcement Learning Robotic RL

Ask PDF

Jun 17, 2026

Pareto Q-Learning with Reward Machines

We present Pareto Q-Learning with Reward Machines (PQLRM), a multi-objective reinforcement learning algorithm for tasks whose reward structure is specified by a set of reward machines (RMs). PQLRM combines Pareto Q-Learning (PQL), which maintains sets of vector-valued Q-estimates to approximate the Pareto front, with enhancements from Q-Learning with Reward Machines (QRM), which exploits the factored automaton structure of the reward signal. This yields a multi-policy algorithm that remains sample-efficient under non-Markovian, RM-encoded rewards. Experimental trials show that PQLRM converges faster than a naive PQL baseline applied to the cross-product MDP and can synthesize Pareto-optimal policies that QRM cannot.

Arnaud Lequen, Clément Legrand-Lixon, Léo Saulières

Linköping University, Sweden · Univ. Lille, CNRS, Centrale Lille, UMR 9189 CRIStal, F-59000 Lille, France · Univ. Toulouse, INRAE-MIAT, Toulouse, France

Multi-Objective Reinforcement Learning Reinforcement Learning Q-Learning Policy Learning

Ask PDF

Jun 24, 2026

Deterministic Pareto-Optimal Policy Synthesis for Multi-Objective Reinforcement Learning

Real-world decision-making often requires balancing multiple conflicting objectives, a challenge that standard Reinforcement Learning (RL) frequently addresses by aggregating rewards into a single scalar signal. While effective for simple tasks, this approach often fails to capture the full spectrum of optimal trade-offs, known as the Pareto frontier. In this paper, we introduce a novel preference-conditioned Bellman operator, motivated from the Chebyshev scalarization, designed to compute deterministic Pareto-optimal policies for Multi-Objective Markov Decision Processes (MOMDPs). We prove that this operator satisfies an enveloping property, where the estimated value functions upper-bound the true Pareto frontier, and demonstrate that it monotonically converges to a coverage set of this frontier. Furthermore, we also show how to extract deterministic policies from these converged Q-estimates. This ensures the agent can recover a policy for any given preference, capturing the entire Pareto-optimal frontier while guaranteeing each synthesized policy remains approximately Pareto-optimal. Experimental results validate that our algorithm successfully recovers complex trade-offs, providing a solution for deterministic Pareto-optimal policy synthesis.

Aniruddha Joshi, Niklas Lauffer, Sanjit Seshia

University of California Berkeley, USA.

Multi-Objective Reinforcement Learning Multi-Objective Optimization Pareto Optimization

Ask PDF