Multi-objective reinforcement learning is a fundamental problem for modern AI, with applications spanning post-training, reasoning models, and robotic control. A central challenge is to learn policies that balance competing objectives, such as helpfulness versus honesty in safety and alignment, or speed versus energy in robotics. The optimal policy can vary sharply with objective-weighting combinations, and the extent of this variation depends crucially on the degree of conflict among objectives, making per-preference policy learning sample-inefficient. In this paper, we introduce objective-conflict coefficients and derive near-optimal regret bounds for online settings using this measure, thereby quantifying the impact of misalignment on learning efficiency. Then, we design routed ensemble policy learning, an algorithm that maintains a small set of basis policies and dynamically routes trajectories to the policy with the highest potential reward. Finally, we consider trajectory selection as another approach to mitigate objective conflict, and design a kernel surrogate trajectory attribution algorithm that estimates how feedback signals from a set of trajectories influence policy learning on other trajectories. The kernel can capture nonlinear interactions such as synergy and antagonism between trajectories. We evaluate our approach across diverse environments and benchmarks: routed ensembles improve average performance by 4.3% over baselines for online environments from Meta-World and MO-Gymnasium, and kernel surrogate attribution improves correlation with leave-one-out analysis over existing baselines by 50%. When applied to trajectory selection, kernel-based attribution yields a 27% improvement in reward over baseline attribution methods.
Figures & tables
Figure 1: We visualize the objective space returns (r1,r2) for policies optimized under varying preference weights w∈[0,1] in three canonical regimes — clustered, linear and concave Pareto frontier. Light yellow indicates that w is closer to 1 , while dark blue indicates w is closer to 0 .
Figure 2: We compare the cumulative regret of routed ensembles (red) against MO-UCBVI (green dashed), uniform ensembles (yellow), and the best stationary policy (blue dotted).
Metrics
Soft Modularization
PaCo
CARE
Uniform Assignment
Algorithm 1
Meta-World
Success rate
82.0±1.1%
73.1±1.1%
80.2±1.8%
81.3±1.4%
85.6±1.8%
Metrics
PGMORL
Q-Pensieve
CAPQL
Uniform Assignment
Algorithm 1
MO-HalfCheetah2d
HV ( ×106 )
0.11±0.08
4.85±0.06
3.29±0.18
3.17±0.11
5.17±0.16
UT ( ×102 )
0.12±0.13
2.14±0.13
0.63±0.09
1.01±0.12
2.47±0.14
MO-Hopper
HV ( ×107 )
1.23±0.04
1.61±0.32
1.39±0.13
1.56±0.16
1.83±0.14
UT ( ×102 )
0.04±0.02
1.47±0.14
1.29±0.12
1.17±0.09
1.56±0.06
Table 1: Comparison of our approach with several baseline methods. We report the average success rate on Meta-World, HV and UT on MuJoCo, as well as HV and Cardinality on Grid-World.
Synthetic
Qwen3-1.7B
Llama-3.2-1B
Method
HelpSteer2
UltraFeedback
HelpSteer2
UltraFeedback
Single Policy
0.48
0.51
0.46
0.34
0.52
Uniform Assignment
0.49
0.47
0.47
0.35
0.54
Algorithm 1
0.56
0.54
0.48
0.35
0.55
Table 2: Comparison results for RLHF settings between our approach, using a single policy, and uniform assignment to the same number of basis policies.
Table 5
Methods
Multitask Minigrid
Meta-World MT10
Synergy MTRL
LDS
Reward
LDS
Reward
LDS
Reward
Influence functions
−0.23±0.08
0.07±0.13
0.71±0.11
16.3±1.4
0.04±0.16
1.77±0.67
TracIn
0.31±0.06
0.25±0.21
0.45±0.15
16.0±2.3
0.16±0.04
2.65±0.03
Linear surrogates
0.37±0.12
0.33±0.19
0.42±0.19
15.8±4.5
0.16±0.04
2.67±0.01
Algorithm 2
0.41±0.04
0.45±0.17
0.80±0.04
18.8±5.6
0.91±0.01
5.85±0.01
Table 3: We report the average target reward when the target policy is trained with source tasks selected by each attribution algorithm.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Distance
MT10
CartPole
Highway
Lunarlander
0.1%
0.01±0.01%
0.12±0.14%
0.02±0.02%
0.06±0.01%
0.5%
0.43±0.73%
0.73±0.10%
0.11±0.09%
0.03±0.02%
1.0%
0.32±0.56%
0.98±0.65%
2.04±0.58%
0.48±0.01%
Appendix
Table 4: We empirically find that the approximation error of ϵ is negligible for several interactive RL environments. We report the mean and standard deviation of the approximation error across 10 random subsets. We defer a detailed description of the setup to the experiments section.
Figure 5: Correlation analysis of attribution scores against LOO scores (left) and objective conflict coefficient Γ (right).
Parameter
Meta-World
MO-Gymnasium
Number of basis policies
3
3
Base optimizer
Adam
Adam
SAC buffer size
106
106
Learning rate (Actor)
3×10−4
3×10−4
Learning rate (Critic)
3×10−4
3×10−4
Discount factor
0.99
0.99
Appendix
Table 5: Hyper-parameters for the routed ensemble procedure used in Algorithm 1 .
Setting
Basis construction
Number of policies
Mean ϵ
Max ϵ
Tabular, aligned
Exact
1
∼10−18
∼10−17
Tabular, linear
Exact
2
∼10−9
∼10−8
Tabular, concave
Exact
3
0.186
0.55
Tabular, concave
Exact
21
1.45×10−4
6.10×10−4
MO-Hopper
Candidate-selected
3
0.443
0.80
MO-Hopper
Candidate-selected
15
0.231
0.51
Appendix
Table 6: Convex-hull occupancy residuals for exact tabular occupancies and empirical benchmark visitation distributions. Mean and maximum residuals are computed over the evaluated preferences.
Deep Sea Treasure
Fruit Tree
Influence functions
0.28±0.51
0.30±0.57
Linear surrogate models
0.62±0.39
0.71±0.53
TracIn
0.70±0.40
0.75±0.45
Algorithm 2
0.75±0.14
0.81±0.18
Appendix
Table 7: Summary of trajectory attribution results on two MO-Gymnasium benchmarks, Deep Sea Treasure and Fruit Tree.
Figure 6: Ablation analysis of the choice of the ensemble size N , measured by hypervolume (HV).
Reward design has been one of the central challenges for real world reinforcement learning (RL) deployment, especially in settings with multiple objectives. Preference-based RL offers an appealing alternative by learning from human preferences over pairs of behavioural outcomes. More recently, RL from AI feedback (RLAIF) has demonstrated that large language models (LLMs) can generate preference labels at scale, mitigating the reliance on human annotators. However, existing RLAIF work typically focuses only on single-objective tasks, leaving the open question of how RLAIF handles systems that involve multiple objectives. In such systems trade-offs among conflicting objectives are difficult to specify, and policies risk collapsing into optimizing for a dominant goal. In this paper, we explore the extension of the RLAIF paradigm to multi-objective self-adaptive systems. We show that multi-objective RLAIF can produce policies that yield balanced trade-offs reflecting different user priorities without laborious reward engineering. We argue that integrating RLAIF into multi-objective RL offers a scalable path toward user-aligned policy learning in domains with inherently conflicting objectives.
Reinforcement Learning (RL) systems are typically trained using a single, well-specified scalar reward function. However, real-world decision-making tasks often involve multiple, competing objectives, such as performance versus efficiency, where ground-truth reward functions are difficult to specify or inaccessible. While Multi-Objective RL (MORL) addresses such trade-offs by modeling rewards as vectors, existing approaches typically assume access to a well-specified reward function for each objective, inheriting the same challenges faced by single-objective RL. Meanwhile, Preference-based RL (PbRL) has shown great potential in solving complex tasks without access to a pre-defined reward function through reward learning from human feedback, yet has largely been studied in single-objective settings. In this work, we bridge this gap with LEMUR: Learning to Align with Multi-Objective Reinforcement Learning with Preference feedback, a novel framework where an agent interactively learns from the preferences of multiple humans to learn optimal multi-objective policies. Our approach jointly learns policies and multiple objective-specific reward models from human feedback, enabling agents to effectively balance competing objectives during learning. We evaluate LEMUR on a variety of benchmark multi-objective tasks, and empirical results demonstrate its superior performance over baseline methods. Our method presents a promising direction for solving multi-objective decision-making tasks without pre-defined reward functions.
Manith Adikari, Bei Peng, Samuele Vinanzi +1
Department of Computer Science, University of Manchester · Centre for Robotics & AI, University of Manchester · School of Computer Science, University of Sheffield +1
Many sequential decision-making tasks involve optimizing multiple conflicting objectives, requiring policies that adapt to different user preferences. In multi-objective reinforcement learning (MORL), one widely studied approach} addresses this by training a single policy network conditioned on preference-weighted rewards. In this paper, we explore a novel algorithmic perspective: leveraging reward-free reinforcement learning (RFRL) for MORL. While RFRL has historically been studied independently of MORL, it learns optimal policies for any possible reward function, making it a natural fit for MORL's challenge of handling unknown user preferences. We propose using the RFRL's training objective as an auxiliary task to enhance MORL, enabling more effective knowledge sharing beyond the multi-objective reward function given at training time. To this end, we adapt a state-of-the-art RFRL algorithm to the MORL setting and introduce a preference-guided exploration strategy that focuses learning on relevant parts of the environment. Through extensive experiments and ablation studies, we demonstrate that our approach significantly outperforms the state-of-the-art MORL methods across diverse MO-Gymnasium tasks, achieving superior performance and data efficiency. This work provides the first systematic adaptation of RFRL to MORL, demonstrating its potential as a scalable and empirically effective solution to multi-objective policy learning.
Ying-Tu Chen, Wei Hung, Bing-Shu Wu +2
National Yang Ming Chiao Tung University, Hsinchu, Taiwan · Massachusetts Institute of Technology