Multi-objective reinforcement learning is a fundamental problem for modern AI, with applications spanning post-training, reasoning models, and robotic control. A central challenge is to learn policies that balance competing objectives, such as helpfulness versus honesty in safety and alignment, or speed versus energy in robotics. The optimal policy can vary sharply with objective-weighting combinations, and the extent of this variation depends crucially on the degree of conflict among objectives, making per-preference policy learning sample-inefficient. In this paper, we introduce objective-conflict coefficients and derive near-optimal regret bounds for online settings using this measure, thereby quantifying the impact of misalignment on learning efficiency. Then, we design routed ensemble policy learning, an algorithm that maintains a small set of basis policies and dynamically routes trajectories to the policy with the highest potential reward. Finally, we consider trajectory selection as another approach to mitigate objective conflict, and design a kernel surrogate trajectory attribution algorithm that estimates how feedback signals from a set of trajectories influence policy learning on other trajectories. The kernel can capture nonlinear interactions such as synergy and antagonism between trajectories. We evaluate our approach across diverse environments and benchmarks: routed ensembles improve average performance by 4.3% over baselines for online environments from Meta-World and MO-Gymnasium, and kernel surrogate attribution improves correlation with leave-one-out analysis over existing baselines by 50%. When applied to trajectory selection, kernel-based attribution yields a 27% improvement in reward over baseline attribution methods.
Figures & tables
Figure 1: We visualize the objective space returns (r1,r2) for policies optimized under varying preference weights w∈[0,1] in three canonical regimes — clustered, linear and concave Pareto frontier. Light yellow indicates that w is closer to 1 , while dark blue indicates w is closer to 0 .
Figure 2: We compare the cumulative regret of routed ensembles (red) against MO-UCBVI (green dashed), uniform ensembles (yellow), and the best stationary policy (blue dotted).
Metrics
Soft Modularization
PaCo
CARE
Uniform Assignment
Algorithm 1
Meta-World
Success rate
82.0±1.1%
73.1±1.1%
80.2±1.8%
81.3±1.4%
85.6±1.8%
Metrics
PGMORL
Q-Pensieve
CAPQL
Uniform Assignment
Algorithm 1
MO-HalfCheetah2d
HV ( ×106 )
0.11±0.08
4.85±0.06
3.29±0.18
3.17±0.11
5.17±0.16
UT ( ×102 )
0.12±0.13
2.14±0.13
0.63±0.09
1.01±0.12
2.47±0.14
MO-Hopper
HV ( ×107 )
1.23±0.04
1.61±0.32
1.39±0.13
1.56±0.16
1.83±0.14
UT ( ×102 )
0.04±0.02
1.47±0.14
1.29±0.12
1.17±0.09
1.56±0.06
Table 1: Comparison of our approach with several baseline methods. We report the average success rate on Meta-World, HV and UT on MuJoCo, as well as HV and Cardinality on Grid-World.
Synthetic
Qwen3-1.7B
Llama-3.2-1B
Method
HelpSteer2
UltraFeedback
HelpSteer2
UltraFeedback
Single Policy
0.48
0.51
0.46
0.34
0.52
Uniform Assignment
0.49
0.47
0.47
0.35
0.54
Algorithm 1
0.56
0.54
0.48
0.35
0.55
Table 2: Comparison results for RLHF settings between our approach, using a single policy, and uniform assignment to the same number of basis policies.
Table 5
Methods
Multitask Minigrid
Meta-World MT10
Synergy MTRL
LDS
Reward
LDS
Reward
LDS
Reward
Influence functions
−0.23±0.08
0.07±0.13
0.71±0.11
16.3±1.4
0.04±0.16
1.77±0.67
TracIn
0.31±0.06
0.25±0.21
0.45±0.15
16.0±2.3
0.16±0.04
2.65±0.03
Linear surrogates
0.37±0.12
0.33±0.19
0.42±0.19
15.8±4.5
0.16±0.04
2.67±0.01
Algorithm 2
0.41±0.04
0.45±0.17
0.80±0.04
18.8±5.6
0.91±0.01
5.85±0.01
Table 3: We report the average target reward when the target policy is trained with source tasks selected by each attribution algorithm.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Distance
MT10
CartPole
Highway
Lunarlander
0.1%
0.01±0.01%
0.12±0.14%
0.02±0.02%
0.06±0.01%
0.5%
0.43±0.73%
0.73±0.10%
0.11±0.09%
0.03±0.02%
1.0%
0.32±0.56%
0.98±0.65%
2.04±0.58%
0.48±0.01%
Appendix
Table 4: We empirically find that the approximation error of ϵ is negligible for several interactive RL environments. We report the mean and standard deviation of the approximation error across 10 random subsets. We defer a detailed description of the setup to the experiments section.
Figure 5: Correlation analysis of attribution scores against LOO scores (left) and objective conflict coefficient Γ (right).
Parameter
Meta-World
MO-Gymnasium
Number of basis policies
3
3
Base optimizer
Adam
Adam
SAC buffer size
106
106
Learning rate (Actor)
3×10−4
3×10−4
Learning rate (Critic)
3×10−4
3×10−4
Discount factor
0.99
0.99
Appendix
Table 5: Hyper-parameters for the routed ensemble procedure used in Algorithm 1 .
Setting
Basis construction
Number of policies
Mean ϵ
Max ϵ
Tabular, aligned
Exact
1
∼10−18
∼10−17
Tabular, linear
Exact
2
∼10−9
∼10−8
Tabular, concave
Exact
3
0.186
0.55
Tabular, concave
Exact
21
1.45×10−4
6.10×10−4
MO-Hopper
Candidate-selected
3
0.443
0.80
MO-Hopper
Candidate-selected
15
0.231
0.51
Appendix
Table 6: Convex-hull occupancy residuals for exact tabular occupancies and empirical benchmark visitation distributions. Mean and maximum residuals are computed over the evaluated preferences.
Deep Sea Treasure
Fruit Tree
Influence functions
0.28±0.51
0.30±0.57
Linear surrogate models
0.62±0.39
0.71±0.53
TracIn
0.70±0.40
0.75±0.45
Algorithm 2
0.75±0.14
0.81±0.18
Appendix
Table 7: Summary of trajectory attribution results on two MO-Gymnasium benchmarks, Deep Sea Treasure and Fruit Tree.
Figure 6: Ablation analysis of the choice of the ensemble size N , measured by hypervolume (HV).
Department of Computer Science, University of Manchester · Centre for Robotics & AI, University of Manchester · School of Computer Science, University of Sheffield +1