Preference-Conditioned Multi-Objective Reinforcement Learning for Tactical Decision Making in Heavy-Duty Vehicles

Deepthi Pathare; Leo Laine; Morteza Haghir Chehreghani

cs.LGJan 26, 2026

Preference-Conditioned Multi-Objective Reinforcement Learning for Tactical Decision Making in Heavy-Duty Vehicles

Authors: Deepthi Pathare, Leo Laine, Morteza Haghir Chehreghani

Organizations: Department of Computer Science and Engineering Chalmers University of Technology and and University of Gothenburg · Department of Mechanics and Maritime Sciences Chalmers University of Technology

Abstract

A central difficulty in applying reinforcement learning to real-world applications such as autonomous driving is that conventional scalar reward formulations obtained by aggregating multiple often competing objectives may obscure the structure of their trade-offs. These challenges are particularly relevant to decision-making for heavy-duty vehicles, which requires balancing safety, efficiency, and operational costs under demanding commercial requirements. We present a preference-conditioned multi-objective reinforcement learning framework that explicitly represents these objectives and utilizes Generalized Policy Improvement and Proximal Policy Optimization to learn policies capturing different trade-offs between them. The proposed method approximates the Pareto front across three conflicting objectives: safety, quantified in terms of collisions and successful completion; energy efficiency, quantified using energy cost; and time efficiency, quantified using driver cost. We evaluate the framework in a scalable highway traffic simulation environment tailored for tactical decision making in heavy-duty trucks. It yields smooth and interpretable Pareto fronts, and evaluations across traffic densities and against an analytical baseline show that the learned policies capture meaningful trade-offs. The preference-conditioned policy enables transitions between different driving behaviors without retraining, demonstrating the potential of multi-objective reinforcement learning for adaptive decision making in autonomous heavy-duty vehicles.

Figures & tables

Figure 1. MOPPO framework interacting with the traffic environment. The framework iteratively trains the actor and critic networks using the preference vector provided by the GPI-based weight selection algorithm described in Section 3.1 . MOPPO framework interacting with the traffic environment. The framework iteratively trains the actor and critic networks using the preference vector provided by the GPI-based weight selection algorithm described in Section \ref{sec:weight_sel}.

Figure 2. Pareto front produced by the GPI-LS MOPPO framework, showing the trade-off between driver cost and energy cost across three traffic settings, along with the success rate. Markers represent mean driver and energy costs over five evaluation episodes, with error bars indicating the standard error. Pareto front produced by the GPI-LS MOPPO framework, showing the trade-off between driver cost and energy cost across three traffic settings, along with the success rate. Markers represent mean driver and energy costs over five evaluation episodes, with error bars indicating the standard error.

Figure 3. Comparison of average speed and cost values obtained for Pareto-optimal policies by GPI-LS MOPPO, with

100\%

success rate, against analytical values. The analytical curves show cost as a function of speed under a constant-speed assumption. Comparison of average speed and cost values (driver cosr, energy cost and total cost) obtained for Pareto-optimal policies by GPI-LS MOPPO, with

100\%

success rate, against analytical values. The analytical curves show cost as a function of speed under a constant-speed assumption.

Figure 4. Comparison of Pareto-frontiers produced by our method (GPI-LS MOPPO), its ablated variant without safety filter and a basline method (GPI-LS with DQN). The faded points in the last two plots correspond to the Pareto policies of our method from the first plot, added for direct comparison. Comparison of Pareto-frontiers produced by our method (GPI-LS MOPPO), its ablated variant without safety filter and a basline method (GPI-LS with DQN). The faded points in last two plots correspond to the policies in the first plot, added for direct comparison.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5. Simulated traffic environment in SUMO with the illustration of moving window. Simulated traffic environment in SUMO with the illustration of moving window.

Parameter	Value
Length of highway segment (l road )	3000 m
Moving window size (l window )	400 m
Heterogeneous vehicle ratio (trucks)	0.2
Heterogeneous vehicle ratio (cars)	0.8
Car max speed distribution	$\mathcal{N}(23,3.8)$ m/s
Truck max speed distribution	$\mathcal{N}(20,0.8)$ m/s

Appendix

Table 1. Traffic simulation parameters.

Figure 6. Overview of the architecture adapted from ( Pathare et al., 2025 ) , extended for multi-objective learning Overview of the architecture adapted from \protect\cite{pathare2025tactical}, extended for multi-objective learning

Parameter	Value
$R_{tar}$	4.41
$P_{c}$	1000
$C_{el}$	0.5 euro per kWh
$C_{dr}$	50 euro per hour
$m$	44000 kg
$C_{d}$	0.6

Appendix

Table 2. Parameter values used for reward computation.

Parameter	Value
Learning rate	$3\times 10^{-4}$
Discount factor ( $\gamma$ )	0.98
Initial epsilon ( $\epsilon_{init}$ )	1
Final epsilon ( $\epsilon_{final}$ )	0.05
Epsilon decay steps	100000
Rollout buffer size	2048

Appendix

Table 3. Hyperparameters used in the GPI-LS MOPPO framework.

Figure 7. Analytical predictions for optimal speed and cost in zero traffic situation. Analytical predictions for optimal speed and cost in zero traffic situation.

Figure 8. Pareto Front showing the trade off between driver cost and energy cost in zero traffic, along with the success rate. The annotated numbers denote the policy numbers referred in the Table 4 Pareto Front showing the trade off between driver cost and energy cost in zero traffic, along with the success rate. The annotated numbers denote the policy numbers referred in the \autoref{tab:policy_results_zero}

Policy	Success	Average	Energy	Driver	Traveled	Total Cost	Total Cost
Number	Rate (%)	Speed (m/s)	Cost (euros)	Cost (euros)	Distance (m)	(euros)	per m (euros)
1	0 $\pm$ 0	1.63 $\pm$ 0.46	0.16 $\pm$ 0.06	2.79 $\pm$ 0.01	341 $\pm$ 100	2.95 $\pm$ 0.08	0.0086 $\pm$ 0.0025
2	0 $\pm$ 0	1.84 $\pm$ 0.65	0.20 $\pm$ 0.10	2.78 $\pm$ 0.00	376 $\pm$ 130	2.98 $\pm$ 0.10	0.0079 $\pm$ 0.0027
3	100 $\pm$ 0	19.03 $\pm$ 0.26	1.63 $\pm$ 0.02	2.20 $\pm$ 0.03	3012 $\pm$ 10	3.83 $\pm$ 0.01	0.0013 $\pm$ 0.0000
4	100 $\pm$ 0	19.59 $\pm$ 0.30	1.65 $\pm$ 0.02	2.12 $\pm$ 0.03	2989 $\pm$ 4	3.77 $\pm$ 0.02	0.0013 $\pm$ 0.0000
5	100 $\pm$ 0	19.73 $\pm$ 0.38	1.66 $\pm$ 0.02	2.11 $\pm$ 0.04	2993 $\pm$ 4	3.77 $\pm$ 0.02	0.0013 $\pm$ 0.0000

Appendix

Table 4. Evaluation of Pareto-optimal policies in zero traffic.

Figure 9. Pareto Front showing the trade off between driver cost and energy cost in medium traffic, along with the success rate. The annotated numbers denote the policy numbers referred in the Table 5 Pareto Front showing the trade off between driver cost and energy cost in medium traffic, along with the success rate. The annotated numbers denote the policy numbers referred in the \autoref{tab:policy_results_medium}

Figure 10. Pareto Front showing the trade off between driver cost and energy cost in high traffic, along with the success rate. The annotated numbers denote the policy numbers referred in the Table 6 Pareto Front showing the trade off between driver cost and energy cost in high traffic, along with the success rate. The annotated numbers denote the policy numbers referred in the \autoref{tab:policy_results_high}

Policy	Success	Average	Energy	Driver	Traveled	Total Cost	Total Cost
Number	Rate (%)	Speed (m/s)	Cost (euros)	Cost (euros)	Distance (m)	(euros)	per m (euros)
1	0 $\pm$ 0	1.41 $\pm$ 0.27	0.12 $\pm$ 0.03	2.78 $\pm$ 0.00	371 $\pm$ 48	2.90 $\pm$ 0.03	0.0078 $\pm$ 0.0010
2	20 $\pm$ 18	5.03 $\pm$ 3.22	0.42 $\pm$ 0.27	2.65 $\pm$ 0.12	990 $\pm$ 477	3.07 $\pm$ 0.16	0.0031 $\pm$ 0.0015
3	20 $\pm$ 18	5.35 $\pm$ 3.26	0.55 $\pm$ 0.36	2.63 $\pm$ 0.13	930 $\pm$ 496	3.18 $\pm$ 0.23	0.0034 $\pm$ 0.0018
4	20 $\pm$ 18	6.40 $\pm$ 3.35	0.57 $\pm$ 0.31	2.61 $\pm$ 0.15	999 $\pm$ 502	3.18 $\pm$ 0.17	0.0032 $\pm$ 0.0016
5	40 $\pm$ 22	8.66 $\pm$ 3.67	0.82 $\pm$ 0.36	2.54 $\pm$ 0.13	1373 $\pm$ 577	3.36 $\pm$ 0.23	0.0024 $\pm$ 0.0010

Appendix

Table 5. Evaluation of Pareto-optimal policies in medium traffic.

Policy	Success	Average	Energy	Driver	Traveled	Total Cost	Total Cost
Number	Rate (%)	Speed (m/s)	Cost (euros)	Cost (euros)	Distance (m)	(euros)	per m (euros)
1	0 $\pm$ 0	1.65 $\pm$ 0.32	0.16 $\pm$ 0.04	2.80 $\pm$ 0.01	252 $\pm$ 89	2.96 $\pm$ 0.05	0.0118 $\pm$ 0.0041
2	0 $\pm$ 0	1.96 $\pm$ 0.58	0.19 $\pm$ 0.07	2.79 $\pm$ 0.01	389 $\pm$ 207	2.98 $\pm$ 0.09	0.0077 $\pm$ 0.0041
3	0 $\pm$ 0	2.13 $\pm$ 0.71	0.23 $\pm$ 0.10	2.78 $\pm$ 0.00	203 $\pm$ 143	3.01 $\pm$ 0.10	0.0148 $\pm$ 0.0105
4	20 $\pm$ 18	4.98 $\pm$ 2.97	0.47 $\pm$ 0.28	2.66 $\pm$ 0.10	906 $\pm$ 512	3.13 $\pm$ 0.17	0.0035 $\pm$ 0.0020
5	20 $\pm$ 18	5.38 $\pm$ 3.26	0.51 $\pm$ 0.32	2.62 $\pm$ 0.14	735 $\pm$ 496	3.13 $\pm$ 0.18	0.0043 $\pm$ 0.0029

Appendix

Table 6. Evaluation of Pareto-optimal policies in high traffic.

Training Steps	Expected Utility		Mean Utility
	Baseline	Ours	Baseline	Ours
300k	-0.1704 $\pm$ 0.0415	0.3588 $\pm$ 0.0216	-85.5685 $\pm$ 31.3080	-1.7035 $\pm$ 1.4485
350k	-0.1455 $\pm$ 0.0332	0.3270 $\pm$ 0.0183	-19.8703 $\pm$ 2.9108	-0.8235 $\pm$ 0.6110
400k	-0.1675 $\pm$ 0.0238	0.1883 $\pm$ 0.1655	-29.0711 $\pm$ 27.2000	-0.8603 $\pm$ 0.1989
450k	-0.1355 $\pm$ 0.0068	0.3311 $\pm$ 0.0255	-25.6834 $\pm$ 21.3722	-0.2739 $\pm$ 0.2631
500k	-0.1593 $\pm$ 0.0353	0.3419 $\pm$ 0.0173	-22.7032 $\pm$ 20.9624	-0.3035 $\pm$ 0.3839

Appendix

Table 7. Evaluation metrics over the training checkpoints for the baseline (GPI-LS with DQN) and our method (GPI-LS MOPPO).

Multi-Objective Reinforcement Learning

Multi-objective reinforcement learning studies how agents can balance competing goals and learn policies for different trade-offs.

See all 144 papers

Balancing Multiple Objectives in Urban Traffic Control with Reinforcement Learning from AI Feedback
Feb 24, 2026Chenyang Zhao, Vinny Cahill, Ivana DusparicMulti-Objective Reinforcement Learning Preference-Based RL
Trinity College Dublin, Dublin, Ireland
Multi-Objective Reinforcement Learning Using Routed Ensembles and Trajectory Attribution
Oct 26, 2026Zhenshuo Zhang, Ruoxuan Xiong, Hongyang R. ZhangMulti-Objective Reinforcement Learning Reinforcement Learning
Northeastern University, Boston, MA · Emory University, Atlanta, GA
PCHC: Enabling Preference Conditioned Humanoid Control via Multi-Objective Reinforcement Learning
Mar 25, 2026Huanyu Li, Dewei Wang, Xinmiao Wang +4Robotic Control Humanoid Robot Control
Harbin Institute of Technology · University of Science and Technology of China · Harbin Engineering University +2

Feb 24, 2026J/K move · Enter open · S save

Balancing Multiple Objectives in Urban Traffic Control with Reinforcement Learning from AI Feedback

Chenyang Zhao, Vinny Cahill, Ivana Dusparic

Trinity College Dublin, Dublin, Ireland

Multi-Objective Reinforcement Learning Preference-Based RL

Open paper AskPDF arXiv

Reward design has been one of the central challenges for real world reinforcement learning (RL) deployment, especially in settings with multiple objectives. Preference-based RL offers an appealing alternative by learning from human preferences over pairs of behavioural outcomes. More recently, RL from AI feedback (RLAIF) has demonstrated that large language models (LLMs) can generate preference labels at scale, mitigating the reliance on human annotators. However, existing RLAIF work typically focuses only on single-objective tasks, leaving the open question of how RLAIF handles systems that involve multiple objectives. In such systems trade-offs among conflicting objectives are difficult to specify, and policies risk collapsing into optimizing for a dominant goal. In this paper, we explore the extension of the RLAIF paradigm to multi-objective self-adaptive systems. We show that multi-objective RLAIF can produce policies that yield balanced trade-offs reflecting different user priorities without laborious reward engineering. We argue that integrating RLAIF into multi-objective RL offers a scalable path toward user-aligned policy learning in domains with inherently conflicting objectives.

Abstract

Multi-Objective Reinforcement Learning

Explore similar work

Balancing Multiple Objectives in Urban Traffic Control with Reinforcement Learning from AI Feedback

Multi-Objective Reinforcement Learning Using Routed Ensembles and Trajectory Attribution

PCHC: Enabling Preference Conditioned Humanoid Control via Multi-Objective Reinforcement Learning