Learned Terminal Costs for Short-Horizon Model Predictive Control of Autonomous Trucks in Interactive Highway Traffic

Hooman Hamze; Benjamin Houshmand; Deepthi Pathare; Erik Börve; Morteza Haghir Chehreghani; Nikolce Murgovski

cs.LGOct 29, 2026

Learned Terminal Costs for Short-Horizon Model Predictive Control of Autonomous Trucks in Interactive Highway Traffic

Authors: Hooman Hamze, Benjamin Houshmand, Deepthi Pathare, Erik Börve, Morteza Haghir Chehreghani, Nikolce Murgovski

Organizations: Chalmers University of Technology · University of Gothenburg · Trucks Technology and Industrial Division, Volvo Group

Abstract

Autonomous highway driving for heavy-duty trucks is a challenging control problem. The greater mass, dimensions and intricate dynamics increase the requirements for cooperation from surrounding traffic in critical maneuvers, requiring a careful balance between safety constraints and efficient long-horizon decision making. Model Predictive Control (MPC) provides rigorous constraint satisfaction but relies on terminal cost which can limit its effectiveness at short, computationally affordable horizons. Reinforcement Learning (RL) captures long-term driving behavior but offers no constraint guarantees and tends toward overly conservative policies when trained under sparse collision penalties. In this paper, we propose a hybrid MPC and RL framework for highway driving in autonomous trucks, where a finite-horizon MPC, solved via dynamic programming, uses a pre-trained RL model for terminal cost estimation. We evaluate the framework in a simulation environment with reactive surrounding traffic and compare it against pure MPC with zero terminal cost, rollout with terminal cost estimation using a base policy, and pure RL. Results show that the hybrid approach achieves zero collisions with the lowest cost of any configuration.

Figures & tables

Figure 1. Architecture of controller The architecture of the proposed controller consists of three main operational tiers: the environment simulator, the state-prediction pipeline, and a two-stage hybrid decision-making block comprising an MPC optimizer and an RL approximator. At the top layer, the ego vehicle interacts with surrounding traffic within a SUMO (Simulation of Urban MObility) environment. The current simulator state yields data on the ego and surrounding vehicles, which branches into two parallel paths. The left pipeline focuses on future-horizon context, estimating the trajectory prediction of surrounding vehicles. Concurrently, the primary control block feeds the immediate traffic state into the MPC optimizer layer, where a discretized state-action grid is constructed. The predictive trajectory data and the current state-action grids converge at the bottom layer inside the RL approximator block. Here, the terminal states of both the ego and surrounding vehicles are evaluated. An RL-based network approximates the terminal cost for these configurations, outputting the terminal costs of all terminal states back to the MPC block. Finally, the MPC optimizer uses these values to execute a backward sweeping dynamic programming (DP) routine. This optimization loop completes by generating the sequence of optimal actions, passing the primary action back to update the SUMO environment.

Figure 2. Holdout evaluation reward across five training runs, shown as mean $\pm$ one standard deviation. Line chart of holdout evaluation reward against training steps in millions, from 0 to 20 million. A solid dark line shows the mean across five independently trained DQN agents, and a shaded band around it spans one standard deviation either side. All runs rise steeply from a large negative reward during the first two to three million steps, then fluctuate around a common level for the remainder of training.

			Performance Results						Detailed Cost Decomposition
Controller	$N$	$N_{r}$	Coll. %	Cost/step	$\sigma_{\mathrm{all}}$	$\sigma_{0}$	Mean Speed	Time (ms)	$\ell_{\mathrm{speed}}$	$\ell_{\mathrm{accel}}$	$\ell_{\mathrm{safety}}$	$\ell_{\mathrm{col}}$	$\ell_{\mathrm{LC}}$	$\ell_{\mathrm{CD}}$	$\ell_{\mathrm{LL}}$
Pure MPC	1	–	86%	234,519	233,924	17,447	25.0	2.5	0	0	130,826	103,692	0.0286	0.0067	0.9703
Pure MPC	2	–	0%	1,769	3,076	3,076	22.2	4.3	8.00	0.36	1,759	0	0.0149	0.0038	1.1031
Pure MPC	3	–	0%	830	2,022	2,022	22.6	6.3	5.82	0.51	823	0	0.0147	0.0030	1.0109
Pure MPC	4	–	0%	1,259	2,625	2,625	22.6	11.4	5.69	0.58	1,252	0	0.0136	0.0030	0.9717
Pure MPC	5	–	0%	754	2,019	2,019	22.6	17.5	5.68	0.58	747	0	0.0133	0.0030	0.9629

Table 1. Stochastic controller performance evaluation and detailed cost decomposition.

Controller	N	Det. Coll.	S. Coll.	Det. Cost	S. Cost
Pure MPC	4	1%	0%	1,590	1,259
Pure MPC	5	0%	0%	448	754
Rollout $N_{r}\!=\!10$	2	1%	0%	2,078	1,072
Rollout $N_{r}\!=\!10$	5	3%	1%	5,072	2,499
Hybrid	2	0%	0%	257	585
Hybrid	3	1%	0%	1,015	329

Table 2. Deterministic vs. stochastic comparison for representative configurations.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Controller	N	$N_{r}$	Coll. %	Cost/step	Mean Speed	Time/step (ms)
Pure MPC	1	–	88%	241,871	25.0	2.5
Pure MPC	2	–	0%	1,084	22.1	4.3
Pure MPC	3	–	0%	647	22.6	6.3
Pure MPC	4	–	1%	1,590	22.6	11.4
Pure MPC	5	–	0%	448	22.7	17.5
Rollout	1	10	34%	53,562	22.5	7.8

Appendix

Table 3. Deterministic controller results.

Controller	N	$N_{r}$	$\ell_{\mathrm{speed}}$	$\ell_{\mathrm{accel}}$	$\ell_{\mathrm{safety}}$	$\ell_{\mathrm{col}}$	$\ell_{\mathrm{LC}}$	$\ell_{\mathrm{LL}}$
Pure MPC	1	–	0.0	0.0	131,885	109,984	0.027	1.00
Pure MPC	2	–	8.2	0.1	1,075	0	0.013	0.98
Pure MPC	3	–	5.8	0.2	640	0	0.013	1.00
Pure MPC	4	–	5.8	0.2	916	667	0.013	0.96
Pure MPC	5	–	5.3	0.2	442	0	0.013	1.01
Rollout	1	10	6.4	0.2	2,500	51,052	0.023	0.49

Appendix

Table 4. Cost decomposition for each controller type (deterministic).

	Reactive		Const. acc.
Direction	$\mu$	$\sigma$	$\mu$	$\sigma$
Front	0.031	0.222	-0.002	0.089
Front-left	0.079	0.295	0	0.112
Front-right	0.059	0.262	0.001	0.096
Back-left	0.243	0.436	0.005	0.133
Back-right	0.167	0.393	0	0.131

Appendix

Table 5. Mean (

\mu

) and standard deviation (

\sigma

) of position prediction error (m) for

i=5

Run	Checkpoint	Mean reward	$v_{\text{avg}}$ (m/s)
1	16.1M	$-31.46\pm 42.44$	20.8
2	12.6M	$-16.90\pm 28.45$	20.1
3	6.8M	$-13.72\pm 22.03$	20.5
4	13.6M	$-8.60\pm 20.39$	20.8
5	17.5M	$-14.30\pm 23.85$	20.6
Across runs	–	$-17.00\pm 8.63$	$20.6$

Appendix

Table 6. Selected model of each training run for the final hyperparameter setting, evaluated on 500 seeds.

Parameter	Value
MPC / DP solver
Prediction horizon $N$	$1$ – $5$
Rollout horizon $N_{r}$	$10$ , $20$
Discount factor $\gamma$	$0.98$
Velocity grid step $V_{\mathrm{step}}$	$0.2\,\text{m/s}$
Acceleration levels	$5$

Appendix

Table 7. MPC/DP solver, DQN network, and training parameters. Bold denotes selected value; parentheses list other values tested.

Parameter	Value
Simulation
Time step $\Delta t$	$0.2\,\text{s}$
Episode length	$300\,\text{s}$
Lanes $N_{\mathrm{lanes}}$	$3$
Traffic density	$30\,\text{veh/km}$
Surrounding speed range	$[20,\,30]\,\text{m/s}$

Appendix

Table 8. Simulation, vehicle, and MPC cost parameters.

Autonomous Driving

Autonomous driving develops vehicles that perceive road scenes, predict interactions, and plan and control safe movement with limited or no human input.

See all 538 papers

Beyond Conservative Automated Driving in Multi-Agent Scenarios via Coupled Model Predictive Control and Deep Reinforcement Learning
Apr 15, 2026Saeed Rahmani, Gözde Körpe, Zhenlin +4Autonomous Driving Planning Multi-Agent Control
Gavin · TU Delft, Faculty of Civil Engineering and Geosciences, Department of Transport and Planning, Delft, The Netherlands · NVIDIA, San Jose, CA, USA
Sharing the Control Authority Between Deep Reinforcement Learning and Model Predictive Control: Application to Multi-Class Transportation Networks
Aug 21, 2026Giray Onur, Azita Dabiri, Bart De SchutterIntelligent Transportation Systems RL Control
Delft Center for Systems and Control, Delft University of Technology, Delft, 2628CD, The Netherlands
Coordinated Lane-Level Variable Speed Limits and Ramp Metering for Successive Weaving Segments Considering Merging/Diverging Risks: A Hybrid Model Predictive Control and Multi-Agent Reinforcement Learning Approach
Sep 28, 2026Guodong Ma, Baofeng Sun, Wenyu Yang +1Intelligent Transportation Systems Model Predictive Control
School of Transportation, Jilin University, Changchun 130022, China · School of Transportation and Logistics, Southwest Jiaotong University, Chengdu 610031, China

Apr 15, 2026J/K move · Enter open · S save

Beyond Conservative Automated Driving in Multi-Agent Scenarios via Coupled Model Predictive Control and Deep Reinforcement Learning

Saeed Rahmani, Gözde Körpe, Zhenlin, Xu+3

Gavin · TU Delft, Faculty of Civil Engineering and Geosciences, Department of Transport and Planning, Delft, The Netherlands · NVIDIA, San Jose, CA, USA

Autonomous Driving Planning Multi-Agent Control RL for Autonomous Driving Model Predictive Control

Open paper AskPDF arXiv

Automated driving at unsignalized intersections is challenging due to complex multi-vehicle interactions and the need to balance safety and efficiency. Model Predictive Control (MPC) offers structured constraint handling through optimization but relies on hand-crafted rules that often produce overly conservative behavior. Deep Reinforcement Learning (RL) learns adaptive behaviors from experience but often struggles with safety assurance and generalization to unseen environments. In this study, we present an integrated MPC-RL framework to improve navigation performance in multi-agent scenarios. Experiments show that MPC-RL outperforms standalone MPC and end-to-end RL across three traffic-density levels. Collectively, MPC-RL reduces the collision rate by 21% and improves the success rate by 6.5% compared to pure MPC. We further evaluate zero-shot transfer to a highway merging scenario without retraining. Both MPC-based methods transfer substantially better than end-to-end PPO, which highlights the role of the MPC backbone in cross-scenario robustness. The framework also shows faster loss stabilization than end-to-end RL during training, which indicates a reduced learning burden. These results suggest that the integrated approach can improve the balance between safety performance and efficiency in multi-agent intersection scenarios, while the MPC component provides a strong foundation for generalization across driving environments. The implementation code is available open-source.

Abstract

Autonomous Driving

Explore similar work

Beyond Conservative Automated Driving in Multi-Agent Scenarios via Coupled Model Predictive Control and Deep Reinforcement Learning

Sharing the Control Authority Between Deep Reinforcement Learning and Model Predictive Control: Application to Multi-Class Transportation Networks

Coordinated Lane-Level Variable Speed Limits and Ramp Metering for Successive Weaving Segments Considering Merging/Diverging Risks: A Hybrid Model Predictive Control and Multi-Agent Reinforcement Learning Approach