Preference-Agile Multi-Objective Optimization for Real-time Vehicle Dispatching
Authors: Jiahuan Jin, Wenhao Zhao, Rong Qu, Jianfeng Ren, Xinan Chen, Qingfu Zhang, Ruibin Bai
Organizations: School of Computer Science, University of Nottingham Ningbo China, Ningbo, China. · School of Computer Science, University of Nottingham, Nottingham, UK. · Department of Computer Science, City University of Hong Kong, Hong Kong
Multi-objective optimization (MOO) has been widely studied in literature because of its versatility in human-centered decision making in real-life applications. Recently, demand for dynamic MOO is fast-emerging due to tough market dynamics that require real-time re-adjustments of priorities for different objectives. However, most existing studies focus either on deterministic MOO problems which are not practical, or non-sequential dynamic MOO decision problems that cannot deal with some real-life complexities. To address these challenges, a preference-agile multi-objective optimization (PAMOO) is proposed in this paper to permit users to dynamically adjust and interactively assign the preferences on the fly. To achieve this, a novel uniform model within a deep reinforcement learning (DRL) framework is proposed that can take as inputs users' dynamic preference vectors explicitly. Additionally, a calibration function is fitted to ensure high quality alignment between the preference vector inputs and the output DRL decision policy. Extensive experiments on challenging real-life vehicle dispatching problems at a container terminal showed that PAMOO obtains superior performance and generalization ability when compared with two most popular MOO methods. Our method presents the first dynamic MOO method for challenging \rev{dynamic sequential MOO decision problems
Figures & tables
Figure 1: A simple scenario example to illustrate the proposed PAMOO algorithm for online truck dispatching in a container terminal. At time t , one idle truck needs to be dispatched for a new task (dedicated to different QCs). Among three choices QC1, QC2 and QC3 with incremental queue lengths and decremental empty travel distances (indicated by three red lines on the left side of figure). Dispatching decisions are made by jointly considering a dynamic preference (based on expert experience on macro-level situational judgments) and fine-grained data reflecting real-time state features.
Figure 2: A route example for a single task. An idle truck receives the dispatching task at yard A. The first and second operation nodes are QC1 and crane at yard B, respectively. The truck route in red represents empty mileage and the route in blue is the loaded travel distance. The objectives are to minimize both aggregated idle time of all QCs and total empty mileages by all trucks.
Figure 3: An illustrative example where truck with task wiq arrives too late at q -th QC, causing a QC idle duration (a) and arrives at QC before the prior task’s completion, resulting in truck queuing (b).
Figure 4: An illustration of learning based interactive adjustments of user preferences in PAMOO. It employs a multi-policy inner loop RL with policies π∗(a∣s,θ) . Once trained, PAMOO makes decisions for any combinations of state s and preference vector p in a single run.
Index
Description
1
The remain task number of each QC
2
The total working truck number of each QC
3
The distance for the truck to complete each tasks
4
The necessary distance before the truck arrive at
each QC
5
The queue lengths at each QC
Table 1: Items of agent’s observation at each time step.
Figure 5: The network structure of the proposed PAMOO for online truck dispatching.
Figure 6: Interpretation of the preference calibration method.
Parameters
Values
batch size B
32
clipping rate ϵ
0.2
discount rate γ
1
epochs per iteration M
10
number of iteration K
5000
episodes per iteration N
10
Table 2: Hyper-parameters used in PAMOO.
Figure 7: The relationship between the hypervolume indicator and the reference policy.
Method
80 Trucks
100 Trucks
120 Trucks
HV
Sparsity
Gap(%)
HV
Sparsity
Gap(%)
HV
Sparsity
Gap(%)
NSGA-II (500)
0.669
91.86
25.0%
0.674
31.68
23.5%
0.708
21.73
20.45%
NSGA-II (1000)
0.729
19.44
18.27%
0.728
20.1
17.37%
0.759
16.1
14.72%
NSGA-II (2000)
0.807
4.01
9.53%
0.784
23.8
11.01%
0.798
9.31
10.34%
MOEA/D (500)
0.682
31.94
23.54%
0.675
75.3
23.38%
0.693
19.75
22.13%
MOEA/D (1000)
0.72
35.1
19.28%
0.746
56.85
15.32%
0.751
53.66
15.62%
Table 3: Comparison Result to NSGA-II and MOEA/D.
Figure 8: Approximate Pareto front obtained by our method and benchmarks on instances of different number of trucks.
Figure 9: Pareto frontiers generated by proposed method that trained by a small number of preferences.
Figure 10: Pareto frontiers generated by PAMOO and outer loop method on instance of 120 trucks.
Preferences
PAMOO
Outer Loop Method
QC Idle Time (min/QC)
Empty Travel Distance (m/task)
QC Idle Time (min/QC)
Empty Travel Distance (m/task)
[1.0, 0.0]
13.04
799.11
12.16
876.9
[0.9, 0.1]
13.21
759.0
12.49
842.33
[0.8, 0.2]
13.26
703.38
12.36
813.71
[0.7, 0.3]
13.69
657.03
12.39
755.92
[0.6, 0.4]
14.6
612.71
12.68
722.04
Table 4: Numerical Result of Comparison to Outer Loop Method.
Figure 11: Sample efficiency of inner-loop (ours) and outer-loop methods compared with NSGA-II and MOEA-D.
Figure 12: Ablation study results on adopting various RL algorithms and network architectures.
Method
80 Trucks
100 Trucks
120 Trucks
HV
Sparsity
Gap(%)
HV
Sparsity
Gap(%)
HV
Sparsity
Gap(%)
Evenly Preferences (11 Pref.)
0.684
253.03
5.79%
0.676
260.71
7.02%
0.679
267.98
7.24%
Evenly Preferences (21 Pref.)
0.709
64.48
2.34%
0.705
66.14
3.03%
0.709
68.91
3.14%
Evenly Preferences (51 Pref.)
0.723
11.42
0.41%
0.721
11.8
0.83%
0.724
11.06
1.09%
Adjusted Preferences (11 Pref.)
0.694
240.93
4.41%
0.683
240.13
6.05%
0.685
241.41
6.42%
Adjusted Preferences (21 Pref.)
0.716
58.29
1.38%
0.718
60.55
1.24%
0.719
60.9
1.78%
Table 5: Comparison Result between Even and Adjusted Preferences.
Dynamic multiobjective optimization problems (DMOPs) feature time-varying objectives, which cause the Pareto optimal solution (POS) set to drift over time and make it difficult to maintain both convergence and diversity under limited response time. Many existing prediction-based dynamic multiobjective evolutionary algorithms (DMOEAs) either depend on learned models with nontrivial training cost or employ one-step population mapping, which may overlook the gradual nature of POS evolution. This paper proposes DD-DMOEA, a training-free diffusion-based dynamic response mechanism for DMOPs. The key idea is to treat the POS obtained in the previous environment as a "noisy" sample set and to guide its evolution toward the current POS through an analytically constructed multi-step denoising process. A knee-point-based auxiliary strategy is used to specify the target region in the new environment, and an explicit probability-density formulation is derived to compute the denoising update without neural training. To reduce the risk of misleading guidance caused by knee-point prediction errors, an uncertainty-aware scheme adaptively adjusts the guidance strength according to the historical prediction deviation. Experiments on the CEC2018 dynamic multiobjective benchmarks show that DD-DMOEA achieves competitive or better convergence-diversity performance and provides faster dynamic response than several state-of-the-art DMOEAs.
Jian Guan, Huolong Wu, Zhenzhong Wang +2
Institute of Artificial Intelligence, Xiamen University, Xiamen, 361005, China · Department of Artificial Intelligence, School of Informatics, Xiamen University; the Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education; and the Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan, Ministry of Culture and Tourism, Xiamen, 361005, China · Department of Artificial Intelligence, Sichuan University, Chengdu, 610065, China
Many decision-making problems in computing and networking systems can be naturally formulated as cost-minimization problems under performance constraints. In dynamic environments, reinforcement learning (RL) is often used to solve such problems at runtime by embedding both costs and constraint violations into a single scalar reward through weighted penalty terms, following a Lagrangian-inspired formulation. However, in this context the behavior of the learned policy critically depends on the choice of these weights, which are typically selected manually. This makes it difficult to identify an appropriate trade-off between optimizing the primary objective and effectively avoiding constraint violations, particularly in non-stationary environments where their relative importance may change. This paper presents MAMO (Multi-Agent system for Multi-Objective constrained optimization), an approach to tackle this balancing problem through multi-agent RL. MAMO decouples task execution from objective design by formulating the selection of reward weights as a learning problem, providing a !rst step towards more autonomous and robust RL-based solutions for constrained optimization problems in dynamic environments.
The Vehicle Routing Problem (VRP) and its variants represent some of the most practically consequential optimization challenges in modern logistics and urban mobility. In this study, we address a dynamic, online variant combining elements of the VRP and the Orienteering Problem (OP), in which a fleet of vehicles must maximise cumulative reward collected within a fixed time horizon while continuously replanning as new tasks arrive. We propose and evaluate a reward-density heuristic for dynamic multi-vehicle assignment, referred to as the Efficiency heuristic. We evaluate this formulation across two application domains: autonomous drone task allocation and urban taxi dispatch, across multiple fleet sizes and task scales. The proposed method is compared with four classical construction heuristics and three metaheuristic algorithms (Adaptive Large Neighbourhood Search, Genetic Algorithm, and Simulated Annealing), all evaluated under identical conditions. Across all tested configurations, the Efficiency heuristic matches the solution quality of the best metaheuristic algorithms while requiring two to three orders of magnitude less planning time, establishing Pareto dominance over all competing methods on the reward-versus-compute frontier. These findings suggest a practical design principle for real-time allocation and dispatch systems: in dynamic, time-constrained routing environments, carefully designed greedy heuristics can match the output of sophisticated search procedures at a fraction of the computational cost, making them preferable for online deployment.
Manish Kolachalam, Rani Malhotra
Autonomous Machines Applied Research Center Infosys Center for Imaging Technologies, Infosys Bengaluru, India