Implicit Q-learning-bootstrapped ant colony optimization for maritime moving-target observation scheduling with agile satellites

He Wang; Junyu Wu; Yeye Liu; Yifan Zhou; Jie Zhang; Hui Li; Yanjie Song; Liang Li

cs.AIAug 25, 2026

Implicit Q-learning-bootstrapped ant colony optimization for maritime moving-target observation scheduling with agile satellites

Authors: He Wang, Junyu Wu, Yeye Liu, Yifan Zhou, Jie Zhang, Hui Li, Yanjie Song, Liang Li

Organizations: College of Intelligent Science and Engineering, Harbin Engineering University, Harbin 150001, China · School of Electrical and Control Engineering, North University of China, Taiyuan 030051, China · School of Information Science and Technology, Dalian Maritime University, Dalian 116026, China

Abstract

Maritime moving-target observation scheduling with agile Earth observation satellites is a dynamic, sequence-dependent combinatorial optimization problem. Sea-surface targets move continuously, causing feasible observation windows to vary with target motion and satellite orbital geometry. The scheduler must jointly determine task selection, satellite assignment, observation-window selection, and observation ordering under time-window, attitude-maneuvering, and onboard-resource constraints. This paper proposes an implicit Q-learning-bootstrapped ant colony optimization method, termed IQACO, for multi-satellite maritime moving-target observation scheduling. Rather than directly learning a task-selection policy, IQACO embeds an offline implicit Q-learning module into constructive ant colony optimization to adaptively adjust the pheromone factor, heuristic factor, and evaporation rate. A compact search-state representation captures pheromone distribution, current and historical-best solution quality, and iteration progress. During online scheduling, ant colony optimization constructs feasible observation sequences, while the learned policy adjusts the search behavior according to the current search state. Experiments on 14 scenarios with different scales and satellite configurations show that IQACO consistently outperforms the compared algorithms, improving the mean objective value over conventional ant colony optimization by 2.86%-9.41%. Further comparative and supplementary experiments demonstrate its effectiveness and robustness across different scheduling conditions and problem settings. These results indicate that offline value learning provides an effective adaptive search-control mechanism for constrained maritime moving-target observation scheduling.

Figures & tables

Figure 1: Schematic of multi-satellite maritime moving-target scheduling, including time-varying observation windows, operational constraints, and the IQL-guided adaptive ACO loop.

Symbol	Description
$\mathcal{T}$ , $\mathcal{S}$	Sets of scheduling tasks and agile satellites
$\mathcal{W}_{i,s}$	Feasible observation-window set for task $i$ and satellite $s$
$i,j$	Task indices
$s$	Satellite index
$k,l$	Observation-window indices
$N_{T}$ , $N_{S}$	Numbers of tasks and satellites

Table I: Notation used in the scheduling model

Panel of Figure 2 — Figure 2: Attitude-maneuvering angular-rate profiles of a rate- and acceleration-limited satellite: (a) trapezoidal profile, used when the slewing angle is large enough to reach the maximum angular rate; (b) triangular profile, used otherwise.

Panel of Figure 3 — Figure 3: Workflow of the proposed IQACO method. The offline stage collects transitions with exploratory parameter adjustments and trains the IQL networks; the online stage uses the trained policy to adjust the pheromone factor, heuristic factor, and evaporation rate for feasible schedule construction.

Panel of Figure 4 — Figure 4: Hierarchical solution encoding. The global schedule is decomposed into chronological satellite-specific sequences of task–satellite–window nodes; the badge of each node marks the selected observation window.

Scenario	Original targets	Satellites
Scene 01	100	3
Scene 02	120	3
Scene 03	100	4
Scene 04	120	4
Scene 05	140	4
Scene 06	160	4

Table II: Configuration of the experimental scenarios

Parameter	Description	Value
$\gamma$	Discount factor	0.99
$\tau_{e}$	Expectile parameter	0.6
$\lambda$	Inverse temperature	5.0
State dimension	Input features	5
Action dimension	Parameter-adjustment action	3
Epochs	Training epochs	200

Table III: IQL training hyperparameters

Figure 5: Convergence comparison on Scenes 02, 05, and 08. Each curve is the mean best-so-far weighted objective value

F

over 20 independent runs, and the shaded band denotes

\pm 1

standard deviation.

Figure 6: Convergence comparison on Scenes 10, 12, and 14. Each curve is the mean best-so-far weighted objective value

F

over 20 independent runs, and the shaded band denotes

\pm 1

standard deviation.

Figure 7: Distribution of the final weighted objective value

F

on Scenes 02, 05, and 08. Each box summarizes 20 independent runs; the box spans the interquartile range, the horizontal line is the median, the triangle is the mean, the whiskers extend to the most extreme values within 1.5 times the interquartile range, and open circles mark outliers.

Figure 8: Distribution of the final weighted objective value

F

on Scenes 10, 12, and 14. Each box summarizes 20 independent runs; the box spans the interquartile range, the horizontal line is the median, the triangle is the mean, the whiskers extend to the most extreme values within 1.5 times the interquartile range, and open circles mark outliers.

Scene	GA	PSO	WOA	ACO	RLGA	IQACO	Gain
01	$0.5857\pm 0.0128$	$0.6003\pm 0.0044$	$0.5867\pm 0.0154$	$0.6348\pm 0.0100$	$0.6083\pm 0.0028$	$\mathbf{0.6564\pm 0.0167}^{\ddagger}$	$3.41\%$
02	$0.5647\pm 0.0109$	$0.5832\pm 0.0044$	$0.5737\pm 0.0134$	$0.6216\pm 0.0084$	$0.5944\pm 0.0046$	$\mathbf{0.6434\pm 0.0110}^{\ddagger}$	$3.51\%$
03	$0.6155\pm 0.0122$	$0.6159\pm 0.0033$	$0.6109\pm 0.0108$	$0.6481\pm 0.0105$	$0.6229\pm 0.0038$	$\mathbf{0.6666\pm 0.0149}^{\ddagger}$	$2.86\%$
04	$0.5900\pm 0.0110$	$0.6028\pm 0.0034$	$0.5884\pm 0.0145$	$0.6293\pm 0.0127$	$0.6102\pm 0.0044$	$\mathbf{0.6518\pm 0.0152}^{\ddagger}$	$3.57\%$
05	$0.5798\pm 0.0120$	$0.5954\pm 0.0040$	$0.5866\pm 0.0172$	$0.6176\pm 0.0119$	$0.6078\pm 0.0029$	$\mathbf{0.6494\pm 0.0111}^{\ddagger}$	$5.14\%$
06	$0.5588\pm 0.0104$	$0.5774\pm 0.0061$	$0.5565\pm 0.0118$	$0.5985\pm 0.0121$	$0.5949\pm 0.0041$	$\mathbf{0.6343\pm 0.0130}^{\ddagger}$	$5.99\%$

Table IV: Statistical Summary of the Final Weighted Objective Value

F

Over 20 Independent Runs

Panel of Figure 9 — Figure 9: Weight sensitivity on Scenes 01, 04, 07, and 11 under the five asymmetric weight configurations W1–W5. Each point denotes the mean weighted objective value $F$ over 10 independent runs, and the error bar represents $\pm 1$ standard deviation. The weights are ordered as $(\eta_{1},\eta_{2},\eta_{3})$ .

Figure 10: Training loss terms of the IQL module over 200 epochs: (a) total loss; (b) Q loss; (c) value loss.

Figure 11: Policy and advantage indicators of the IQL module over 200 epochs: (a) policy loss; (b) action MSE; (c) mean advantage

A(\mathbf{s},\mathbf{a})=\hat{Q}(\mathbf{s},\mathbf{a})-V(\mathbf{s})

Figure 12: Value estimation and weight statistics of the IQL module over 200 epochs: (a) mean

Q

value; (b) mean

V

value; (c) mean advantage-weighted regression weight

\exp(A/\lambda)

Explore similar work

Apr 28, 2026cs.NI

EOS-Bench: A Comprehensive Benchmark for Earth Observation Satellite Scheduling

Earth observation satellite imaging scheduling is a challenging NP-hard combinatorial optimisation problem central to space mission operations. While next-generation agile Earth observation satellites (EOS) increase operational flexibility, they also significantly raise scheduling complexity. The lack of a unified, open-source benchmark makes it difficult to compare algorithms across studies. This paper introduces EOS-Bench, a comprehensive framework for systematic and reproducible evaluation of scheduling methods. By integrating high-fidelity orbital dynamics and platform constraints, EOS-Bench generates 1,390 scenarios and 13,900 benchmark instances, spanning from small-scale validation cases to large coordination problems with up to 1,000 satellites and 10,000 requests. We further propose a scenario characterisation scheme to quantify structural difficulty based on factors such as opportunity density, task flexibility, conflict intensity, and satellite congestion. A multidimensional evaluation protocol is introduced, assessing performance across five metrics: task profit, completion rate, workload balance, timeliness, and runtime. The framework is evaluated using mixed-integer programming, heuristics, meta-heuristics, and deep reinforcement learning across both agile and non-agile settings. Results show that EOS-Bench effectively distinguishes solver performance across scales and conditions, revealing trade-offs between solution quality and computational efficiency, and providing deeper insight into scenario complexity. EOS-Bench offers a unified and extensible open testbed for advancing research in Earth observation satellite scheduling. The code and data are available at https://github.com/Ethan19YQ/EOS-Bench.

Qian Yin, Jiaxing Li, Jiaqi Cheng +23

School of Traﬃc and Transportation Engineering, Central South University, Changsha 410083, China · School of Engineering and Materials Science, Queen Mary University of London, London E1 4NS, UK · College of Automation, Central South University, Changsha, 410083, China +20

Mixed-Integer Programming

Ask PDF

Jul 28, 2026cs.AI

Distributed Constraint Optimization via Online Learning and Iterative Pricing with Application to Large-Scale Satellite Scheduling

Distributed constraint optimization problems (DCOPs) provide a popular framework for distributed decision making under limited communication, but many real-world instances are too large to solve monolithically. We address this challenge from two complementary directions. We revisit the connection between DCOPs and potential games, and adapt modern online learning algorithms for equilibrium finding to DCOPs. We show that these algorithms are competitive with representative incomplete DCOP algorithms. We then turn to decomposition frameworks for large-scale DCOPs, motivated by large-scale decentralized satellite scheduling. We propose a new framework that separates a DCOP into two interacting subproblems: a high-level meta-DCOP for task allocation, and independent local optimization problems for scheduling. To couple the two levels, we develop a novel iterative pricing method that updates the meta-level utilities using feedback from the local optimizers. Combining our online learning methods with our iterative pricing framework, we obtain near-optimal performance on real-world decentralized satellite scheduling problem instances, fulfilling over 99% of observation requests compared with 87% for state-of-the-art baselines.

Itai Zilberstein, Pranav Rajbhandari, Steve Chien +1

Department of Computer Science, Carnegie Mellon University · Jet Propulsion Laboratory, California Institute of Technology · Additional affiliations: Strategy Robot, Inc., Strategic Machine, Inc., Optimized Markets, Inc.

Decentralized Optimization Dynamic Pricing

Ask PDF

May 4, 2026cs.IT

Dueling DDQN-Based Adaptive Multi-Objective Handover Optimization for LEO Satellite Networks

In this paper, we propose a dueling double deep Q-network (DDQN)-based adaptive multi-objective handover framework for low Earth orbit (LEO) satellite networks. The proposed method enables dynamic trade-off learning among throughput, blocking probability, and switching cost under time-varying network conditions. Simulation results demonstrate that the proposed approach consistently outperforms conventional baselines, achieving up to 10.3% throughput improvement and near-zero blocking under typical operating conditions.

Po-Heng Chou, Chiapin Wang, Chung-Chi Huang +1

Research Center for Information Technology Innovation (CITI), Academia Sinica (AS), Taipei 11529, Taiwan · Department of Electrical Engineering, National Taiwan Normal University (NTNU), Taipei 106308 Taiwan

Low Earth Orbit Satellite Networks Deep Q-Networks Handoff Optimal Learner

Ask PDF