Diffusion-Guided Cooperative Policy Learning for Target Tracking Based on Underwater Mobile Agent Networks

Jiaao Ma; Chuan Lin; Guangjie Han; Shengchao Zhu; Zhenyu Wang; Chen An

cs.NIMar 31, 2026

Diffusion-Guided Cooperative Policy Learning for Target Tracking Based on Underwater Mobile Agent Networks

Authors: Jiaao Ma, Chuan Lin, Guangjie Han, Shengchao Zhu, Zhenyu Wang, Chen An

Organizations: Software College, Northeastern University, Shenyang, China · Key Laboratory of Maritime Intelligent Network Information Technology, Ministry of Education, Hohai University · College of Computer Science and Software Engineering, Hohai University, Nanjing, 210013, China · College of Computer Science and Engineering, Northeastern University, Shenyang, China

Abstract

Multi-agent reinforcement learning (MARL) provides a promising solution for cooperative target tracking in networks of autonomous underwater vehicles (AUVs). However, existing methods still face three major challenges: 1) policy non-stationarity caused by concurrent updates among multiple agents; 2) inefficient policy learning caused by the heterogeneous quality of experiences accumulated during exploration; and 3) policy drift between stochastic exploration and deterministic execution under dynamic underwater disturbances. To address these challenges, this paper develops a four-layer hierarchical MARL architecture comprising global training scheduling, multi-agent coordination, local policy generation, and real-time action execution. Building on this architecture, we propose a Supervised Diffusion-Aided MARL (SDA-MARL) algorithm with three closely coupled mechanisms. First, a dual-decision policy integrates a diffusion-based generative branch with a Deep Deterministic Policy Gradient (DDPG) branch, while segregated experience pools reduce training interference between the two branches. Second, a supervised sample-selection mechanism identifies high-quality tracking transitions and uses their actions to guide reverse diffusion, enabling the generative policy to concentrate on effective regions of the action space. Third, a behavioral-cloning loss transfers diffusion-generated actions to the deterministic DDPG Actor, thereby aligning exploration with execution and suppressing policy drift. Experiments conducted in six-degree-of-freedom underwater environments across multiple AUV-target configurations show that SDA-MARL achieves faster convergence, higher tracking accuracy, more consistent inter-AUV velocities, and shorter tracking paths than the compared MARL methods.

Figures & tables

Fig. 1: Architecture of hierarchical multi-AUV MARL framework based on supervised diffusion

Fig. 2: Workflow of proposed generative multi-AUV MARL algorithm

Fig. 3: Proposed tracking architecture based on SDA-MARL

Panel of Figure 4 — Fig. 4: Convergence speed evaluation

Parameter	Description	Value
$N_{A}$	Number of AUVs	[2,4,6,8]
$N_{T}$	Number of Targets	[1,2,3]
$l_{r}$	Learning rate	$1e^{-3}$
$N_{E}$	Training rounds	4000
$N_{h}$	Hidden layer neurons	256
$\gamma$	Discount factor	0.95

TABLE I: Simulation parameters

Algorithms	2 AUVs Tracking 1 Target	4 AUVs Tracking 2 Targets	6 AUVs Tracking 2 Targets	8 AUVs Tracking 3 Targets
Algorithms	$Mean\pm SD$	$Mean\pm SD$	$Mean\pm SD$	$Mean\pm SD$
SDA-MARL	66.44% $\pm$ 0.06%	68.56% $\pm$ 1.56%	61.38% $\pm$ 0.91%	61.67% $\pm$ 2.34%
DBSM	45.70% $\pm$ 1.75%	47.12% $\pm$ 2.82%	14.93% $\pm$ 1.41%	12.69% $\pm$ 1.77%
MA-A3C	41.04% $\pm$ 3.58%	21.74% $\pm$ 2.56%	12.53% $\pm$ 1.32%	10.30% $\pm$ 0.61%
MAPPO	12.37% $\pm$ 4.88%	8.17% $\pm$ 3.90%	10.50% $\pm$ 1.55%	28.20% $\pm$ 2.16%
MASAC	9.81% $\pm$ 0.19%	5.44% $\pm$ 0.12%	14.50% $\pm$ 1.91%	9.04% $\pm$ 0.08%

TABLE II: Tracking accuracy comparison

Algorithms	2 AUVs Tracking 1 Target	4 AUVs Tracking 2 Targets	6 AUVs Tracking 2 Targets	8 AUVs Tracking 3 Targets
Algorithms	$Mean$	$Mean$	$Mean$	$Mean$
SDA-MARL	0.015477	0.017505	0.018757	0.016895
DBSM	0.030882	0.030721	0.026736	0.026041
MA-A3C	0.027531	0.030716	0.028926	0.027545
MAPPO	0.039937	0.035275	0.035289	0.040667
MASAC	0.038340	0.033087	0.034900	0.034166

TABLE III: Velocity difference mean value comparison

Algorithms	2 AUVs Tracking 1 Target	4 AUVs Tracking 2 Targets	6 AUVs Tracking 2 Targets	8 AUVs Tracking 3 Targets
Algorithms	$SD$	$SD$	$SD$	$SD$
SDA-MARL	0.000271	0.000165	0.000062	0.000085
DBSM	0.000231	0.000183	0.000157	0.001350
MA-A3C	0.000441	0.000385	0.000145	0.000200
MAPPO	0.000593	0.000367	0.000565	0.000223
MASAC	0.000201	0.000075	0.000065	0.000143

TABLE IV: Velocity difference standard deviation comparison

Algorithms	2 AUVs Tracking 1 Target	4 AUVs Tracking 2 Targets	6 AUVs Tracking 2 Targets	8 AUVs Tracking 3 Targets
Algorithms	$Mean\pm SD$	$Mean\pm SD$	$Mean\pm SD$	$Mean\pm SD$
SDA-MARL	$1.93\pm 0.0053$	$3.48\pm 0.0044$	$6.10\pm 0.0341$	$6.80\pm 0.0257$
DBSM	$2.54\pm 0.0192$	$4.91\pm 0.0424$	$6.26\pm 0.0475$	$6.98\pm 0.4590$
MA-A3C	$2.22\pm 0.0385$	$4.87\pm 0.0641$	$6.78\pm 0.0405$	$8.36\pm 0.0686$
MAPPO	$3.34\pm 0.0330$	$5.68\pm 0.0459$	$8.55\pm 0.1274$	$13.09\pm 0.0997$
MASAC	$3.33\pm 0.0127$	$4.39\pm 0.0311$	$10.80\pm 0.0768$	$11.60\pm 0.0528$

TABLE V: Path length comparison

Fig. 5: Convergence speed across different numbers of diffusion time steps in SDA-MARL

Panel of Figure 11 — Fig. 6: Ablation evaluation

Panel of Figure 12 — Fig. 7: Availability evaluation.

Cooperative MARL

Cooperative multi-agent reinforcement learning (MARL) trains agents to coordinate toward shared goals under partial information and decentralized control.

See all 244 papers

Multi-AUV Ad-hoc network-based Target Tracking: A Value Gradient Guidance Multi-Agent Diffusion Reinforcement Learning Approach
Aug 12, 2026Jiaao Ma, Chuan Lin, Guangjie Han +4Multi-Agent Control Diffusion Policy
Software College, Northeastern University, Shenyang, China · Key Laboratory of Maritime Intelligent Network Information Technology, Ministry of Education, Hohai University, Changzhou, China · College of Computer Science and Software Engineering, Hohai University, Nanjing, 210013, China
Task-Semantic Graph-Driven Distributed Agent Networking for Underwater Target Tracking
May 15, 2026Shengchao Zhu, Guangjie Han, Chuan Lin +1Underwater Robotics Decentralized MARL
College of Computer Science and Software Engineering, Hohai University, Nanjing 210098, China · College of Information Science and Engineering, Hohai University, Changzhou 210013, China · Software College, Northeastern University, Shenyang 110819, China
Multi-AUV Ad-hoc Networks-Based Multi-Target Tracking Based on Scene-Adaptive Embodied Intelligence
Mar 28, 2026Kai Tian, Jialun Wang, Chuan Lin +4Multi-Object Tracking Underwater Robotics
Software College, Northeastern University, Shenyang, China. · Key Laboratory of Maritime Intelligent Network Information Technology, Ministry of Education, Hohai University.

Aug 12, 2026J/K move · Enter open · S save

Multi-AUV Ad-hoc network-based Target Tracking: A Value Gradient Guidance Multi-Agent Diffusion Reinforcement Learning Approach

Jiaao Ma, Chuan Lin, Guangjie Han, Shengchao Zhu+3

Software College, Northeastern University, Shenyang, China · Key Laboratory of Maritime Intelligent Network Information Technology, Ministry of Education, Hohai University, Changzhou, China · College of Computer Science and Software Engineering, Hohai University, Nanjing, 210013, China

Multi-Agent Control Diffusion Policy Underwater Robotics Multi-Agent Systems Autonomous Underwater Navigation Cooperative MARL

Open paper AskPDF arXiv

Multi-AUV ad-hoc network-based target tracking requires networked autonomous underwater vehicles (AUVs) to cooperatively track maneuvering targets under constrained acoustic communication, dynamic topology, and uncertain ocean disturbances. Although multi-agent reinforcement learning (MARL) enables decentralized coordination through centralized training, existing methods suffer from high-dimensional joint state-action modeling, noise-sensitive policy generation, leading to unstable training and degraded tracking. To address these issues, we propose VGG-MADiffRL, a value-gradient-guided multi-agent diffusion RL algorithm, and MDCA, a diffusion?based hierarchical control architecture. Leveraging underwater mission characteristics, we model sonar detection mechanisms and ocean current disturbances, formulating cooperative tracking for multi-AUV ad-hoc networks as an MDP. The proposed MDCA constitutes a three-tier closed-loop control framework: a global intelligent control layer, a local online training layer, and a physical action execution layer. This structure enables synergistic optimization across task allocation, local decision processes, and execution feedback. Within MDCA, the local online training layer is the policy learning framework; VGG-MADiffRL builds on diffusion policies and incorporates value gradients to guide action generation in the reverse denoising process, steering the generated actions towards higher expected returns. It employs twin value networks with joint optimization and soft target updates to mitigate overestimation and training oscillations, promoting more stable convergence. Experimental results show that VGG-MADiffRL consistently achieves faster convergence, higher tracking accuracy, and smoother training dynamics in cooperative tracking scenarios, validating its effectiveness and practical engineering value in dynamic underwater settings.

Abstract

Cooperative MARL

Explore similar work

Multi-AUV Ad-hoc network-based Target Tracking: A Value Gradient Guidance Multi-Agent Diffusion Reinforcement Learning Approach

Task-Semantic Graph-Driven Distributed Agent Networking for Underwater Target Tracking

Multi-AUV Ad-hoc Networks-Based Multi-Target Tracking Based on Scene-Adaptive Embodied Intelligence