Organizations: the College of Information Science and Electronic Engineering, Zhejiang University · the Department of Engineering, King’s College London, London WC2R 2LS, U.K. · Zhejiang Lab
In wireless communication systems, efficient and adaptive resource allocation is essential for enhancing overall Quality of Service (QoS). Compared with conventional Model-Free Reinforcement Learning (MFRL), Model-Based RL (MBRL) first learns a generative world model for subsequent planning, enabling more stable training through the reuse of historical experience. However, deploying MBRL in large-scale wireless networks remains challenging due to high-dimensional stochastic dynamics, strong inter-agent cooperation, and stringent communication constraints. To overcome these challenges, we propose a Multi-Agent Conditional Diffusion Model Planner (MA-CDMP) for decentralized communication resource management. Following the Decentralized Training with Decentralized Execution (DTDE) paradigm, MA-CDMP models each communication node as an autonomous agent and employs Diffusion Models (DMs) to capture environment dynamics. An inverse dynamics model is further incorporated to guide action generation, improving sample efficiency and policy scalability. Moreover, to efficiently approximate large-scale agent interactions, we introduce a Mean Field (MF) mechanism to assist the classifier in conditional DMs. This mitigates inter-agent non-stationarity and enhances cooperation with minimal communication overhead. We further establish an upper bound on the distributional approximation error induced by MF-based diffusion generation, guaranteeing convergence stability and reliable modeling of multi-agent stochastic dynamics. Extensive experiments demonstrate that MA-CDMP consistently outperforms existing MARL baselines in terms of average reward and QoS metrics, showcasing its scalability and practicality for real-world wireless network optimization.
Figures & tables
Notations
Definition
N
Number of nodes
M
Number of time slots in a frame
L
Number of channels
K
Diffusion steps
H
Planning horizon
ϵθ
Diffusion noise model
TABLE I: Main parameters and notations used in this paper.
Fig. 1: Overview of the multi-node wireless topology with MF-TDMA-based MAC scheduling.
Fig. 2: The illustration of the MA-CDMP algorithm for resource allocation in wireless communication networks.
Parameters Description
Value
Number of time slots in a frame
M=10
Number of channels
L=4
Simulation area
10 km ×10 km
1-hop transmission radius of each node
3.6 km
Packet transmission rate
2 Mbps
Transmit power
100 W
TABLE II: List of Key Parameter Settings for the Simulation.
Fig. 3: Comparison of MA-CDMP with baselines in terms of average reward.
Algorithms
Packet Loss Rate (%) ↓
Average Throughput (kbps) ↑
Average Delay (ms) ↓
MA-DD
6.88±1.43
304.91±18.32
29.40±4.20
MA-Diffuser
4.40±1.69
312.72±19.90
22.00±6.90
MA-DT
4.54±0.91
315.76±18.07
23.30±4.10
MA-TD3BC
6.72±1.42
294.74±17.32
27.50±5.20
CFCQL
6.25±3.06
305.68±22.30
24.10±12.40
MA-CDMP
2.29±0.99
309.82±18.29
15.80±3.30
TABLE III: QoS comparison of MA-CDMP with baselines. Results are reported as mean ± standard deviation over 6 independent seeds.
Fig. 4: Comparison of MA-CDMP with MADiff in terms of average reward.
Fig. 5: Comparison of MA-CDMP with MA-CDMP_ w/o_MF under varying node number scenarios in terms of average reward.
Fig. 6: Comparison of different planning horizons H in terms of average reward.
Fig. 7: Comparison of different conditional guidance scales ζ in terms of average reward.
Fig. 8: Comparison of different diffusion steps K in terms of average reward.
Random access (RA) is one of the most foundational medium access control (MAC) layer scheduling schemes for handling unpredictable data traffic from multiple terminals. While multi-agent reinforcement learning (MARL) has been explored to optimize RA-based wireless networks, its reliance on experience-driven, distributed policy learning incurs significant training overhead for each optimization task, limiting its feasibility in real-world applications. In this work, we propose to leverage a foundation model (FM) to improve MARL efficiency across diverse RA network optimization tasks. Specifically, we design an FM-aided actor-critic algorithm within a consensus-based decentralized MARL architecture and provide its convergence analysis under local reward exchanges and nonlinear value function approximations to show that our algorithm achieves the same convergence order as the conventional MARL with critic model exchanges and linear approximations. Our numerical results show that our FM-based approach significantly enhances MARL speed for RA network optimization.
Myeung Suk Oh, Zhiyao Zhang, Alvaro Velasquez +2
The Ohio State University Columbus, Ohio, USA · University of Colorado Boulder Boulder, Colorado, USA · Pathfinder Applied Research, LLC Ashburn, Virginia, USA
Diffusion-based planning has achieved strong results in single-agent offline reinforcement learning, yet scaling to many-agent systems remains intractable due to the curse of dimensionality in the joint trajectory space. We introduce MF-Diffuser, a framework that lifts trajectory planning to the Wasserstein space of trajectory distributions, where the propagation of chaos ensures a small representative subset of agents captures the full population dynamics. Our approach features a value-weighted chaotic entropy objective that reconciles generative fidelity with return maximization, and a hierarchical coarse-to-fine strategy that progressively grows the agent population during denoising. We establish end-to-end suboptimality bounds with four interpretable terms, revealing that mean-field approximation error scales as O(H2/N) while offline distribution shift provably does not grow with population size N, and prove the generated policy is an approximate mean-field Nash equilibrium with explicit convergence guarantees. Experiments on three mean-field RL benchmarks -- spanning stage games, sequential dynamics, and adversarial team competition -- show MF-Diffuser achieves the best return in the majority of settings, with the largest gains on suboptimal offline data and at extreme scales (N≥103).
This paper develops a hybrid offline-online multi-agent reinforcement learning framework based on decision transformers. The policy is first pretrained offline via supervised sequence modeling of trajectories generated by existing policies, providing a safe and sample-efficient initialization. It is then fine-tuned online using a hybrid objective that incorporates critic-guided gradients, enabling performance improvements beyond the offline policy. To facilitate stable offline-to-online transfer and effective multi-agent coordination, the framework incorporates return-weighted sampling, a critic conditioned on neighbors' actions, and neighborhood-correlated exploration. The approach is fully distributed: both training and execution rely only on local observations and limited information exchange among neighboring agents. Evaluations with dynamic traffic arrivals in two settings: (i) joint scheduling and power allocation and (ii) coordinated beamforming, show that the proposed method achieves quality-of-service (QoS) performance comparable to centralized methods. Moreover, when pretrained on lower-quality datasets, online fine-tuning is also observed to surpass the initial offline policy. These results demonstrate a promising learning-based alternative for wireless resource management.
Yiming Zhang, Kun Yang, Cong Shen +1
Department of Electrical and Computer Engineering, Northwestern University, Evanston, IL 60208 USA · Department of Electrical and Computer Engineering, University of Virginia, Charlottesville, VA 22904 USA