Multi-Agent Reinforcement Learning Counteracts Delayed CSI in Multi-Satellite Systems

Marios Aristodemou; Yasaman Omid; Sangarapillai Lambotharan; Mahsa Derakhshan; Lajos Hanzo

cs.ITMar 17, 2026

Multi-Agent Reinforcement Learning Counteracts Delayed CSI in Multi-Satellite Systems

Authors: Marios Aristodemou, Yasaman Omid, Sangarapillai Lambotharan, Mahsa Derakhshan, Lajos Hanzo

Organizations: Department of Computer Science, University of York, York, U.K. · Wolfson School of Mechanical, Electrical and Manufacturing Engineering, Loughborough University, Loughborough, U.K. · Institute for Digital Technologies, Loughborough University (London Campus), London, U.K. · School of Electronics and Computer Science, University of Southampton, Southampton, U.K.

Abstract

The integration of satellite communication networks with next-generation (NG) technologies is a promising approach towards global connectivity. However, the quality of services is highly dependant on the availability of accurate channel state information (CSI). Channel estimation in satellite communications is challenging due to the high propagation delay between terrestrial users and satellites, which results in outdated CSI observations on the satellite side. In this paper, we study the downlink transmission of multiple satellites acting as distributed base stations (BS) to mobile terrestrial users. We propose a multi-agent reinforcement learning (MARL) algorithm which aims for maximising the sum-rate of the users, while coping with the outdated CSI. We design a novel bi-level optimisation, procedure themes as dual stage proximal policy optimisation (DS-PPO), for tackling the problem of large continuous action spaces as well as of independent and non-identically distributed (non-IID) environments in MARL. Specifically, the first stage of DS-PPO maximises the sum-rate for an individual satellite and the second stage maximises the sum-rate when all the satellites cooperate to form a distributed multi-antenna BS. Our numerical results demonstrate the robustness of DS-PPO to CSI imperfections as well as the sum-rate improvement attached by the use of DS-PPO. In addition, we provide the convergence analysis for the DS-PPO along with the computational complexity.

Figures & tables

Fig. 1: System model: a cluster of $L$ satellites providing service for $K$ users.

Fig. 2: DS-PPO for enhanced TPM generation.

$\xi_{1}$	$\xi_{2}$	$\xi_{3}$	$\xi_{4}$	$\xi_{5}$
$120$	$\frac{5\xi_{1}}{4}=150$	$\frac{7\xi_{1}}{4}=210$	$\frac{9\xi_{1}}{4}=270$	$3\xi_{1}=360$

TABLE I: Parameters used in

f(c_{t})

, that ensure a minimum guaranteed sum-rate of around

300

Mbps

$\bm{L}$	$\bm{K}$	Per-Episode (GFLOPS)	Total Training (TFLOPS)
4	2	1.7	0.66
4	4	2.9	1.13
4	6	4.3	1.67
6	2	2.6	1.01
6	4	4.5	1.75
6	6	6.7	2.61

TABLE II: FLOPS Analysis for DS-PPO (

M=9

T=512

, 389 episodes, 200K samples)

Parameters	Symbol	First Stage	Second stage
Discount Factor	$\gamma$	0.99	0.95
GAE	$\lambda$	0.95	0.9
Learning Rate	$\eta$	0.003	0.004
Minibatches	$\gamma$	32	6
Clip coefficient	$\epsilon$	0.1	0.01
Maximum Gradient Norm	$\lVert\nabla_{s,a}\rVert$	1	0.6

TABLE III: PPO Hyper Parameters

Fig. 3: DS-PPO performance with $K\in\{2,4\}$ users in perfect CSI and delayed CSI , in presence of handovers. The delayed observations effects are neglectable in both cases.

Fig. 4: Comparative results with $L\in\{4,6\}$ satellites and $K=6$ users in perfect CSI and delayed CSI scenarios. The increment in satellites clearly increases the sum rate by $20\%$

Fig. 5: Comparative results with $L\in\{4,6,8\}$ satellites and $K=6$ users in delayed CSI scenarios. The highest sum rate is guaranteed with $L=6$

Fig. 6: Comparative results with other algorithms.

Method	Avg. sum-rate	DS-PPO Gain
DS-PPO, delayed CSI	$\approx 350$ (Mbps)	–
IPPO, delayed CSI	$\approx 200$ (Mbps)	$\approx 75\%$
MADDPG, delayed CSI	$\approx 125$ (Mbps)	$\approx 180\%$
CSI prediction [ 6 ] + precoding [ 10 ] , perfect CSI	$\approx 220$ (Mbps)	$\approx 59\%$
CSI prediction [ 6 ] + precoding [ 10 ] , imperfect CSI	$\approx 100$ (Mbps)	$\approx 250\%$

TABLE IV: Performance summary of DS-PPO and baseline methods for

L=4

K=4

, and delayed CSI.

Explore similar work

Learning to Focus: CSI-Free Hierarchical MARL for Reconfigurable Reflectors
Apr 6, 2026Hieu Le, Mostafa Ibrahim, Oguz Bedir +2Reconfigurable Intelligent Surfaces Millimeter Wave
Electrical and Computer Engineering Texas A&M University College Station, Texas, USA · Engineering Technology and Industrial Distribution Texas A&M University College Station, Texas, USA · School of Performance, Visualization, and Fine Arts Texas A&M University College Station, Texas, USA
Communication Gain and Delay Cost Under Cross-Timestep Delays in Cooperative Multi-Agent Reinforcement Learning
Apr 4, 2026Zihong Gao, Hongjian Liang, Lei Hao +1Multi-Agent Reinforcement Learning Mean Field Games
The State Key Laboratory for Manufacturing Systems Engineering School of Automation Science and Engineering, Xi’an Jiaotong University
Decoupled Delay Compensation: Enhancing Pre-trained MARL Policies via Learned Dynamics Filtering
May 25, 2026Maxim Mednikov, Oren GalMulti-Agent Reinforcement Learning Agentic Reinforcement Learning
University of Haifa, Israel

Apr 6, 2026, cs.AIJ/K move · Enter open · S save

Learning to Focus: CSI-Free Hierarchical MARL for Reconfigurable Reflectors

Hieu Le, Mostafa Ibrahim, Oguz Bedir, Jian Tao+1

Electrical and Computer Engineering Texas A&M University College Station, Texas, USA · Engineering Technology and Industrial Distribution Texas A&M University College Station, Texas, USA · School of Performance, Visualization, and Fine Arts Texas A&M University College Station, Texas, USA

Reconfigurable Intelligent Surfaces Millimeter Wave Multi-Agent Reinforcement Learning Hierarchical

Open paper AskPDF arXiv

Reconfigurable Intelligent Surfaces (RIS) have the potential to engineer smart radio environments for next-generation millimeter-wave (mmWave) networks. However, the prohibitive computational overhead of Channel State Information (CSI) estimation and the dimensionality explosion inherent in centralized optimization severely hinder practical large-scale deployments. To overcome these bottlenecks, we introduce a per-element CSI-free paradigm powered by a Hierarchical Multi-Agent Reinforcement Learning (HMARL) architecture to control mechanically reconfigurable reflective surfaces. By substituting pilot-based channel estimation for each element of the device with accessible user localization data, our framework leverages spatial intelligence for macro-scale wave propagation management. The control problem is decomposed into a two-tier neural architecture: a high-level controller executes temporally extended, discrete user-to-reflector allocations, while low-level controllers autonomously optimize continuous focal points using Multi-Agent Proximal Policy Optimization (MAPPO) under a Centralized Training with Decentralized Execution (CTDE) scheme. Comprehensive deterministic ray-tracing evaluations in an indoor mmWave scenario demonstrate that this hierarchical framework achieves received signal strength indicator (RSSI) improvements of up to 7.79 dB over centralized Proximal Policy Optimization (PPO) baselines. Furthermore, the system maintains resilient beam-focusing performance under practical sub-meter localization tracking errors for up to four users and two reflector arrays. By eliminating execution-time CSI overhead while preserving high-fidelity signal redirection, this work provides a scalable and cost-effective step toward intelligent indoor wireless environments.

Abstract

Explore similar work

Learning to Focus: CSI-Free Hierarchical MARL for Reconfigurable Reflectors

Communication Gain and Delay Cost Under Cross-Timestep Delays in Cooperative Multi-Agent Reinforcement Learning

Decoupled Delay Compensation: Enhancing Pre-trained MARL Policies via Learned Dynamics Filtering