Bypassing the CSI Bottleneck: MARL-Driven Spatial Control for Reflector Arrays
Authors: Hieu Le, Oguz Bedir, Mostafa Ibrahim, Jian Tao, Sabit Ekin
Organizations: Electrical and Computer Engineering, Texas A&M University, College Station, Texas, USA · Engineering Technology and Industrial Distribution, Texas A&M University, College Station, Texas, USA · School of Performance, Visualization, and Fine Arts, Texas A&M University, College Station, Texas, USA · Engineering Technology, and Electrical and Computer Engineering, Texas A&M University, College Station, Texas, USA
Reconfigurable Intelligent Surfaces (RIS) are pivotal for next-generation smart radio environments, yet their practical deployment is severely bottlenecked by the intractable computational overhead of Channel State Information (CSI) estimation. To bypass this fundamental physical-layer barrier, we propose an AI-native, data-driven paradigm that replaces complex channel modeling with spatial intelligence. This paper presents a fully autonomous Multi-Agent Reinforcement Learning (MARL) framework to control mechanically adjustable metallic reflector arrays. By mapping high-dimensional mechanical constraints to a reduced-order virtual focal point space, we deploy a Centralized Training with Decentralized Execution (CTDE) architecture. Using Multi-Agent Proximal Policy Optimization (MAPPO), our decentralized agents learn cooperative beam-focusing strategies relying on user coordinates, achieving CSI-free operation. High-fidelity ray-tracing simulations in dynamic non-line-of-sight (NLOS) environments demonstrate that this multi-agent approach rapidly adapts to user mobility, yielding up to a 26.86 dB enhancement over static flat reflectors and outperforming single-agent and hardware-constrained DRL baselines in both spatial selectivity and temporal stability. Crucially, the learned policies exhibit good deployment resilience, sustaining stable signal coverage even under 1.0-meter localization noise. These results validate the efficacy of MARL-driven spatial abstractions as a scalable, highly practical pathway toward AI-empowered wireless networks.
Figures & tables
Figure 1: Reflector Design.
Figure 2: Workflow of the Deep Reinforcement Learning and the integrated environment with Sionna and Blender.
Parameter
Value
Number of neurons (hidden layers)
256
Optimizer
Adam
Learning rate
2.0×10−4
Learning rate schedule
Constant
Discount factor ( γ )
0.985
Replay buffer size
1000
Table I: MAPPO Hyperparameters.
Figure 3: Reward convergence for proposed and baseline algorithms for reflecting element control. Our proposed beam-focusing-ma framework achieves a peak convergence value of 42, outperforming the beam-focusing-sa and column-based-ma baseline methods (both converging near 25). Mean cumulative rewards are depicted by solid lines, with shaded envelopes representing the variance across multiple training trials.
Figure 4: Heat map visualizations of spatial signal focusing capabilities: (a) an unassisted NLOS environment and (b) a conventional flat reflector. The proposed (e) multi-agent beam-focusing model demonstrates superior spatial selectivity compared to both the (c) single-agent and (d) column-based multi-agent baselines. RSSI values are color-coded in dBm; user positions are identified by purple circles, and the cylindrical obstacle is depicted in brown.
Figure 5: Temporal RSSI performance under dynamic user mobility. Over a 300-step evaluation period, the proposed multi-agent beam-focusing framework (beam-focusing-ma) achieves superior stability and signal strength, averaging -66.83 dBm. This outperforms both the hardware-constrained column-based approach (-73.65 dBm) and the single-agent baseline (-79.70 dBm). Solid lines trace the mean RSSI, shaded regions illustrate ±1 standard deviation, and horizontal dashed lines indicate the overall temporal averages.
Figure 6: Temporal RSSI performance under varying degrees of user localization noise. The system demonstrates graceful degradation, maintaining robust signal averages ranging from -68.15 dBm (ideal location data) to -72.36 dBm (1.0 m noise). Mean values are depicted by solid lines and overall temporal averages by dashed lines. The shaded regions (±1 standard deviation) illustrate the performance variance, visually highlighting the increased temporal instability the agents experience under severe positional uncertainty.
Electrical and Computer Engineering Texas A&M University College Station, Texas, USA · Engineering Technology and Industrial Distribution Texas A&M University College Station, Texas, USA · School of Performance, Visualization, and Fine Arts Texas A&M University College Station, Texas, USA