Bypassing the CSI Bottleneck: MARL-Driven Spatial Control for Reflector Arrays
Authors: Hieu Le, Oguz Bedir, Mostafa Ibrahim, Jian Tao, Sabit Ekin
Organizations: Electrical and Computer Engineering, Texas A&M University, College Station, Texas, USA · Engineering Technology and Industrial Distribution, Texas A&M University, College Station, Texas, USA · School of Performance, Visualization, and Fine Arts, Texas A&M University, College Station, Texas, USA · Engineering Technology, and Electrical and Computer Engineering, Texas A&M University, College Station, Texas, USA
Reconfigurable Intelligent Surfaces (RIS) are pivotal for next-generation smart radio environments, yet their practical deployment is severely bottlenecked by the intractable computational overhead of Channel State Information (CSI) estimation. To bypass this fundamental physical-layer barrier, we propose an AI-native, data-driven paradigm that replaces complex channel modeling with spatial intelligence. This paper presents a fully autonomous Multi-Agent Reinforcement Learning (MARL) framework to control mechanically adjustable metallic reflector arrays. By mapping high-dimensional mechanical constraints to a reduced-order virtual focal point space, we deploy a Centralized Training with Decentralized Execution (CTDE) architecture. Using Multi-Agent Proximal Policy Optimization (MAPPO), our decentralized agents learn cooperative beam-focusing strategies relying on user coordinates, achieving CSI-free operation. High-fidelity ray-tracing simulations in dynamic non-line-of-sight (NLOS) environments demonstrate that this multi-agent approach rapidly adapts to user mobility, yielding up to a 26.86 dB enhancement over static flat reflectors and outperforming single-agent and hardware-constrained DRL baselines in both spatial selectivity and temporal stability. Crucially, the learned policies exhibit good deployment resilience, sustaining stable signal coverage even under 1.0-meter localization noise. These results validate the efficacy of MARL-driven spatial abstractions as a scalable, highly practical pathway toward AI-empowered wireless networks.
Figures & tables
Figure 1: Reflector Design.
Figure 2: Workflow of the Deep Reinforcement Learning and the integrated environment with Sionna and Blender.
Parameter
Value
Number of neurons (hidden layers)
256
Optimizer
Adam
Learning rate
2.0×10−4
Learning rate schedule
Constant
Discount factor ( γ )
0.985
Replay buffer size
1000
Table I: MAPPO Hyperparameters.
Figure 3: Reward convergence for proposed and baseline algorithms for reflecting element control. Our proposed beam-focusing-ma framework achieves a peak convergence value of 42, outperforming the beam-focusing-sa and column-based-ma baseline methods (both converging near 25). Mean cumulative rewards are depicted by solid lines, with shaded envelopes representing the variance across multiple training trials.
Figure 4: Heat map visualizations of spatial signal focusing capabilities: (a) an unassisted NLOS environment and (b) a conventional flat reflector. The proposed (e) multi-agent beam-focusing model demonstrates superior spatial selectivity compared to both the (c) single-agent and (d) column-based multi-agent baselines. RSSI values are color-coded in dBm; user positions are identified by purple circles, and the cylindrical obstacle is depicted in brown.
Figure 5: Temporal RSSI performance under dynamic user mobility. Over a 300-step evaluation period, the proposed multi-agent beam-focusing framework (beam-focusing-ma) achieves superior stability and signal strength, averaging -66.83 dBm. This outperforms both the hardware-constrained column-based approach (-73.65 dBm) and the single-agent baseline (-79.70 dBm). Solid lines trace the mean RSSI, shaded regions illustrate ±1 standard deviation, and horizontal dashed lines indicate the overall temporal averages.
Figure 6: Temporal RSSI performance under varying degrees of user localization noise. The system demonstrates graceful degradation, maintaining robust signal averages ranging from -68.15 dBm (ideal location data) to -72.36 dBm (1.0 m noise). Mean values are depicted by solid lines and overall temporal averages by dashed lines. The shaded regions (±1 standard deviation) illustrate the performance variance, visually highlighting the increased temporal instability the agents experience under severe positional uncertainty.
Reconfigurable Intelligent Surfaces (RIS) have the potential to engineer smart radio environments for next-generation millimeter-wave (mmWave) networks. However, the prohibitive computational overhead of Channel State Information (CSI) estimation and the dimensionality explosion inherent in centralized optimization severely hinder practical large-scale deployments. To overcome these bottlenecks, we introduce a per-element CSI-free paradigm powered by a Hierarchical Multi-Agent Reinforcement Learning (HMARL) architecture to control mechanically reconfigurable reflective surfaces. By substituting pilot-based channel estimation for each element of the device with accessible user localization data, our framework leverages spatial intelligence for macro-scale wave propagation management. The control problem is decomposed into a two-tier neural architecture: a high-level controller executes temporally extended, discrete user-to-reflector allocations, while low-level controllers autonomously optimize continuous focal points using Multi-Agent Proximal Policy Optimization (MAPPO) under a Centralized Training with Decentralized Execution (CTDE) scheme. Comprehensive deterministic ray-tracing evaluations in an indoor mmWave scenario demonstrate that this hierarchical framework achieves received signal strength indicator (RSSI) improvements of up to 7.79 dB over centralized Proximal Policy Optimization (PPO) baselines. Furthermore, the system maintains resilient beam-focusing performance under practical sub-meter localization tracking errors for up to four users and two reflector arrays. By eliminating execution-time CSI overhead while preserving high-fidelity signal redirection, this work provides a scalable and cost-effective step toward intelligent indoor wireless environments.
Hieu Le, Mostafa Ibrahim, Oguz Bedir +2
Electrical and Computer Engineering Texas A&M University College Station, Texas, USA · Engineering Technology and Industrial Distribution Texas A&M University College Station, Texas, USA · School of Performance, Visualization, and Fine Arts Texas A&M University College Station, Texas, USA
Reconfigurable Intelligent Surfaces promise to transform wireless environments, yet practical deployment is hindered by the prohibitive overhead of Channel State Information (CSI) estimation and the dimensionality explosion inherent in centralized optimization. This paper proposes a Hierarchical Multi-Agent Reinforcement Learning (HMARL) framework for the control of mechanically reconfigurable reflective surfaces in millimeter-wave (mmWave) systems. We introduce a "CSI-free" paradigm that substitutes pilot-based channel estimation with readily available user localization data. To manage the massive combinatorial action space, the proposed architecture utilizes Multi-Agent Proximal Policy Optimization (MAPPO) under a Centralized Training with Decentralized Execution (CTDE) paradigm. The proposed architecture decomposes the control problem into two abstraction levels: a high-level controller for user-to-reflector allocation and decentralized low-level controllers for low-level focal point optimization. Comprehensive ray-tracing evaluations demonstrate that the framework achieves 2.81-7.94 dB RSSI improvements over centralized baselines, with the performance advantage widening as system complexity increases. Scalability analysis reveals that the system maintains sustained efficiency, exhibiting minimal per-user performance degradation and stable total power utilization even when user density doubles. Furthermore, robustness validation confirms the framework's viability across varying reflector aperture sizes (45-99 tiles) and demonstrates graceful performance degradation under localization errors up to 0.5 m. By eliminating CSI overhead while maintaining high-fidelity beam-focusing, this work establishes HMARL as a practical solution for intelligent mmWave environments.
Cognitive radio networks (CRNs) are a key mechanism for alleviating spectrum scarcity by enabling secondary users (SUs) to opportunistically access licensed frequency bands without harmful interference to primary users (PUs). To address unreliable direct SU links and energy constraints common in next-generation wireless networks, this work introduces an adaptive, energy-aware hybrid reconfigurable intelligent surface (RIS) for underlay multiple-input single-output (MISO) CRNs. Distinct from prior approaches relying on static RIS architectures, our proposed RIS dynamically alternates between passive and active operation modes in real time according to harvested energy availability. We also model our scenario under practical hardware impairments and cascaded fading channels. We formulate and solve a joint transmit beamforming and RIS phase optimization problem via the soft actor-critic (SAC) deep reinforcement learning (DRL) method, leveraging its robustness in continuous and highly dynamic environments. Notably, we conduct the first systematic study of reward poisoning attacks on DRL agents in RIS-enhanced CRNs, and propose a lightweight, real-time defense based on reward clipping and statistical anomaly filtering. Numerical results demonstrate that the SAC-based approach consistently outperforms established DRL baselines, and that the dynamic hybrid RIS strikes a superior trade-off between throughput and energy consumption compared to fully passive and fully active alternatives. We further show the effectiveness of our defense in maintaining SU performance even under adversarial conditions. Our results advance the practical and secure deployment of RIS-assisted CRNs, and highlight crucial design insights for energy-constrained wireless systems.
Deemah H. Tashman, Soumaya Cherkaoui
LINCS Laboratory, Department of Computer and Software Engineering, Polytechnique Montréal, Montréal, QC, Canada, H3T 1J4