cs.LGMar 7, 2026

Learning to Reflect: Hierarchical Multi-Agent Reinforcement Learning for CSI-Free mmWave Beam-Focusing

Authors: Hieu Le, Oguz Bedir, Mostafa Ibrahim, Jian Tao, Sabit Ekin

Organizations: Texas A&M University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reconfigurable Intelligent Surfaces promise to transform wireless environments, yet practical deployment is hindered by the prohibitive overhead of Channel State Information (CSI) estimation and the dimensionality explosion inherent in centralized optimization. This paper proposes a Hierarchical Multi-Agent Reinforcement Learning (HMARL) framework for the control of mechanically reconfigurable reflective surfaces in millimeter-wave (mmWave) systems. We introduce a "CSI-free" paradigm that substitutes pilot-based channel estimation with readily available user localization data. To manage the massive combinatorial action space, the proposed architecture utilizes Multi-Agent Proximal Policy Optimization (MAPPO) under a Centralized Training with Decentralized Execution (CTDE) paradigm. The proposed architecture decomposes the control problem into two abstraction levels: a high-level controller for user-to-reflector allocation and decentralized low-level controllers for low-level focal point optimization. Comprehensive ray-tracing evaluations demonstrate that the framework achieves 2.81-7.94 dB RSSI improvements over centralized baselines, with the performance advantage widening as system complexity increases. Scalability analysis reveals that the system maintains sustained efficiency, exhibiting minimal per-user performance degradation and stable total power utilization even when user density doubles. Furthermore, robustness validation confirms the framework's viability across varying reflector aperture sizes (45-99 tiles) and demonstrates graceful performance degradation under localization errors up to 0.5 m. By eliminating CSI overhead while maintaining high-fidelity beam-focusing, this work establishes HMARL as a practical solution for intelligent mmWave environments.

Figures & tables

Figure 1: Reflector with hexagonal tiles. Each tile can be rotated in both elevation ( θ\theta ) and azimuth ( ϕ\phi ) angles.
Figure 2: Temporal coordination between hierarchical controllers. The high-level allocation controller makes high-level user-to-reflector assignment decisions every TT timesteps, while low-level controllers continuously optimize focal points at every timestep. This temporal abstraction enables high-level planning at the high level while maintaining rapid low-level adaptation at the low level.
Figure 3: Hierarchical Multi-Agent Reinforcement Learning Architecture. The high-level controller performs high-level user-to-reflector allocation using global system state at extended temporal intervals ( TT timesteps), while low-level controllers execute low-level focal point optimization using masked local observations at every timestep. The framework employs Hierarchical Multi-Agent Reinforcement Learning (HMARL), enabling global state access during training while maintaining practical deployment scalability through local observation-based execution. Dashed arrows represent aggregated reward feedback from low-level controllers to the high-level coordinator.
Figure 4: Experimental setup of a conference room for simulation. The AP is depicted in blue, while the users are shown in green. The users are within the UE region.
ParameterValue
System Configuration
Carrier frequency & transmit power60 GHz, 5 dBm
User scenarios & reflector arrays2, 4 users; 2 arrays
Coverage area10 m ×\times 10 m
Neural Network & Optimization
High-level layer & activationattention + 128 neurons, ReLU
Table I: System parameters and training hyperparameters
Figure 5: Training performance of the two-user scenario: episode-averaged reward for two-user scenario across four methods over 3,200 episodes.
Figure 6: Training performance of the four-user scenario: episode-averaged reward for four-user scenario across four methods over 3,200 episodes.
Figure 7: Deployment RSSI Performance for Two-User Configuration: Hierarchical MARL with and without Compatibility Matrix, Centralized PPO, and Random Assignment. Solid lines represent mean RSSI, and shaded regions denote empirical standard deviation.
Figure 8: Deployment RSSI Performance for Four-User Configuration: Hierarchical MARL with and without Compatibility Matrix, Centralized PPO, and Random Assignment. Solid lines represent mean RSSI, and shaded regions denote empirical standard deviation.
Figure 9: Mean RSSI per User: Comparison of 2-User and 4-User Deployment Scenarios. Dashed lines indicate mean performance over 300 timesteps; shaded regions denote empirical standard deviation.
Figure 10: Total System RSSI: Comparison of 2-User and 4-User Deployment Scenarios. Dashed lines indicate mean total performance over 300 timesteps; shaded regions denote empirical standard deviation.
Figure 11: RSSI Performance Across Reflector Aperture Configurations: 5-Row (45 tiles), 7-Row (63 tiles), 9-Row (81 tiles), and 11-Row (99 tiles) Implementations. Solid lines represent mean RSSI over 300 evaluation timesteps; shaded regions denote empirical standard deviation. Dashed horizontal lines indicate mean performance for each configuration.
Figure 12: Reward function sensitivity analysis: path-loss compensated reward signals. Path-Loss exponents fn∈{f0,f1,f2,f3,f4}f_{n}\in\{f_{0},f_{1},f_{2},f_{3},f_{4}\} . Solid lines represent mean RSSI over 300 evaluation timesteps; shaded regions denote empirical standard deviation.
Figure 13: RSSI Performance Under User Position Localization Errors: Performance across localization error levels ( σerror∈[0.1,3.0]\sigma_{\text{error}}\in[0.1,3.0] m), evaluated with dynamic user repositioning. Solid lines represent mean RSSI over 300 evaluation timesteps; shaded regions denote empirical standard deviation. Results demonstrate graceful degradation up to 0.5 m error, establishing practical localization accuracy requirements for reliable deployment.
More in this topic

Millimeter-Wave Communications

See all 16 papers

Explore similar work

CardsList

  1. Learning to Focus: CSI-Free Hierarchical MARL for Reconfigurable Reflectors

    Apr 6, 2026Hieu Le, Mostafa Ibrahim, Oguz Bedir +2Reconfigurable Intelligent SurfacesHierarchical RL

    Electrical and Computer Engineering Texas A&M University College Station, Texas, USA · Engineering Technology and Industrial Distribution Texas A&M University College Station, Texas, USA · School of Performance, Visualization, and Fine Arts Texas A&M University College Station, Texas, USA

  2. Bypassing the CSI Bottleneck: MARL-Driven Spatial Control for Reflector Arrays

    Apr 6, 2026Hieu Le, Oguz Bedir, Mostafa Ibrahim +2Wireless CommunicationsReconfigurable Intelligent Surfaces

    Electrical and Computer Engineering, Texas A&M University, College Station, Texas, USA · Engineering Technology and Industrial Distribution, Texas A&M University, College Station, Texas, USA · School of Performance, Visualization, and Fine Arts, Texas A&M University, College Station, Texas, USA +1

  3. Multi-Block Attention for Efficient Channel Estimation in IRS-Assisted mmWave MIMO

    May 14, 2026Mehrdad Momen-Tayefeh, Mehrshad Momen-Tayefeh, Maryam SabbaghianWireless CommunicationsChannel Estimation

    School of Electrical and Computer Engineering, University of Tehran, Tehran, Iran · Department of Computer Engineering, Sharif University of Technology, Tehran, Iran