cs.LGMar 7, 2026

Learning to Reflect: Hierarchical Multi-Agent Reinforcement Learning for CSI-Free mmWave Beam-Focusing

Authors: Hieu Le, Oguz Bedir, Mostafa Ibrahim, Jian Tao, Sabit Ekin

Organizations: Texas A&M University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reconfigurable Intelligent Surfaces promise to transform wireless environments, yet practical deployment is hindered by the prohibitive overhead of Channel State Information (CSI) estimation and the dimensionality explosion inherent in centralized optimization. This paper proposes a Hierarchical Multi-Agent Reinforcement Learning (HMARL) framework for the control of mechanically reconfigurable reflective surfaces in millimeter-wave (mmWave) systems. We introduce a "CSI-free" paradigm that substitutes pilot-based channel estimation with readily available user localization data. To manage the massive combinatorial action space, the proposed architecture utilizes Multi-Agent Proximal Policy Optimization (MAPPO) under a Centralized Training with Decentralized Execution (CTDE) paradigm. The proposed architecture decomposes the control problem into two abstraction levels: a high-level controller for user-to-reflector allocation and decentralized low-level controllers for low-level focal point optimization. Comprehensive ray-tracing evaluations demonstrate that the framework achieves 2.81-7.94 dB RSSI improvements over centralized baselines, with the performance advantage widening as system complexity increases. Scalability analysis reveals that the system maintains sustained efficiency, exhibiting minimal per-user performance degradation and stable total power utilization even when user density doubles. Furthermore, robustness validation confirms the framework's viability across varying reflector aperture sizes (45-99 tiles) and demonstrates graceful performance degradation under localization errors up to 0.5 m. By eliminating CSI overhead while maintaining high-fidelity beam-focusing, this work establishes HMARL as a practical solution for intelligent mmWave environments.

Figures & tables

Figure 1: Reflector with hexagonal tiles. Each tile can be rotated in both elevation ( θ\theta ) and azimuth ( ϕ\phi ) angles.
Figure 2: Temporal coordination between hierarchical controllers. The high-level allocation controller makes high-level user-to-reflector assignment decisions every TT timesteps, while low-level controllers continuously optimize focal points at every timestep. This temporal abstraction enables high-level planning at the high level while maintaining rapid low-level adaptation at the low level.
Figure 3: Hierarchical Multi-Agent Reinforcement Learning Architecture. The high-level controller performs high-level user-to-reflector allocation using global system state at extended temporal intervals ( TT timesteps), while low-level controllers execute low-level focal point optimization using masked local observations at every timestep. The framework employs Hierarchical Multi-Agent Reinforcement Learning (HMARL), enabling global state access during training while maintaining practical deployment scalability through local observation-based execution. Dashed arrows represent aggregated reward feedback from low-level controllers to the high-level coordinator.
Figure 4: Experimental setup of a conference room for simulation. The AP is depicted in blue, while the users are shown in green. The users are within the UE region.
ParameterValue
System Configuration
Carrier frequency & transmit power60 GHz, 5 dBm
User scenarios & reflector arrays2, 4 users; 2 arrays
Coverage area10 m ×\times 10 m
Neural Network & Optimization
High-level layer & activationattention + 128 neurons, ReLU
Table I: System parameters and training hyperparameters
Figure 5: Training performance of the two-user scenario: episode-averaged reward for two-user scenario across four methods over 3,200 episodes.
Figure 6: Training performance of the four-user scenario: episode-averaged reward for four-user scenario across four methods over 3,200 episodes.
Figure 7: Deployment RSSI Performance for Two-User Configuration: Hierarchical MARL with and without Compatibility Matrix, Centralized PPO, and Random Assignment. Solid lines represent mean RSSI, and shaded regions denote empirical standard deviation.
Figure 8: Deployment RSSI Performance for Four-User Configuration: Hierarchical MARL with and without Compatibility Matrix, Centralized PPO, and Random Assignment. Solid lines represent mean RSSI, and shaded regions denote empirical standard deviation.
Figure 9: Mean RSSI per User: Comparison of 2-User and 4-User Deployment Scenarios. Dashed lines indicate mean performance over 300 timesteps; shaded regions denote empirical standard deviation.
Figure 10: Total System RSSI: Comparison of 2-User and 4-User Deployment Scenarios. Dashed lines indicate mean total performance over 300 timesteps; shaded regions denote empirical standard deviation.
Figure 11: RSSI Performance Across Reflector Aperture Configurations: 5-Row (45 tiles), 7-Row (63 tiles), 9-Row (81 tiles), and 11-Row (99 tiles) Implementations. Solid lines represent mean RSSI over 300 evaluation timesteps; shaded regions denote empirical standard deviation. Dashed horizontal lines indicate mean performance for each configuration.
Figure 12: Reward function sensitivity analysis: path-loss compensated reward signals. Path-Loss exponents fn∈{f0,f1,f2,f3,f4}f_{n}\in\{f_{0},f_{1},f_{2},f_{3},f_{4}\} . Solid lines represent mean RSSI over 300 evaluation timesteps; shaded regions denote empirical standard deviation.
Figure 13: RSSI Performance Under User Position Localization Errors: Performance across localization error levels ( σerror∈[0.1,3.0]\sigma_{\text{error}}\in[0.1,3.0] m), evaluated with dynamic user repositioning. Solid lines represent mean RSSI over 300 evaluation timesteps; shaded regions denote empirical standard deviation. Results demonstrate graceful degradation up to 0.5 m error, establishing practical localization accuracy requirements for reliable deployment.
More in this topic

Millimeter-Wave Communications

See all 16 papers

Explore similar work

CardsList
Apr 6, 2026

Learning to Focus: CSI-Free Hierarchical MARL for Reconfigurable Reflectors

Reconfigurable Intelligent Surfaces (RIS) have the potential to engineer smart radio environments for next-generation millimeter-wave (mmWave) networks. However, the prohibitive computational overhead of Channel State Information (CSI) estimation and the dimensionality explosion inherent in centralized optimization severely hinder practical large-scale deployments. To overcome these bottlenecks, we introduce a per-element CSI-free paradigm powered by a Hierarchical Multi-Agent Reinforcement Learning (HMARL) architecture to control mechanically reconfigurable reflective surfaces. By substituting pilot-based channel estimation for each element of the device with accessible user localization data, our framework leverages spatial intelligence for macro-scale wave propagation management. The control problem is decomposed into a two-tier neural architecture: a high-level controller executes temporally extended, discrete user-to-reflector allocations, while low-level controllers autonomously optimize continuous focal points using Multi-Agent Proximal Policy Optimization (MAPPO) under a Centralized Training with Decentralized Execution (CTDE) scheme. Comprehensive deterministic ray-tracing evaluations in an indoor mmWave scenario demonstrate that this hierarchical framework achieves received signal strength indicator (RSSI) improvements of up to 7.79 dB over centralized Proximal Policy Optimization (PPO) baselines. Furthermore, the system maintains resilient beam-focusing performance under practical sub-meter localization tracking errors for up to four users and two reflector arrays. By eliminating execution-time CSI overhead while preserving high-fidelity signal redirection, this work provides a scalable and cost-effective step toward intelligent indoor wireless environments.

Hieu Le, Mostafa Ibrahim, Oguz Bedir +2

Electrical and Computer Engineering Texas A&M University College Station, Texas, USA · Engineering Technology and Industrial Distribution Texas A&M University College Station, Texas, USA · School of Performance, Visualization, and Fine Arts Texas A&M University College Station, Texas, USA

Reconfigurable Intelligent SurfacesHierarchical RLWireless CommunicationsWireless Resource AllocationMulti-Agent Reinforcement Learning
AskPDF
Apr 6, 2026

Bypassing the CSI Bottleneck: MARL-Driven Spatial Control for Reflector Arrays

Reconfigurable Intelligent Surfaces (RIS) are pivotal for next-generation smart radio environments, yet their practical deployment is severely bottlenecked by the intractable computational overhead of Channel State Information (CSI) estimation. To bypass this fundamental physical-layer barrier, we propose an AI-native, data-driven paradigm that replaces complex channel modeling with spatial intelligence. This paper presents a fully autonomous Multi-Agent Reinforcement Learning (MARL) framework to control mechanically adjustable metallic reflector arrays. By mapping high-dimensional mechanical constraints to a reduced-order virtual focal point space, we deploy a Centralized Training with Decentralized Execution (CTDE) architecture. Using Multi-Agent Proximal Policy Optimization (MAPPO), our decentralized agents learn cooperative beam-focusing strategies relying on user coordinates, achieving CSI-free operation. High-fidelity ray-tracing simulations in dynamic non-line-of-sight (NLOS) environments demonstrate that this multi-agent approach rapidly adapts to user mobility, yielding up to a 26.86 dB enhancement over static flat reflectors and outperforming single-agent and hardware-constrained DRL baselines in both spatial selectivity and temporal stability. Crucially, the learned policies exhibit good deployment resilience, sustaining stable signal coverage even under 1.0-meter localization noise. These results validate the efficacy of MARL-driven spatial abstractions as a scalable, highly practical pathway toward AI-empowered wireless networks.

Hieu Le, Oguz Bedir, Mostafa Ibrahim +2

Electrical and Computer Engineering, Texas A&M University, College Station, Texas, USA · Engineering Technology and Industrial Distribution, Texas A&M University, College Station, Texas, USA · School of Performance, Visualization, and Fine Arts, Texas A&M University, College Station, Texas, USA +1

Wireless CommunicationsReconfigurable Intelligent SurfacesMulti-Agent Control
AskPDF
May 14, 2026

Multi-Block Attention for Efficient Channel Estimation in IRS-Assisted mmWave MIMO

Intelligent Reflecting Surfaces (IRSs) are a promising technology for enhancing the spectral and energy efficiency of millimeter-wave (mmWave) multiple-input multiple-output (MIMO) systems. In these systems, accurate channel estimation remains challenging due to the passive nature of IRS elements and the high pilot overhead in large-scale deployments. This paper presents a deep learning-based Multi-Block Attention (MBA) framework for efficient cascaded channel estimation in IRS-assisted mmWave MIMO systems that utilize orthogonal frequency division multiplexing (OFDM). First, we show the optimality of the discrete Fourier transform (DFT) and Hadamard matrices as phase configurations for least squares (LS) estimation. To reduce training overhead, we selectively deactivate IRS elements and compensate for induced feature loss using a two-stage architecture: (i) a Convolutional Attention Network (CAN) for spatial correlation recovery and (ii) a Complex Multi-Convolutional Network (CMN) for noise suppression. The MBA architecture mitigates error propagation through attention-guided feature refinement and denoising. Simulation results indicate that the MBA method reduces pilot overhead by up to 87% compared to the LS estimator. Additionally, at signal-to-noise ratios of 10 dB, our proposed method achieves approximately 51% lower normalized mean squared error (NMSE) than leading methods. It also maintains low computational complexity and adapts effectively to various propagation environments.

Mehrdad Momen-Tayefeh, Mehrshad Momen-Tayefeh, Maryam Sabbaghian

School of Electrical and Computer Engineering, University of Tehran, Tehran, Iran · Department of Computer Engineering, Sharif University of Technology, Tehran, Iran

Wireless CommunicationsChannel EstimationMillimeter-Wave CommunicationsAttention Mechanisms
AskPDF