cs.NIMar 9, 2026

Optimizing Reinforcement Learning Training over Digital Twin Enabled Multi-fidelity Networks

Authors: Hanzhi Yu, Hasan Farooq, Julien Forgeat, Shruti Bothe, Kristijonas Cyras, Md Moin Uddin Chowdhury, Mingzhe Chen

Organizations: Department of Electrical and Computer Engineering, University of Miami, Coral Gables, FL 33146 USA · Ericsson Research, Santa Clara, CA, 95054, USA · European AI Office, European Commission, Brussels, Belgium

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

In this paper, we investigate a novel digital network twin (DNT) assisted deep learning (DL) model training framework. In particular, we consider a physical network where a base station (BS) uses several antennas to serve multiple mobile users, and a DNT that is a virtual representation of the physical network. The BS must adjust its antenna tilt angles to optimize the data rates of all users. Due to user mobility, the BS may not be able to accurately track network dynamics such as wireless channels and user mobilities. Hence, a reinforcement learning (RL) approach is used to dynamically adjust the antenna tilt angles. To train the RL, we can use data collected from the physical network and the DNT. The data collected from the physical network is more accurate but incurs more communication overhead compared to the data collected from the DNT. Therefore, it is necessary to determine the ratio of data collected from the physical network and the DNT to improve the training of the RL model. We formulate this problem as an optimization problem whose goal is to jointly optimize the tilt angle adjustment policy and the data collection strategy, aiming to maximize the data rates of all users while constraining the time delay introduced by collecting data from the physical network. To solve this problem, we propose a hierarchical RL framework that integrates robust adversarial loss and proximal policy optimization (PPO). Simulation results show that our proposed method reduces the physical network data collection delay by up to 28.01% and 1x compared to a hierarchical RL that uses vanilla PPO as the first level RL, and the baseline that uses robust-RL at the first level and selects the data collection ratio randomly.

Figures & tables

Fig. 1: The considered DNT enabled cellular network.
Fig. 2: The structure of the proposed hierarchical RL framework.
Data TypePhysical Network DataDNT Data
Static DataU,U\mathcal{U},UU,U\mathcal{U},U
C,C\mathcal{C},CC,C\mathcal{C},C
PPPP
θV,θH\theta^{\textrm{V}},\theta^{\textrm{H}}θV,θH\theta^{\textrm{V}},\theta^{\textrm{H}}
λV,λH\lambda^{\textrm{V}},\lambda^{\textrm{H}}λV,λH\lambda^{\textrm{V}},\lambda^{\textrm{H}}
Runtime Datal1,tU,...,lU,tU\boldsymbol{l}_{1,t}^{\textrm{U}},...,\boldsymbol{l}_{U,t}^{\textrm{U}}l^1,tU,...,l^U,tU\hat{\boldsymbol{l}}_{1,t}^{\textrm{U}},...,\hat{\boldsymbol{l}}_{U,t}^{\textrm{U}}
TABLE I: Summary of DNT Data
ParamValueParamValueParamValue
UU10CC3P0P_{0}1
gutUg^{\textrm{U}}_{ut}1δuct\delta_{uct}1aa1
β\beta2N0N_{0}1×10−61\times 10^{-6}WW1
θV\theta^{\textrm{V}}30∘30^{\circ}θH\theta^{\textrm{H}}120∘120^{\circ}NN3
ϕ1tA\phi^{\textrm{A}}_{1t}0∘0^{\circ}ϕ2tA\phi^{\textrm{A}}_{2t}120∘120^{\circ}ϕ3tA\phi^{\textrm{A}}_{3t}240∘240^{\circ}
DD1τmax\tau_{max}150ψminT\psi^{\textrm{T}}_{min}0∘0^{\circ}
TABLE II: System Parameters [ 40 , 41 , 30 ]
Fig. 3: The delay of collecting data from the physical network as the number epochs varies.
Fig. 4: The convergence of the second level PPO.
Fig. 5: The convergence of the first level RL.
Fig. 6: Performance of the second level PPO under different DNT error levels ε\varepsilon .
Fig. 7: The optimal performance of the second level RL as the number of users varies.
Fig. 8: The performance of first level robust-RL as value of κ\kappa varies.
Fig. 9: The optimal performance of the second level RLs as the number of users varies.
More in this topic

Hierarchical RL

Hierarchical reinforcement learning studies policies that organize decisions into reusable skills, subgoals, or actions operating at different timescales.

See all 157 papers

Explore similar work

CardsList

  1. TwinLoop: Simulation-in-the-Loop Digital Twins for Online Multi-Agent Reinforcement Learning

    Apr 8, 2026Nan Zhang, Zishuo Wang, Shuyu Huang +4Digital TwinsMulti-Agent Reinforcement Learning

    Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China · Research Institute of Trustworthy Autonomous Systems, Southern University of Science and Technology, Shenzhen, China · Department of Informatics and Telecommunications, University of Thessaly, Lamia, Greece

  2. DT2\text{DT}^2: Decision-Targeted Digital Twins

    Jun 24, 2026Harry Amad, Mihaela van der SchaarDecision-Focused LearningDigital Twins

    Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, United Kingdom.

  3. Low-Altitude Fluid Antenna Network with Multi-Agent Reinforcement Learning

    Aug 28, 2026Tong Zhang, Yanfei Su, Shuai Wang +3Wireless CommunicationsWireless Resource Allocation

    Guangdong Provincial Key Laboratory of Aerospace Communication and Networking Technology, Harbin Institute of Technology, Shenzhen, 518055, China · Harbin Institute of Technology, Shenzhen, China · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China +3