Organizations: Department of Electrical and Computer Engineering, University of Miami, Coral Gables, FL 33146 USA · Ericsson Research, Santa Clara, CA, 95054, USA · European AI Office, European Commission, Brussels, Belgium
In this paper, we investigate a novel digital network twin (DNT) assisted deep learning (DL) model training framework. In particular, we consider a physical network where a base station (BS) uses several antennas to serve multiple mobile users, and a DNT that is a virtual representation of the physical network. The BS must adjust its antenna tilt angles to optimize the data rates of all users. Due to user mobility, the BS may not be able to accurately track network dynamics such as wireless channels and user mobilities. Hence, a reinforcement learning (RL) approach is used to dynamically adjust the antenna tilt angles. To train the RL, we can use data collected from the physical network and the DNT. The data collected from the physical network is more accurate but incurs more communication overhead compared to the data collected from the DNT. Therefore, it is necessary to determine the ratio of data collected from the physical network and the DNT to improve the training of the RL model. We formulate this problem as an optimization problem whose goal is to jointly optimize the tilt angle adjustment policy and the data collection strategy, aiming to maximize the data rates of all users while constraining the time delay introduced by collecting data from the physical network. To solve this problem, we propose a hierarchical RL framework that integrates robust adversarial loss and proximal policy optimization (PPO). Simulation results show that our proposed method reduces the physical network data collection delay by up to 28.01% and 1x compared to a hierarchical RL that uses vanilla PPO as the first level RL, and the baseline that uses robust-RL at the first level and selects the data collection ratio randomly.
Figures & tables
Fig. 1: The considered DNT enabled cellular network.
Fig. 2: The structure of the proposed hierarchical RL framework.
Data Type
Physical Network Data
DNT Data
Static Data
U,U
U,U
C,C
C,C
P
P
θV,θH
θV,θH
λV,λH
λV,λH
Runtime Data
l1,tU,...,lU,tU
l^1,tU,...,l^U,tU
TABLE I: Summary of DNT Data
Param
Value
Param
Value
Param
Value
U
10
C
3
P0
1
gutU
1
δuct
1
a
1
β
2
N0
1×10−6
W
1
θV
30∘
θH
120∘
N
3
ϕ1tA
0∘
ϕ2tA
120∘
ϕ3tA
240∘
D
1
τmax
150
ψminT
0∘
TABLE II: System Parameters [ 40 , 41 , 30 ]
Fig. 3: The delay of collecting data from the physical network as the number epochs varies.
Fig. 4: The convergence of the second level PPO.
Fig. 5: The convergence of the first level RL.
Fig. 6: Performance of the second level PPO under different DNT error levels ε .
Fig. 7: The optimal performance of the second level RL as the number of users varies.
Fig. 8: The performance of first level robust-RL as value of κ varies.
Fig. 9: The optimal performance of the second level RLs as the number of users varies.
Hierarchical reinforcement learning studies policies that organize decisions into reusable skills, subgoals, or actions operating at different timescales.
Decentralised online learning enables runtime adaptation in cyber-physical multi-agent systems, but when operating conditions change, learned policies often require substantial trial-and-error interaction before recovering performance. To address this, we propose TwinLoop, a simulation-in-the-loop digital twin framework for online multi-agent reinforcement learning. When a context shift occurs, the digital twin is triggered to reconstruct the current system state, initialise from the latest agent policies, and perform accelerated policy improvement with simulation what-if analysis before synchronising updated parameters back to the agents in the physical system. We evaluate TwinLoop in a vehicular edge computing task-offloading scenario with changing workload and infrastructure conditions. The results suggest that digital twins can improve post-shift adaptation efficiency and reduce reliance on costly online trial-and-error.
Nan Zhang, Zishuo Wang, Shuyu Huang +4
Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China · Research Institute of Trustworthy Autonomous Systems, Southern University of Science and Technology, Shenzhen, China · Department of Informatics and Telecommunications, University of Thessaly, Lamia, Greece
A digital twin (DT) is a virtual model of a real-world system that can assist decision-making by simulating scenarios induced by different policies. However, typical machine learning-based DTs do not optimise for this use case. We prove that, when model capacity is limited, training DTs to minimise one-step transition errors can produce suboptimal models for ranking sets of policies according to a reward function. We further show that this holds empirically, even with expressive model classes. To address this, we introduce DT2, a decision-targeted DT training paradigm. Firstly, DT2 uses fitted Q-evaluation to estimate values of candidate policies from offline data. A DT is then trained to generate rollouts that preserve pairwise policy rankings derived from these proxy ground-truth values with an architecture-agnostic loss function. We empirically demonstrate the efficacy of our method across a range of settings and architectures. DT2 consistently improves policy ranking and reduces decision regret during policy selection relative to conventional DT training, both for policies used during training and for unseen policies, while maintaining a good level of raw simulation fidelity.
Harry Amad, Mihaela van der Schaar
Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, United Kingdom.
Low-altitude wireless networks (LAWNs) integrate terrestrial and aerial platforms to provide ubiquitous communication, sensing, and localization services for unmanned aerial vehicles (UAVs) and electric vertical takeoff and landing (eVTOL) aircraft. However, dynamic air-ground and air-air channels, abrupt blockages, and heterogeneous interference hinder the realization of this goal. Nevertheless, fluid antenna (FA), a cutting-edge multiple-input multiple-output (MIMO) technique, overcomes these challenges by reconfiguring antenna positions to unlock additional spatial degrees-of-freedom. In this paper, towards bringing low-altitude FA networks into reality, we study the fast and high-performance FA reconfiguration for low-altitude FA networks with multi-agent reinforcement learning (MARL). Specifically, we present an electromagnetic digital twin (EM-DT)-assisted MARL framework. To fill the sim-to-real gap, we introduce a two-stage transfer learning framework. Our case study shows that joint FA positions and beamforming optimization can enhance the system sum-rate by 118.5%, compared to the fixed position baseline. This gain comes from the dynamic millisecond timescale reconfiguration of FA arrays and the adaptive steering of beams toward aerial users with mobility.
Tong Zhang, Yanfei Su, Shuai Wang +3
Guangdong Provincial Key Laboratory of Aerospace Communication and Networking Technology, Harbin Institute of Technology, Shenzhen, 518055, China · Harbin Institute of Technology, Shenzhen, China · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China +3