Delay-Aware Reinforcement Learning for O-RAN Control under Stochastic Reward Feedback
Authors: Xingqi Wu, Junaid Farooq, Tao Li, Juntao Chen, Ying Wang
Organizations: Department of Electrical and Computer Engineering, University of Michigan-Dearborn, MI 48128 USA · Department of Systems Engineering, City University of Hong Kong, Hong Kong. · Department of Computer and Information Science, Fordham University, New York, NY USA · Department of Systems Engineering, Stevens Institute of Technology, Hoboken, NJ USA
Reinforcement learning (RL) is increasingly employed for real-time control in Open Radio Access Networks (O-RANs). However, the asynchronous nature of O-RAN feedback introduces stochastic delays between actions and their observable rewards, leading to misaligned credit assignment and unstable training. This letter formulates the O-RAN control problem as a Markov decision process (MDP) with delayed and stochastic rewards, where the true reward corresponding to an action is received after an unknown delay. To address this, we propose a delay-aware deep RL framework that integrates a long short-term memory (LSTM)-based delay predictor and a temporal reward realignment module within an off-policy learning pipeline. Using empirical delay traces collected from the Open AI Cellular (OAIC) testbed, we demonstrate that the proposed framework achieves faster convergence and significantly reduces latency violations and packet drops compared to conventional RL methods that ignore feedback delay. The results confirm that reward-delay compensation is critical for reliable AI-driven O-RAN control.
Figures & tables
Fig. 1 : Empirical observation of delayed feedback in the OAIC O-RAN testbed. The PRB allocation is reduced to zero at time t , but the downlink data rate only drops after a delay interval dt .
Fig. 2 : Control loop in O-RAN showing stochastic feedback delay between RIC-issued actions and observed rewards across the E2 interface.
Fig. 3 : Integrated training architecture for mitigating delayed reward corruption using LSTM-driven delay prediction and off-policy DRL.
Lr Params
Value
Env Params
Value
K
15
M , dm(t) (km)
4 ∼ 9, [0.4, 0.5]
αsoft-update
1e-3
C (Mbps)
1000
Ns1,Ns2,Ns3
10, 10, 10
fc (GHz)
3
μDRL , μLSTM
5e-5, 1e-3
δm(t) (Mb)
[0.003, 0.3]
SbatchDRL
256
ψξ (dB)
3.5
SbatchLSTM
32
Tn , Ts (s)
150, 0.2
TABLE I : Simulation Parameters.
Fig. 4 : Training rewards of resource orchestration agents under varying strategies.
Fig. 5 : ECCDF of per-slice latency for DQN, SAC, and DDPG with and without DRA compensation.
Fig. 6 : Aggregate QoS metrics for increasing UE count comparing DQN, SAC, DDPG, and their DRA-enhanced counterparts.
Reinforcement learning control uses reward-driven learning to make decisions in dynamic systems, from robots and vehicles to energy and infrastructure.
Stochastic communication delays in teleoperation introduce signal discontinuities that undermine control stability and degrade control performance. Consequently, the conventional reinforcement learning (RL) methods struggle with the delayed observations due to the delay-induced observations, leading to high-frequency chattering. To address this, we propose a hybrid control framework, delay-resilient RL, integrating a state estimator utilizing Long Short-Term Memory (LSTM) with a residual RL policy, which is resilient to stochastic delays. The LSTM reconstructs smooth, continuous state estimates from delayed observations, enabling the RL agent to learn a residual torque compensation policy that balances tracking accuracy with velocity smoothness. Experimental validation on Franka Panda robots demonstrates that our approach significantly outperforms the state-of-the-art baselines, ensuring robust and stable teleoperation even under high-variance stochastic delays.
Kaize Deng, Zewen Yang
Technical University of Munich, 80333 Munich, Germany
Reinforcement learning in real world environments often suffers from severe performance degradation due to delayed feedback. Existing approaches typically mitigate performance degradation caused by observation delays by constructing augmented states or predicting the true states. However, these methods often overlook the inherent discrepancy between delayed state and true states induced by stochastic MDP. We theoretically prove the existence of such a discrepancy and show that it leads to the degradation of the optimal policy. To address this challenge, we propose Diffusion Guided Uncertainty Aware Delayed Policy Optimization (DUPO). Our method explicitly models the relationship between delayed state message and the current state using a diffusion model, and leverages the resulting discrepancy estimates to weight delayed policies. Extensive experiments on continuous robotic control tasks with multiple stochastic delays demonstrate that DUPO consistently outperforms existing methods and remains effective even under long and random delay scenarios.
Junqi Tu, Zejiao Liu, Fangfei Li +1
The Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, East China University of Science and Technology, Shanghai, China · The School of Mathematics, East China University of Science and Technology, Shanghai, China
In standard reinforcement learning (RL) settings, the interaction between the agent and the environment is typically modeled as a Markov decision process (MDP), which assumes that the agent observes the system state instantaneously, selects an action without delay, and executes it immediately. In real-world dynamic environments, such as cyber-physical systems, this assumption often breaks down due to delays in the interaction between the agent and the system. These delays can vary stochastically over time and are typically unobservable when deciding on an action. Existing methods deal with this uncertainty conservatively by assuming a known fixed upper bound on the delay, even if the delay is often much lower. In this work, we introduce the interaction layer, a general framework that enables agents to adaptively handle unobservable and time-varying delays. Specifically, the agent generates a matrix of possible future actions, anticipating a horizon of potential delays, to handle both unpredictable delays and lost action packets sent over networks. Building on this framework, we develop a model-based algorithm, Actor-Critic with Delay Adaptation (ACDA), which dynamically adjusts to delay patterns. Our method significantly outperforms state-of-the-art approaches across a wide range of locomotion benchmark environments, including real-world measured delays.
John Wikman, Alexandre Proutiere, David Broman
EECS and Digital Futures, KTH Royal Institute of Technology, Stockholm, Sweden.