Organizations: Department of Electrical and Computer Engineering, Illinois Institute of Technology, Chicago, IL 60616, USA · School of Computer Science and Engineering, California State University, San Bernardino, CA 92407, USA
The age of information (AoI) has become a central measure of data freshness in modern wireless systems. Reinforcement learning (RL) has emerged as a promising approach to AoI optimization, enabling adaptive decision-making in dynamic and stochastic wireless environments without requiring precise system models. Yet existing surveys either focus on classical AoI formulations or provide broad discussions of RL in wireless networks without addressing freshness as a unified learning problem. Motivated by this gap, this survey examines RL specifically through the lens of AoI and generalized freshness optimization. Motivated by this gap, this survey examines RL specifically through the lens of AoI and generalized freshness optimization. We organize AoI and its variants into native, function-based, and application-oriented families, providing a clearer view of how freshness should be modeled in B5G and 6G systems. Building on this foundation, we introduce a policy-centric taxonomy that reflects the decisions most relevant to freshness, consisting of update-control RL, medium-access RL, risk-sensitive RL, and multi-agent RL. This structure provides a coherent framework for understanding how learning can support important wireless networking problems, including sampling, scheduling, UAV trajectory planning, medium access, and distributed coordination. We further synthesize recent progress in RL-driven freshness control and highlight open challenges. The goal is to establish a unified foundation for learning-based freshness optimization in next-generation wireless networks.
how far the receiver lags behind the freshest update at the source.
Measures communication delay impact.
[ 97 ]
Age of Synchronization (AoS)
Time since receiver desynchronized from source.
Cache / database freshness.
[ 98 ]
Age of Version (AoV)
Version gap between source and cache.
Dynamic content caching.
[ 99 ]
Query AoI (QAoI)
AoI sampled at query instants.
Pull-based systems.
[ 100 ]
Table 4: Application-oriented AoI Metrics
Figure 3: General wireless systems.
Paper
State
Action
Reward
RL Algorithm
System Model
[ 119 ]
Current AoI, number of past retransmission attempts
Idle, transmit new packet, retransmit the previous failed packet
AoI and resource usage
SARSA
Single source-destination pair
[ 120 ]
Current AoI, number of past retransmission attempts
Idle, transmit new packet, retransmit the previous failed packet
AoI and number of transmission
Model-based
Single source to multi destinations
[ 121 ]
AoI at the destination, AoI at the queue
Sampling, scheduling
Cost of combined constraints
Model-based
Multiple sources to single destination
[ 122 ]
Battery, energy consumption, AoI at both tx and rx, number of retransmissions
Idle, transmit new packet, retransmit the previous failed packet
AoI at the receiver
PG, GR
Single source-destination pair
[ 123 ]
The most recent successful transmission event, the timestamp of the event, the timestamp of the packets sent, number of packets in the queue
Scheduling
AoI
DQN, Policy Gradient
Multiple sources to multi-destination
[ 124 ]
Age of every sensor, throughput achieved, time spent for downloading
Scheduling
Average AoI and penalties for exceeding AoI threshold
A3C
Multi sources to single destination
Table 5: Summary of RL for Freshness Optimization through Sampling and Scheduling Policies
Paper
State
Action
Reward
RL Algorithm
Objective Policy
[ 139 ]
Battery level of ground node, AoI of ground node, location of UAV, the difference between the remaining time and the required time to reach its final location
Movement, scheduling action
AoI
Proposed DRL with Q-Learning
Trajectory planning, scheduling
[ 143 ]
Sensor visited at the current time
Next sensor to be chosen
Punishment from high AoI
Q-learning
Trajectory planning
[ 144 ]
Projection of UAV on ground, AoI of all sensor nodes at UAV, time difference between the remaining time of UAV and the minimum time to reach the final destination, difference between the remaining energy of UAV and energy required for reaching the destination in the remaining time
Movement, scheduling
Additional reward for reaching destination with a non-zero residual energy, punishment for violating the constraints
DQN
Trajectory planning, scheduling
[ 145 ]
Energy levels of the nodes, current time step
Scheduling
Normalized weighted sum of AoI
Proposed DRL
Scheduling
[ 142 ]
Current location, target position, average energy efficiency, average AoI
Trajectory coordination
Energy efficiency while penalizing actions that violate constraints related to energy or AoI
DQN
Trajectory planning
[ 141 ]
Location of the UAV, current AoI of all nodes, coverage condition
Flight distance, flight speed, flight direction
Combination of average AoI, energy consumption of nodes and propulsion energy of UAV
TD3
Trajectory planning
Table 6: Summary of RL for Freshness Optimization through Trajectory Planning Policies
Paper
State
Action
Reward
RL Algorithm
MAC Component
[ 148 ]
AoI at terminal’s queue, AoI at destination
Probability
Utility minus transmission tax for transmission; no tax if silent
DDPG
Access mechanism ( p -persistent CSMA)
[ 149 ]
Current urgency order, transmitting packet age, previous action, feedback, buffer state
Transmit, stay silent
Difference between age of transmitted and updated packets
PPO
Access mechanism (slotted ALOHA)
[ 150 ]
Packet collision probability
Congestion window parameter
Average AoI (random sampling)
Double DQN
Access mechanism (CSMA/CA)
[ 151 ]
Average peak AoI, AoI of user selected by AP after collision
Mini-slot scheduling
Reduced AoI
SAC
Access mechanism (slotted ALOHA)
[ 152 ]
AoI in buffer, AoI at destination, buffer state, AoI threshold indicator
Transmit, stay silent
Reduced AoI with penalty for excessive AoI
DQN based
Access mechanism (slotted ALOHA)
Table 7: Summary of RL for Freshness Optimization in MAC
Figure 4: The architecture of MAC components from [ 153 ] .
Paper
RL Algorithm
Type of the Framework
Type of the MARL Agent
Objective Policy
[ 164 ]
DRQN based
Fully decentralized
Cooperative
Scheduling and power control policy
[ 165 ]
DDPG
CTDE
Mixed
Trajectory planning
[ 166 ]
Actor-critic based
Fully decentralized
Cooperative
Trajectory planning
[ 138 ]
QMIX
CTDE
Cooperative
Sampling and device selection
[ 148 ]
DDPG
Fully decentralized
Mixed
Access Mechanism
[ 167 ]
DQN
Fully decentralized
Competitive
Trajectory planning and scheduling
Table 8: Summary of MARL for Freshness Optimization
Figure 5: Comparison between centralized, fully decentralized, and CTDE architectures.
Offline Reinforcement Learning (RL) is a promising approach for next-generation wireless networks, where online exploration is unsafe and large amounts of operational data can be reused across the model lifecycle. However, the behavior of offline RL algorithms under genuinely stochastic dynamics -- inherent to wireless systems due to fading, noise, and traffic mobility -- remains insufficiently understood. We address this gap by evaluating Bellman-based (Conservative Q-Learning), sequence-based (Decision Transformers), and hybrid (Critic-Guided Decision Transformers) offline RL methods in an open-access stochastic telecom environment (mobile-env). Our results show that Conservative Q-Learning consistently produces more robust policies across different sources of stochasticity, making it a reliable default choice in lifecycle-driven AI management frameworks. Sequence-based methods remain competitive and can outperform Bellman-based approaches when sufficient high-return trajectories are available. These findings provide practical guidance for offline RL algorithm selection in AI-driven network control pipelines, such as O-RAN and future 6G functions, where robustness and data availability are key operational constraints.
Nicolas Helson, Pegah Alizadeh, Anastasios Giovanidis
For status update systems operating over unreliable energy-constrained wireless channels, we address Weaver's long-standing Level-C question: do my packets actually improve the plant's behavior? Each fresh sample carries a stochastic expiration time -- governed by the plant's instability dynamics -- after which the information becomes useless for control. Casting the problem as a coupon-collector variant with expiring coupons, we (i) formulate a two-dimensional average-reward MDP, (ii) prove that the optimal schedule is doubly thresholded in the receiver's freshness timer and the sender's stored lifetime, (iii) derive a closed-form policy for deterministic lifetimes, and (iv) design a Structure-Aware Q-learning algorithm (SAQ) that learns the optimal policy without knowing the channel success probability or lifetime distribution. Simulations validate our theoretical predictions: SAQ matches optimal Value Iteration performance while converging significantly faster than baseline Q-learning, and expiration-aware scheduling achieves up to 50% higher reward than age-based baselines by adapting transmissions to state-dependent urgency -- thereby delivering Level-C effectiveness under tight resource constraints.
Youssef Ahmed, Arnob Ghosh, Chih-Chun Wang +1
Department of ECE, The Ohio State University, Columbus, OH, USA · Department of Electrical and Computer Engineering, New Jersey Institute of Technology, Newark, NJ, USA · Elmore Family School of ECE, Purdue University, West Lafayette, IN, USA +1
Age of Information (AoI) has become a central metric for the design of wireless update systems, especially in applications where fresh measurements support tracking, estimation, and control. Despite its popularity, the use of mean AoI or peak AoI as a surrogate for closed-loop performance is often motivated by intuition rather than by a control-theoretic derivation. This paper examines whether minimizing the mean AoI is in fact optimal for networked control systems. For scalar linear time-invariant systems with delayed intermittent updates, we show that, under state-independent scheduling policies, the infinite-horizon LQR tracking problem reduces to an optimization over the distribution of inter-scheduling intervals. The resulting objective depends on higher-order statistical moments, and in unstable or correlated regimes on exponential moments, of the inter-scheduling process rather than only on its mean. Consequently, policies with identical mean AoI can induce substantially different tracking costs. We further extend the analysis to disturbances with exponentially decaying autocorrelation and derive equivalent cost formulations that expose the role of the full interval distribution. Finally, we evaluate the theory using real vehicle trajectories from the NGSIM US-101 dataset. The empirical results match the predicted performance trends, demonstrating that mean AoI alone is insufficient for control-oriented network design.
Abdullah Y. Etcibasi, C. Emre Koksal, Eylem Ekici
Department of Electrical and Computer Engineering, The Ohio State University, Columbus, OH 43210, USA.