A Survey of Freshness-Aware Wireless Networking with Reinforcement Learning
Organizations: Department of Electrical and Computer Engineering, Illinois Institute of Technology, Chicago, IL 60616, USA · School of Computer Science and Engineering, California State University, San Bernardino, CA 92407, USA
Abstract
The age of information (AoI) has become a central measure of data freshness in modern wireless systems. Reinforcement learning (RL) has emerged as a promising approach to AoI optimization, enabling adaptive decision-making in dynamic and stochastic wireless environments without requiring precise system models. Yet existing surveys either focus on classical AoI formulations or provide broad discussions of RL in wireless networks without addressing freshness as a unified learning problem. Motivated by this gap, this survey examines RL specifically through the lens of AoI and generalized freshness optimization. Motivated by this gap, this survey examines RL specifically through the lens of AoI and generalized freshness optimization. We organize AoI and its variants into native, function-based, and application-oriented families, providing a clearer view of how freshness should be modeled in B5G and 6G systems. Building on this foundation, we introduce a policy-centric taxonomy that reflects the decisions most relevant to freshness, consisting of update-control RL, medium-access RL, risk-sensitive RL, and multi-agent RL. This structure provides a coherent framework for understanding how learning can support important wireless networking problems, including sampling, scheduling, UAV trajectory planning, medium access, and distributed coordination. We further synthesize recent progress in RL-driven freshness control and highlight open challenges. The goal is to establish a unified foundation for learning-based freshness optimization in next-generation wireless networks.
Figures & tables
| Survey | Freshness Metrics Discussed | Machine Learning Methods | Reinforcement Learning & Policy-Centric Discussion | Application Scenario / Scope |
| [ 2 ] | AoI, Peak AoI, average AoI, age-penalty functions (listed individually; no classification). | ML mentioned as future direction | None; RL papers only briefly cited. | General AoI theory across wireless networks |
| [ 40 ] | AoI, PAoI, Value of Information Update (VoIU), age penalty function (briefly mentioned). | None | None | RF-powered IoT networks |
| [ 41 ] | AoI, AoI-related penalty functions | ML mentioned briefly for generic optimization and prediction | None | Cellular IoT |
| [ 42 ] | AoI, PAoI, VoI | None | None | Massive IoT / Ambient Intelligence networks. |
| [ 43 ] | AoI, PAoI, AoX family (AUD, AoCI, etc.) | RL | RL papers listed, but no RL policy taxonomy | IoT systems |
| [ 44 ] | AoI, PAoI | None | None | Cellular wireless networks (5G) |
| Variant | Definition / Key Idea | Use Cases | Selected Work |
|---|---|---|---|
| Classical AoI | Time elapsed since generation of the latest received update. | Baseline freshness measure. | [ 1 ] |
| Peak AoI (PAoI) | AoI just before each successful reception. | Tracks worst-case information freshness. | [ 90 ] |
| AoI Distribution | Computes the stationary distribution of classical AoI, including moments, distributions, and extreme-value behavior via renewal and queueing analysis. | Characterizing full stochastic behavior of native AoI processes. | [ 91 , 92 ] |
| Variant | Definition / Key Idea | Use Cases | Selected Work |
|---|---|---|---|
| Age-penalty / Utility Function | Generalized penalty or utility function of AoI. | Captures heterogeneous urgency. | [ 14 ] |
| Statistical / Risk-sensitive AoI | Probabilistic or CVaR-based penalty. | Risk-aware freshness control. | [ 93 ] |
| CoUD / VoIU | Cumulative cost or value gained per update. | Quantifies information value. | [ 15 ] |
| Effective AoI | Function of estimation error instead of time. | Control-oriented AoI. | [ 94 ] |
| Variant | Definition / Key Idea | Use Cases | Selected Work |
|---|---|---|---|
| On-demand AoI | AoI evaluated only when the destination requests a status update (event-driven freshness). | Energy-limited or request-driven systems. | [ 95 ] |
| Effective AoI (EAoI) | freshness matters only when information is actually used for decision-making. | Request-driven systems, proactive update scheduling, decision-centric freshness. | [ 96 ] |
| Relative AoI (rAoI) | how far the receiver lags behind the freshest update at the source. | Measures communication delay impact. | [ 97 ] |
| Age of Synchronization (AoS) | Time since receiver desynchronized from source. | Cache / database freshness. | [ 98 ] |
| Age of Version (AoV) | Version gap between source and cache. | Dynamic content caching. | [ 99 ] |
| Query AoI (QAoI) | AoI sampled at query instants. | Pull-based systems. | [ 100 ] |
| Paper | State | Action | Reward | RL Algorithm | System Model |
| [ 119 ] | Current AoI, number of past retransmission attempts | Idle, transmit new packet, retransmit the previous failed packet | AoI and resource usage | SARSA | Single source-destination pair |
| [ 120 ] | Current AoI, number of past retransmission attempts | Idle, transmit new packet, retransmit the previous failed packet | AoI and number of transmission | Model-based | Single source to multi destinations |
| [ 121 ] | AoI at the destination, AoI at the queue | Sampling, scheduling | Cost of combined constraints | Model-based | Multiple sources to single destination |
| [ 122 ] | Battery, energy consumption, AoI at both tx and rx, number of retransmissions | Idle, transmit new packet, retransmit the previous failed packet | AoI at the receiver | PG, GR | Single source-destination pair |
| [ 123 ] | The most recent successful transmission event, the timestamp of the event, the timestamp of the packets sent, number of packets in the queue | Scheduling | AoI | DQN, Policy Gradient | Multiple sources to multi-destination |
| [ 124 ] | Age of every sensor, throughput achieved, time spent for downloading | Scheduling | Average AoI and penalties for exceeding AoI threshold | A3C | Multi sources to single destination |
| Paper | State | Action | Reward | RL Algorithm | Objective Policy |
| [ 139 ] | Battery level of ground node, AoI of ground node, location of UAV, the difference between the remaining time and the required time to reach its final location | Movement, scheduling action | AoI | Proposed DRL with Q-Learning | Trajectory planning, scheduling |
| [ 143 ] | Sensor visited at the current time | Next sensor to be chosen | Punishment from high AoI | Q-learning | Trajectory planning |
| [ 144 ] | Projection of UAV on ground, AoI of all sensor nodes at UAV, time difference between the remaining time of UAV and the minimum time to reach the final destination, difference between the remaining energy of UAV and energy required for reaching the destination in the remaining time | Movement, scheduling | Additional reward for reaching destination with a non-zero residual energy, punishment for violating the constraints | DQN | Trajectory planning, scheduling |
| [ 145 ] | Energy levels of the nodes, current time step | Scheduling | Normalized weighted sum of AoI | Proposed DRL | Scheduling |
| [ 142 ] | Current location, target position, average energy efficiency, average AoI | Trajectory coordination | Energy efficiency while penalizing actions that violate constraints related to energy or AoI | DQN | Trajectory planning |
| [ 141 ] | Location of the UAV, current AoI of all nodes, coverage condition | Flight distance, flight speed, flight direction | Combination of average AoI, energy consumption of nodes and propulsion energy of UAV | TD3 | Trajectory planning |
| Paper | State | Action | Reward | RL Algorithm | MAC Component |
| [ 148 ] | AoI at terminal’s queue, AoI at destination | Probability | Utility minus transmission tax for transmission; no tax if silent | DDPG | Access mechanism ( -persistent CSMA) |
| [ 149 ] | Current urgency order, transmitting packet age, previous action, feedback, buffer state | Transmit, stay silent | Difference between age of transmitted and updated packets | PPO | Access mechanism (slotted ALOHA) |
| [ 150 ] | Packet collision probability | Congestion window parameter | Average AoI (random sampling) | Double DQN | Access mechanism (CSMA/CA) |
| [ 151 ] | Average peak AoI, AoI of user selected by AP after collision | Mini-slot scheduling | Reduced AoI | SAC | Access mechanism (slotted ALOHA) |
| [ 152 ] | AoI in buffer, AoI at destination, buffer state, AoI threshold indicator | Transmit, stay silent | Reduced AoI with penalty for excessive AoI | DQN based | Access mechanism (slotted ALOHA) |
| Paper | RL Algorithm | Type of the Framework | Type of the MARL Agent | Objective Policy |
| [ 164 ] | DRQN based | Fully decentralized | Cooperative | Scheduling and power control policy |
| [ 165 ] | DDPG | CTDE | Mixed | Trajectory planning |
| [ 166 ] | Actor-critic based | Fully decentralized | Cooperative | Trajectory planning |
| [ 138 ] | QMIX | CTDE | Cooperative | Sampling and device selection |
| [ 148 ] | DDPG | Fully decentralized | Mixed | Access Mechanism |
| [ 167 ] | DQN | Fully decentralized | Competitive | Trajectory planning and scheduling |
Age of Information
Age of Information (AoI) measures how stale received data is and guides timely updates in communication, control, and distributed systems.
See all 22 papers