CALF: Communication-Aware Learning Framework for Distributed Reinforcement Learning

Carlos Purves; Pietro Lio'

cs.LGMar 13, 2026

CALF: Communication-Aware Learning Framework for Distributed Reinforcement Learning

Authors: Carlos Purves, Pietro Lio'

Organizations: University of Cambridge

Abstract

Distributed reinforcement learning policies face network delays, jitter, and packet loss when deployed across edge devices and cloud servers. Standard RL training assumes zero-latency interaction, causing severe performance degradation under realistic network conditions. We introduce CALF (Communication-Aware Learning Framework), which trains policies under realistic network models during simulation. Systematic experiments demonstrate that network-aware training substantially reduces deployment performance gaps compared to network-agnostic baselines. Distributed policy deployments across heterogeneous hardware validate that explicitly modelling communication constraints during training enables robust real-world execution. These findings establish network conditions as a major axis of sim-to-real transfer for Wi-Fi-like distributed deployments, complementing physics and visual domain randomisation.

Figures & tables

Figure 1: Real-world distributed systems employ hybrid communication strategies to maintain operation under varying network conditions. Unmanned aerial vehicles, for instance, choose between multiple communication channels (satellite, radio frequency, optical tether) based on signal quality and operational constraints, defaulting to autonomous operation when no reliable connection exists. This illustrates the challenge CALF addresses: policies must function across heterogeneous network conditions rather than assuming perfect connectivity. We focus on LAN-like scenarios (Wi-Fi, Ethernet); WAN/adversarial scenarios motivate the need for configurable impairments but are not evaluated here.

Goal	Capability	Enables
Network Realism	Synthetic + trace-based network models	Controlled ablations (RQ2), realistic training
Deployment Parity	Same code across simulation/hardware	Fair comparison of network effects (RQ1)
Reproducibility	Deterministic seeds, versioning	Scientific rigour, exact replication
Heterogeneity	Edge devices to cloud servers	Realistic distributed settings (RQ3)

Table 1: CALF design goals and their role in answering research questions.

Figure 2: CALF’s three progressive deployment modes enable incremental validation from pure simulation to distributed deployment. Mode 1 (Local Sim) provides a zero-latency baseline for rapid development with environment and policy co-located. Mode 2 (Sim + Simulated Network) introduces NetworkShim services that inject realistic latency, jitter, and packet loss for network-aware training. Mode 3 (Edge Sim) validates distributed deployment on real hardware (Raspberry Pi for environment, Desktop for policy) communicating over real Wi-Fi/Ethernet networks. This progressive approach ensures that network-aware policies trained in Mode 2 transfer successfully to distributed edge deployment in Mode 3, addressing the network axis of the sim-to-real gap.

Regime	Mode	Impairments	Purpose
Baseline	Mode 1 (local)	None	Traditional RL (no awareness)
Delay-Only	Mode 2	50ms fixed latency	Constant-delay awareness
Full Net-Aware	Mode 2	Latency + jitter + loss	Full stochastic model

Table 2: Training regimes for network-aware evaluation.

Task	Representation	Rationale
CartPole	Frame stack $k{=}d{+}1$	Infer velocities under delay
MiniGrid	LSTM	Variable delay + partial observability
Ablation	+ Action history	Track in-flight actions

Table 3: State representation strategies for handling observation delays.

Training Regime	Sim-Clean	Sim+Net	Real-Eth	Wi-Fi-N	Wi-Fi-D
Baseline	495 $\pm$ 7	310 $\pm$ 48	288 $\pm$ 62	173 $\pm$ 71	92 $\pm$ 54
Delay-Only	482 $\pm$ 11	468 $\pm$ 16	425 $\pm$ 32	348 $\pm$ 49	218 $\pm$ 58
Full Net-Aware	476 $\pm$ 9	472 $\pm$ 13	458 $\pm$ 22	442 $\pm$ 27	378 $\pm$ 41

Table 4: CartPole: Mean Episode Return (

\pm

std) over 10 seeds. Each cell reports mean performance across seeds, with each seed evaluated over 50 episodes per deployment mode.

Figure 3: CartPole: Performance comparison across deployment modes for each training regime. Full network-aware training maintains robust performance under real network conditions, while baseline training exhibits severe degradation. Delay-only training provides partial robustness, demonstrating the necessity of modelling jitter and packet loss in addition to latency. Error bars represent

\pm

1 std across 10 seeds.

Training Regime	Sim-Clean	Sim+Net	Real-Eth	Wi-Fi-N	Wi-Fi-D
Baseline	94 $\pm$ 4	76 $\pm$ 9	73 $\pm$ 11	61 $\pm$ 13	44 $\pm$ 16
Delay-Only	91 $\pm$ 5	87 $\pm$ 6	84 $\pm$ 7	77 $\pm$ 9	64 $\pm$ 11
Full Net-Aware	89 $\pm$ 4	87 $\pm$ 5	85 $\pm$ 6	81 $\pm$ 7	74 $\pm$ 9

Table 5: MiniGrid: Success Rate (%

\pm

std) over 10 seeds. Each cell reports percentage of episodes successfully reaching the goal, with each seed evaluated over 50 episodes per deployment mode.

Training Regime	Real-Wi-Fi-Degraded
Baseline (none)	92 $\pm$ 54
Latency-Only (50 ms)	275 $\pm$ 52
Stochastic Add. Delay ( $\sigma$ =40 ms)	315 $\pm$ 47
Loss-Only (10%)	308 $\pm$ 49
Combined (full model)	378 $\pm$ 41

Table 6: CartPole Ablation: Mean Episode Return in Real-Wi-Fi-Degraded. Ten seeds per training condition, each evaluated over 50 episodes.

Deployment Configuration	Episode Return	E2E Latency (p50/p95)
Flat (Desktop)	448 $\pm$ 28	38 ms / 62 ms
Flat (Pi)	472 $\pm$ 21	6 ms / 11 ms
Hierarchical (Distributed)	465 $\pm$ 24	22 ms / 45 ms

Table 7: CartPole Policy Graph: Performance comparison for distributed deployment. All configurations evaluated over Real-Wi-Fi-Normal network.

Deployment Configuration	Success Rate (%)
Flat (Desktop)	77 $\pm$ 9
Flat (Pi)	82 $\pm$ 7
Hierarchical (Distributed)	79 $\pm$ 8

Table 8: MiniGrid Policy Graph: Success rate comparison. All configurations evaluated over Real-Wi-Fi-Normal network.

Configuration	Latency p50 (ms)	Latency p95 (ms)
Local (Pi only)	5.2	9.8
Ethernet (Pi $\leftrightarrow$ Desktop)	8.7	14.3
Wi-Fi-Normal	34.5	68.2
Real-Wi-Fi-Degraded	82.1	152.7

Table 9: End-to-End Latency: Median and 95th percentile latency measured over 1000 environment steps during CartPole policy graph deployment. “Local” indicates co-located environment and policy; “Remote” indicates networked communication.

Distributed RL

Distributed reinforcement learning coordinates agents, learners, and compute to scale policy training across networks and machines.

See all 20 papers

Align and Filter: Improving Performance in Asynchronous On-Policy RL
Mar 2, 2026Homayoun Honari, Roger Creus Castanyer, Michael Przystupa +3Asynchronous RL Policy Gradient Methods
Mila – Quebec AI Institute, Canada · Université de Montréal, Canada · University of Alberta, Canada
Distributed Training using an Intelligent Network
Aug 26, 2026Nihar Shah, Ben BlierDistributed Training Communication-Efficient Distributed Training
DoubleZero Foundation
Position: Deployed Reinforcement Learning should be Continual
Jun 1, 2026Parnian Behdin, Kevin Roice, Golnaz MesbahiNon-Stationary RL Reinforcement Learning
Alberta Machine Intelligence Institute (Amii), Edmonton, Canada.

Mar 2, 2026J/K move · Enter open · S save

Align and Filter: Improving Performance in Asynchronous On-Policy RL

Homayoun Honari, Roger Creus Castanyer, Michael Przystupa, Michael Noukhovitch+2

Mila – Quebec AI Institute, Canada · Université de Montréal, Canada · University of Alberta, Canada

Asynchronous RL Policy Gradient Methods

Open paper AskPDF arXiv

Distributed training and increasing the gradient update frequency are practical strategies to accelerate learning and improve performance, but both exacerbate a central challenge: policy lag, which is the mismatch between the behavior policy generating data and the learning policy being updated. Policy lag can hinder the scaling of on-policy learning algorithms to larger problems. In this paper, we identify the sources of policy lag caused by distributed learning and high update frequency. We use the findings to propose total Variation-based Advantage aligned Constrained policy Optimization (\methodacronym) as a practical approach to mitigate policy lag. We empirically validate our method and show that it offers better robustness to policy lag in classic RL tasks and a modern RL for LLM math reasoning task.

Abstract

Distributed RL

Explore similar work

Align and Filter: Improving Performance in Asynchronous On-Policy RL

Distributed Training using an Intelligent Network

Position: Deployed Reinforcement Learning should be Continual