Selecting Offline Reinforcement Learning Algorithms for Stochastic Network Control

Nicolas Helson; Pegah Alizadeh; Anastasios Giovanidis

cs.NIMar 4, 2026

Selecting Offline Reinforcement Learning Algorithms for Stochastic Network Control

Authors: Nicolas Helson, Pegah Alizadeh, Anastasios Giovanidis

Organizations: Ericsson Research, Massy, France

Abstract

Offline Reinforcement Learning (RL) is a promising approach for next-generation wireless networks, where online exploration is unsafe and large amounts of operational data can be reused across the model lifecycle. However, the behavior of offline RL algorithms under genuinely stochastic dynamics -- inherent to wireless systems due to fading, noise, and traffic mobility -- remains insufficiently understood. We address this gap by evaluating Bellman-based (Conservative Q-Learning), sequence-based (Decision Transformers), and hybrid (Critic-Guided Decision Transformers) offline RL methods in an open-access stochastic telecom environment (mobile-env). Our results show that Conservative Q-Learning consistently produces more robust policies across different sources of stochasticity, making it a reliable default choice in lifecycle-driven AI management frameworks. Sequence-based methods remain competitive and can outperform Bellman-based approaches when sufficient high-return trajectories are available. These findings provide practical guidance for offline RL algorithm selection in AI-driven network control pipelines, such as O-RAN and future 6G functions, where robustness and data availability are key operational constraints.

Figures & tables

Panel of Figure 2 — Fig. 3: Return distributions of the training trajectories for mobile-env under different mobility stochasticity settings.

Model	Limited Stochasticity	High Stochasticity	$\Delta$ (High–Limited)
DT	95.1 (9.71)	81.5 (51.6)	$-13.6$
CGDT	95.6 (9.75)	83.0 (51.2)	$-12.6$
CQL	94.6 (13.1)	84.8 (46.9)	$\mathbf{-9.8}$

TABLE I: Rescaled mean return (standard deviation in parentheses) under limited and high mobility stochasticity in mobile-env .

Setting	Model	Mean Return (Std)	$\Delta$ vs. Full
Full dataset	DT	81.5 (51.6)	-
	CGDT	83.0 (51.2)	-
	CQL	84.8 (46.9)	-
-50% medium/expert	DT	73.2 (53.3)	$-8.3$
	CGDT	78.1 (55.6)	$-4.9$
	CQL	81.0 (51.2)	$\mathbf{-3.8}$

TABLE II: Ablation study under the high mobility stochasticity.

Setting	Model	Mean Return (Std)	$\Delta$ vs. no fading
	DT	47.71 (65.57)	-33.8
Rayleigh	CGDT	80.38 (62.14)	-2.62
	CQL	93.97 (64.35)	+9.17

TABLE III: Performance under high mobility with Rayleigh fading.

Model	No Stochasticity	High Stochasticity	$\Delta$ (High–No)
DT	90.4 (16.2)	83.5 (25.5)	$-6.9$
CGDT	91.7 (14.5)	91.6 (19.6)	$\mathbf{-0.1}$
CQL	101.2 (14.2)	96.9 (17.9)	$-4.3$

TABLE IV: LunarLander : Rescaled mean return and (standard deviation).

	Rayleigh	Rician $K=3$	Rician $K=10$	No fading
Expert Mean	49.01	57.79	62.64	67.74

TABLE V: Expert dataset under different fading conditions.

Setting	Model	Mean Return (Std)	$\Delta$ vs. no fading
	DT	47.71 (65.57)	-33.8
Rayleigh	CGDT	80.38 (62.14)	-2.62
	CQL	93.97 (64.35)	+9.17
	DT	84.18 (56.06)	+2.68
Rician $K=3$	CGDT	82.35 (55.88)	-0.65
	CQL	89.29 (55.29)	+4.49

TABLE VI: mobile-env : Performance under the high mobility stochasticity setting with different fading models.

Scenario	Hyperparameters
Limited mobility	DT: $L=5$ CGDT: $L=5,\;\tau_{c}=0.5,\;\tau_{p}=0.8,\;\alpha=0.1$ CQL: $\alpha=1.0,\;l_{r}=3\times 10^{-4}$ QDT: $L=5$
High mobility	DT: $L=10$ CGDT: $L=10,\;\tau_{c}=0.5,\;\tau_{p}=0.8,\;\alpha=0.2$ CQL: $\alpha=0.1,\;l_{r}=3\times 10^{-4}$ QDT: $L=10$
Rayleigh / Rice fading	Same as high mobility stochasticity

TABLE VII: Summary of hyperparameters across all mobile-env scenarios.

Panel of Figure 10 — Fig. 4: Target return plots for mobile-env in the limited mobility stochasticity setting with the medium/expert dataset.

Panel of Figure 11 — Fig. 5: Target return plots for mobile-env in the high mobility stochasticity setting with the medium/expert dataset.

Panel of Figure 12 — Fig. 6: Target return plots of QDT for mobile-env in limited and high mobility stochasticity setting with medium/expert dataset.

Model	No Stochasticity	High Stochasticity
DT	90.4 (16.2)	83.5 (25.5)
CGDT	91.7 (14.5)	91.6 (19.6)
CQL	101.2 (14.2)	96.9 (17.9)
QDT	80.0 (26.2)	80.0 (23.5)

TABLE VIII: lunar-lander : Comparison of QDT mean return and (standard deviation) under deterministic and stochastic environments, against other Offline RL algorithms.

Model	Limited Stochasticity	High Stochasticity
DT	95.1 (9.71)	81.5 (51.6)
CGDT	95.6 (9.75)	83.0 (51.2)
CQL	94.6 (13.1)	84.8 (46.9)
QDT	95.8 (9.88)	78.3 (49.9)

TABLE IX: mobile-env : Comparison of QDT mean return and (standard deviation) under limited and high mobility stochasticity against other Offline RL algorithms.

Conservative RL

See all 10 papers

Hybrid Offline-Online Multi-Agent Decision Transformers for Wireless Resource Management
Aug 28, 2026Yiming Zhang, Kun Yang, Cong Shen +1Wireless Resource Allocation Decentralized MARL
Department of Electrical and Computer Engineering, Northwestern University, Evanston, IL 60208 USA · Department of Electrical and Computer Engineering, University of Virginia, Charlottesville, VA 22904 USA
NetRL: Powering Network Decisions via NetGRPO-Driven Reinforcement Learning
Oct 12, 2026Lingfeng Peng, Xiaohui Xie, Yang Bao +1RL Control RL for LLM Agents
Tsinghua University
Residuals-based Offline Reinforcement Learning
Apr 1, 2026Qing Zhu, Xian YuOffline RL Deep Q-Learning
Department of Integrated Systems Engineering, The Ohio State University, Columbus, OH, USA

Aug 28, 2026J/K move · Enter open · S save

Hybrid Offline-Online Multi-Agent Decision Transformers for Wireless Resource Management

Yiming Zhang, Kun Yang, Cong Shen, Dongning Guo

Department of Electrical and Computer Engineering, Northwestern University, Evanston, IL 60208 USA · Department of Electrical and Computer Engineering, University of Virginia, Charlottesville, VA 22904 USA

Wireless Resource Allocation Decentralized MARL Offline MARL Offline-To-Online RL Decision Transformer

Open paper AskPDF arXiv

This paper develops a hybrid offline-online multi-agent reinforcement learning framework based on decision transformers. The policy is first pretrained offline via supervised sequence modeling of trajectories generated by existing policies, providing a safe and sample-efficient initialization. It is then fine-tuned online using a hybrid objective that incorporates critic-guided gradients, enabling performance improvements beyond the offline policy. To facilitate stable offline-to-online transfer and effective multi-agent coordination, the framework incorporates return-weighted sampling, a critic conditioned on neighbors' actions, and neighborhood-correlated exploration. The approach is fully distributed: both training and execution rely only on local observations and limited information exchange among neighboring agents. Evaluations with dynamic traffic arrivals in two settings: (i) joint scheduling and power allocation and (ii) coordinated beamforming, show that the proposed method achieves quality-of-service (QoS) performance comparable to centralized methods. Moreover, when pretrained on lower-quality datasets, online fine-tuning is also observed to surpass the initial offline policy. These results demonstrate a promising learning-based alternative for wireless resource management.

Abstract

Conservative RL

Explore similar work

Hybrid Offline-Online Multi-Agent Decision Transformers for Wireless Resource Management

NetRL: Powering Network Decisions via NetGRPO-Driven Reinforcement Learning

Residuals-based Offline Reinforcement Learning