A Constrained RL Approach for Cost-Efficient Delivery of Latency-Sensitive Applications

Ozan Aygün; Vincenzo Norman Vitale; Antonia M. Tulino; Hao Feng; Elza Erkip; Jaime Llorca

cs.NIMar 4, 2026

A Constrained RL Approach for Cost-Efficient Delivery of Latency-Sensitive Applications

Authors: Ozan Aygün, Vincenzo Norman Vitale, Antonia M. Tulino, Hao Feng, Elza Erkip, Jaime Llorca

Organizations: New York University, NY, USA · Università degli Studi Napoli Federico II, Naples, Italy · Intel Labs · Centre Tecnologic de Telecommunicacions de Catalunya (CTTC), Barcelona, Spain

Abstract

Next-generation networks aim to provide performance guarantees to real-time interactive services that require timely and cost-efficient packet delivery. In this context, the goal is to reliably deliver packets with strict deadlines imposed by the application while minimizing overall resource allocation cost. A large body of work has leveraged stochastic optimization techniques to design efficient dynamic routing and scheduling solutions under average delay constraints; however, these methods fall short when faced with strict per-packet delay requirements. We formulate the minimum-cost delay-constrained network control problem as a constrained Markov decision process and utilize constrained deep reinforcement learning (CDRL) techniques to effectively minimize total resource allocation cost while maintaining timely throughput above a target reliability level. Results indicate that the proposed CDRL-based solution can ensure timely packet delivery even when existing baselines fall short, and it achieves lower cost compared to other throughput-maximizing methods.

Figures & tables

Figure 1: Illustration of lifetime-based queue dynamics: Packets turn from green to red as their lifetime decreases, blue arrows represent lifetime queue evolution, and packets without edges and with dashed edges indicate available packets at times

t-1

and

t

, respectively.

Figure 2: Illustration of an edge network topology.

Figure 3: Progression of

\bm{\lambda}

and average timely throughput of commodities as CDRL-NC agents train for the edge network. Solid curves represent the instantaneous timely throughput whose throughput constraints are marked with horizontal dotted dashed lines, and dashed curves represent the instantaneous

\bm{\lambda}

values. When the timely throughput is low,

\bm{\lambda}

values increase in order to prioritize timely throughput more in the reward function on Eq. ( 16 ). As the timely throughput targets are satisfied, the

\bm{\lambda}

values become more stable since the rightmost term on Eq. ( 19 ) gets closer to zero.

Panel of Figure 4 — Figure 4: Reliability and cost per episode of two commodities in the edge network for reliability targets $\delta^{1}=0.7$ , $\delta^{2}=0.6$ (marked with horizontal dashed black line) and initial lifetimes $L^{1}=4$ . The arrival rates are kept the same for both commodities. Even though all approaches are able to stay above the reliability targets, CDRL-NC is able to perform timely packet delivery with less cost than both BP and UMW. When $\bar{b}^{1}=\bar{b}^{2}=10$ , CDRL-NC is able to satisfy the reliability targets while BP fails to meet the reliability target for commodity 1.

Online Resource Allocation

Online resource allocation studies how to assign scarce capacity to arriving requests under uncertainty while balancing reward, cost, fairness, and feasibility.

See all 95 papers

From Prior-Guided Heuristics to Deployable Agents: Accelerating Demonstration-Driven Reinforcement Learning for Deadline-Constrained Network Control
Sep 3, 2026Vincenzo Norman Vitale, Mohammad Solki, Antonia Maria Tulino +2Learning from Demonstration Multi-Agent Reinforcement Learning
DIETI, University of Naples Federico II, Via Claudio 21, Naples, 80125, Italy · New York University, 6 MetroTech Center, Brooklyn, New York, 11201, USA · University of Southern California, 3740 McClintock Ave., Los Angeles, 90089, California, USA +2
Beyond Peak Backlog: Conditional Energy and Temporal Geometry in Capacity-Constrained Delayed Bandit Optimization
Aug 17, 2026Anling Xiang, Yuwen Yang, Yang ShenBandits with Delayed Feedback Bandit Convex Optimization
Department of Intelligent Communication, School of Journalism and Communication, Minzu University of China, Beijing, China · ZeeLin (Beijing) Technology Co., Ltd., Beijing, China · School of Journalism and Communication, Tsinghua University, Beijing, China +1
NetRL: Powering Network Decisions via NetGRPO-Driven Reinforcement Learning
Oct 12, 2026Lingfeng Peng, Xiaohui Xie, Yang Bao +1RL Control RL for LLM Agents
Tsinghua University

Sep 3, 2026J/K move · Enter open · S save

From Prior-Guided Heuristics to Deployable Agents: Accelerating Demonstration-Driven Reinforcement Learning for Deadline-Constrained Network Control

Vincenzo Norman Vitale, Mohammad Solki, Antonia Maria Tulino, Andreas F. Molisch+1

DIETI, University of Naples Federico II, Via Claudio 21, Naples, 80125, Italy · New York University, 6 MetroTech Center, Brooklyn, New York, 11201, USA · University of Southern California, 3740 McClintock Ave., Los Angeles, 90089, California, USA+2

Learning from Demonstration Multi-Agent Reinforcement Learning Offline-To-Online RL

Open paper AskPDF arXiv

Timely delivery of delay-sensitive information over dynamic, heterogeneous networks is essential for NextG interactive applications, yet providing strict End-to-End (E2E) peak latency guarantees remains an open challenge. Two obstacles limit the adoption of learning-based network control in this setting: traditional volume-based routing metrics, while highly effective for general traffic management, are not designed to capture traffic urgency; and Deep Reinforcement Learning (DRL) controllers trained from scratch suffer from sample inefficiency, long training times, and early-stage exploration volatility. This paper introduces a deployment-focused network control framework that addresses both obstacles. First, we present Effective Congestion (EC), a deadline-aware metric family that quantifies interface congestion by packet urgency and proactively filters non-viable traffic, coupled with a Uniform Path Grouping (UPG) distribution heuristic promoting robust load-balancing; the resulting policies are embedded into Multi-Agent Deep Reinforcement Learning Effective Congestion (

p^*

) (MADRL EC (

p^*

)), a hybrid architecture combining a distributed scheduler with a centralized RL-based router. Second, we introduce a unified training objective that generalizes existing policy-learning paradigms---behavioral cloning, offline Reinforcement Learning (RL), online RL, and offline-to-online schemes---as special cases, combining a live-reward term, a pre-collected-reward term, and a policy-imitation term. From this objective, we derive the Model-Guided Annealed Reinforcement Learning (MGA-RL) protocol, instantiated on a Deep Deterministic Policy Gradient (DDPG) backbone: a deployment-oriented, demonstration-driven training approach that generalizes conventional Offline-to-Online (O2O) schemes, in which trajectories from a lightweight [...]

Abstract

Online Resource Allocation

Explore similar work

From Prior-Guided Heuristics to Deployable Agents: Accelerating Demonstration-Driven Reinforcement Learning for Deadline-Constrained Network Control

Beyond Peak Backlog: Conditional Energy and Temporal Geometry in Capacity-Constrained Delayed Bandit Optimization

NetRL: Powering Network Decisions via NetGRPO-Driven Reinforcement Learning