Goal-Oriented Bundle Management and Flow Control for Deep-Space Communications via Reinforcement Learning

Sajjad Baghaeea; Aimin Lib; Shanmugarajan Balasubramanian; Elif Uysal

cs.NIOct 9, 2026

Goal-Oriented Bundle Management and Flow Control for Deep-Space Communications via Reinforcement Learning

Authors: Sajjad Baghaeea, Aimin Lib, Shanmugarajan Balasubramanian, Elif Uysal

Organizations: Communication Networks Research Group (CNG), Dept. of Electrical and Electronics Engineering, Middle East Technical University (METU), Ankara, Türkiye

Abstract

Deep-space communications operate under intermittent connectivity, long propagation delays, and relay storage constraints. In such networks, the Bundle Protocol enables store-carry-forward communication, requiring relay nodes to manage limited storage and scarce contact opportunities. Relay-side scheduling and dropping decisions are inherently goal-oriented: transmission resources should be allocated to bundles most valuable to downstream mission objectives. Conventional schedulers, such as FIFO, do not explicitly optimize task utility and may provide unfair service across heterogeneous planetary sources. This paper presents a closed-loop modeling and learning framework for relay-side bundle and flow management in a contact-driven deep-space network, where lunar and Martian updates are forwarded through a shared-buffer relay to Earth. We quantify destination-side information freshness using excess age $Δ_i^e(t)=[Δ_i(t)-d_i(t)]^+$ , where $Δ_i(t)$ denotes the Age of Information of source $i$ and $d_i(t)$ is a source-dependent delay baseline. Based on this metric, we define a goal-oriented loss that supports source-dependent nonlinear penalties, allowing prioritization according to mission relevance. We formulate relay control as a constrained semi-Markov decision process (CSMDP) and solve it using masked Lagrangian Proximal Policy Optimization (PPO). Feasibility-aware action masking restricts transmissions to bundles that can complete within the remaining contact window. DTNSim simulations indicate improved age-based utility and inter-flow fairness compared with baseline schedulers. The proposed framework jointly optimizes relay-side bundle scheduling, dropping, and flow control under contact-driven dynamics with explicit fairness considerations across heterogeneous sources.

Figures & tables

Figure 1: Earth–Relay–Moon–Mars network.

Figure 2: Masked Lagrangian PPO architecture for event-driven relay control.

Parameter	Setting
Episode interval	$t_{0}=1800$ s, $T=16200$ s; $H=14400$ s
Traffic generation	Independent Poisson; mean interarrival: Moon $180$ s, Mars $360$ s
Bundle sizes	$20/100/300$ Mbit with probabilities $0.5/0.3/0.2$
Bundle TTL	$5400$ s from original generation
Relay capacity $B_{\max}$	$1.2\times 10^{9}$ bits
Contact cycle $T_{c}$	$1800$ s

Table 1: Frozen evaluation and PPO training configuration.

Policy	Aggregate $\downarrow$	Moon $\downarrow$	Mars $\downarrow$
Masked Lag. PPO	$\mathbf{887.63}$	$\mathbf{216.64}$	$\mathbf{670.98}$
FIFO	$1747.87$	$686.89$	$1060.97$
LIFO	$993.57$	$232.83$	$760.74$
EDF	$1692.38$	$806.15$	$886.23$
Freshest-first	$1009.56$	$232.82$	$776.74$

Table 2: Excess-AoI performance over the

64

held-out traces. Values are means in seconds; lower is better.

Policy	Drops $\downarrow$	Exp. $\downarrow$	Delivered	Delivered Gbit	Bundle DR	Bit DR	$U_{\mathrm{contact}}$
Masked Lag. PPO	10.02	0.05	103.72	8.086	0.860	0.686	0.845
FIFO	16.20	0.00	96.28	8.578	0.801	0.727	0.897
LIFO	17.88	0.20	95.42	8.286	0.794	0.703	0.865
EDF	16.91	0.00	95.61	8.577	0.796	0.727	0.896
Freshest-first	16.78	0.17	96.56	8.262	0.803	0.701	0.863

Table 3: Mean operational performance over

64

traces. Expiries count relay-side events used in the constraint.

Figure 3: Relay operation under the PPO policy.

Policy	$\widehat{J}_{C_{\mathrm{drop}}}$	$\widehat{J}_{C_{\mathrm{exp}}}$	$\widehat{J}_{C_{\mathrm{fair},M}}$	$\widehat{J}_{C_{\mathrm{fair},S}}$	Mean-feasible
Budget	$17$	$1$	$0$	$0$	—
Masked Lag. PPO	10.02	0.05	$-2007.89$	$-1693.17$	Yes
FIFO	16.20	0.00	$-1764.52$	$-1138.05$	Yes
LIFO	17.88	0.20	$-2010.25$	$-1659.19$	No
EDF	16.91	0.00	$-1592.67$	$-1452.85$	Yes
Freshest-first	16.78	0.17	$-2010.25$	$-1658.08$	Yes

Table 4: Mean episodic constraint returns over the

64

held-out traces. A policy is mean-feasible when all four empirical means satisfy their corresponding budgets.

Explore similar work

Deep RL for Fast Long-Horizon Operations Scheduling on NASA's Carruthers Geocorona Observatory Mission
Jun 20, 2026Alex Zhang, Jackson Craig, Lara WaldropReinforcement Learning Long-Horizon Planning
Department of Electrical and Computer Engineering, University of Illinois Urbana-Champaign, IL, USA
Dueling DDQN-Based Adaptive Multi-Objective Handover Optimization for LEO Satellite Networks
May 4, 2026Po-Heng Chou, Chiapin Wang, Chung-Chi Huang +1Multi-Objective Reinforcement Learning Deep Q-Learning
Research Center for Information Technology Innovation (CITI), Academia Sinica (AS), Taipei 11529, Taiwan · Department of Electrical Engineering, National Taiwan Normal University (NTNU), Taipei 106308 Taiwan
Heterogeneous Multi-Agent Reinforcement Learning for Radio Resource Management under Coupled Finite-Horizon Constraints
Aug 3, 2026Yeonseo Jeong, Wonhyeok Ko, Sungweon Hong +1Stochastic Optimization Fair Resource Allocation
Department of Electronic Engineering, Hanyang University, Seoul, Korea · Department of AI-Semiconductor, Hanyang University, Seoul, Korea

Jun 20, 2026, astro-ph.IMJ/K move · Enter open · S save

Deep RL for Fast Long-Horizon Operations Scheduling on NASA's Carruthers Geocorona Observatory Mission

Alex Zhang, Jackson Craig, Lara Waldrop

Department of Electrical and Computer Engineering, University of Illinois Urbana-Champaign, IL, USA

Reinforcement Learning Long-Horizon Planning Constrained RL

Open paper AskPDF arXiv

Spacecraft operations scheduling is a highly constrained, long-horizon combinatorial optimization problem that traditionally relies on heuristics, constraint programming, or manual planning. We present a scalable deep reinforcement learning framework developed and deployed for NASA's Carruthers Geocorona Observatory mission. Our framework introduces a macro-action abstraction known as activity blocks coupled with dynamic action-masking to navigate the intractably large search space and strictly enforce complex power, thermal, and instrument constraints. The resulting architecture generates globally feasible schedules with overwhelming probability, establishes operational trust, and executes a full training cycle in under six hours, circumventing the need for policy robustness by enabling rapid, on-demand retraining. Further, resulting schedules outperform baseline heuristics in scheduled science quality. The deep reinforcement learning framework was deployed as the default operational scheduler for the Carruthers Geocorona Observatory mission from the outset of the mission, demonstrating that deep reinforcement learning can be trusted for real spacecraft operations under complex, evolving constraints.

Abstract

Explore similar work

Deep RL for Fast Long-Horizon Operations Scheduling on NASA's Carruthers Geocorona Observatory Mission

Dueling DDQN-Based Adaptive Multi-Objective Handover Optimization for LEO Satellite Networks

Heterogeneous Multi-Agent Reinforcement Learning for Radio Resource Management under Coupled Finite-Horizon Constraints