Organizations: Communication Networks Research Group (CNG), Dept. of Electrical and Electronics Engineering, Middle East Technical University (METU), Ankara, Türkiye
Deep-space communications operate under intermittent connectivity, long propagation delays, and relay storage constraints. In such networks, the Bundle Protocol enables store-carry-forward communication, requiring relay nodes to manage limited storage and scarce contact opportunities. Relay-side scheduling and dropping decisions are inherently goal-oriented: transmission resources should be allocated to bundles most valuable to downstream mission objectives. Conventional schedulers, such as FIFO, do not explicitly optimize task utility and may provide unfair service across heterogeneous planetary sources. This paper presents a closed-loop modeling and learning framework for relay-side bundle and flow management in a contact-driven deep-space network, where lunar and Martian updates are forwarded through a shared-buffer relay to Earth. We quantify destination-side information freshness using excess age Δie(t)=[Δi(t)−di(t)]+, where Δi(t) denotes the Age of Information of source i and di(t) is a source-dependent delay baseline. Based on this metric, we define a goal-oriented loss that supports source-dependent nonlinear penalties, allowing prioritization according to mission relevance. We formulate relay control as a constrained semi-Markov decision process (CSMDP) and solve it using masked Lagrangian Proximal Policy Optimization (PPO). Feasibility-aware action masking restricts transmissions to bundles that can complete within the remaining contact window. DTNSim simulations indicate improved age-based utility and inter-flow fairness compared with baseline schedulers. The proposed framework jointly optimizes relay-side bundle scheduling, dropping, and flow control under contact-driven dynamics with explicit fairness considerations across heterogeneous sources.
Figures & tables
Figure 1: Earth–Relay–Moon–Mars network.
Figure 2: Masked Lagrangian PPO architecture for event-driven relay control.
Parameter
Setting
Episode interval
t0=1800 s, T=16200 s; H=14400 s
Traffic generation
Independent Poisson; mean interarrival: Moon 180 s, Mars 360 s
Bundle sizes
20/100/300 Mbit with probabilities 0.5/0.3/0.2
Bundle TTL
5400 s from original generation
Relay capacity Bmax
1.2×109 bits
Contact cycle Tc
1800 s
Table 1: Frozen evaluation and PPO training configuration.
Policy
Aggregate ↓
Moon ↓
Mars ↓
Masked Lag. PPO
887.63
216.64
670.98
FIFO
1747.87
686.89
1060.97
LIFO
993.57
232.83
760.74
EDF
1692.38
806.15
886.23
Freshest-first
1009.56
232.82
776.74
Table 2: Excess-AoI performance over the 64 held-out traces. Values are means in seconds; lower is better.
Policy
Drops ↓
Exp. ↓
Delivered
Delivered Gbit
Bundle DR
Bit DR
Ucontact
Masked Lag. PPO
10.02
0.05
103.72
8.086
0.860
0.686
0.845
FIFO
16.20
0.00
96.28
8.578
0.801
0.727
0.897
LIFO
17.88
0.20
95.42
8.286
0.794
0.703
0.865
EDF
16.91
0.00
95.61
8.577
0.796
0.727
0.896
Freshest-first
16.78
0.17
96.56
8.262
0.803
0.701
0.863
Table 3: Mean operational performance over 64 traces. Expiries count relay-side events used in the constraint.
Figure 3: Relay operation under the PPO policy.
Policy
JCdrop
JCexp
JCfair,M
JCfair,S
Mean-feasible
Budget
17
1
0
0
—
Masked Lag. PPO
10.02
0.05
−2007.89
−1693.17
Yes
FIFO
16.20
0.00
−1764.52
−1138.05
Yes
LIFO
17.88
0.20
−2010.25
−1659.19
No
EDF
16.91
0.00
−1592.67
−1452.85
Yes
Freshest-first
16.78
0.17
−2010.25
−1658.08
Yes
Table 4: Mean episodic constraint returns over the 64 held-out traces. A policy is mean-feasible when all four empirical means satisfy their corresponding budgets.
Spacecraft operations scheduling is a highly constrained, long-horizon combinatorial optimization problem that traditionally relies on heuristics, constraint programming, or manual planning. We present a scalable deep reinforcement learning framework developed and deployed for NASA's Carruthers Geocorona Observatory mission. Our framework introduces a macro-action abstraction known as activity blocks coupled with dynamic action-masking to navigate the intractably large search space and strictly enforce complex power, thermal, and instrument constraints. The resulting architecture generates globally feasible schedules with overwhelming probability, establishes operational trust, and executes a full training cycle in under six hours, circumventing the need for policy robustness by enabling rapid, on-demand retraining. Further, resulting schedules outperform baseline heuristics in scheduled science quality. The deep reinforcement learning framework was deployed as the default operational scheduler for the Carruthers Geocorona Observatory mission from the outset of the mission, demonstrating that deep reinforcement learning can be trusted for real spacecraft operations under complex, evolving constraints.
Alex Zhang, Jackson Craig, Lara Waldrop
Department of Electrical and Computer Engineering, University of Illinois Urbana-Champaign, IL, USA
In this paper, we propose a dueling double deep Q-network (DDQN)-based adaptive multi-objective handover framework for low Earth orbit (LEO) satellite networks. The proposed method enables dynamic trade-off learning among throughput, blocking probability, and switching cost under time-varying network conditions. Simulation results demonstrate that the proposed approach consistently outperforms conventional baselines, achieving up to 10.3% throughput improvement and near-zero blocking under typical operating conditions.
Po-Heng Chou, Chiapin Wang, Chung-Chi Huang +1
Research Center for Information Technology Innovation (CITI), Academia Sinica (AS), Taipei 11529, Taiwan · Department of Electrical Engineering, National Taiwan Normal University (NTNU), Taipei 106308 Taiwan
Maximizing throughput under proportional fairness in dense wireless networks requires jointly managing user association, scheduling, base station (BS) activation, and handover control under hard finite-horizon energy and handover budgets, which induces a fundamental tension between BS-side energy management and user-side handover regulation. While multi-agent reinforcement learning (MARL) is a natural framework for such distributed sequential control, its application here faces two difficulties: finite-horizon budget constraints cannot be evaluated at each time slot, and the nonlinear proportional fairness utility admits no principled per-slot decomposition. We propose HeLyMARL, a Lyapunov-embedded heterogeneous MARL framework that resolves both via drift-plus-penalty decomposition with virtual queues. The energy and handover constraint pressures are internalized directly into a unified per-slot reward, converting the constrained finite-horizon problem into an unconstrained MARL problem. Comparison against two Lagrangian-based alternatives reveals a timescale separation: Lagrangian relaxation regulates constraints only across training episodes, whereas the virtual queues of HeLyMARL bound cumulative budget consumption at every partial horizon within an episode, a pacing guarantee beyond the reach of greedy Lyapunov-based control. Simulations show that HeLyMARL is the only method that sustains the throughput-fairness balance together with uninterrupted service throughout the horizon, outperforming conventional MARL, Lyapunov-based, and constrained MARL benchmarks without premature budget exhaustion.
Yeonseo Jeong, Wonhyeok Ko, Sungweon Hong +1
Department of Electronic Engineering, Hanyang University, Seoul, Korea · Department of AI-Semiconductor, Hanyang University, Seoul, Korea