A physics-constrained deep reinforcement learning method for autonomous mission planning of Titan deep space probes
Authors: Hanmin Li, Tao Wang, Shuangliang Liu, Zhe Zhang, Maodeng Li
Organizations: School of Intelligent Systems Engineering, Sun Yat-sen University, Shenzhen, 518107, China · Deep Space Exploration Laboratory, Hefei, 230031, China
Titan is a compelling target for future deep space exploration, while its extreme Earth-Titan distance and complex environment make continuous ground-in-the-loop planning impractical. Existing planning methods, primarily developed for Earth-observation scenarios, have limited generalizability to uncertain deep space conditions and adaptability to time-varying mission states. To address these challenges, this paper proposes a physics-constrained maskable proximal policy optimization (PCM-PPO) method for Titan autonomous mission planning. A Titan-specific discrete-time Markov decision process is established to characterize mission feasibility and resource constraints. A system dynamics and geometric model is incorporated to enhance physical representation, including central gravity, J2 perturbation, third-body attraction, and downlink occultation. To accommodate opportunity targets under a bounded action dimension, PCM-PPO organizes predefined and opportunity targets through a fixed-capacity target pool and represents the mission state with a target-wise state matrix and a global state vector. A physics-constrained action mask constructs an admissible decision domain from current mission states, guiding policy learning toward physically executable decisions and avoiding infeasible exploration. An integrated training strategy facilitates policy optimization under coupled constraints. Comparative, ablation, and uncertainty experiments validate the effectiveness of PCM-PPO. PCM-PPO provides a feasible and interpretable scheme for autonomous mission planning of Titan deep space probes.
Figures & tables
Fig. 1: Multi-body dynamics and geometric constraints in the Titan mission environment.
Fig. 2: Markov decision process formulation for Titan mission planning.
Aadm(sk)={aj∈A∣mj,k=1}
Algorithm 1 Physics-constrained action mask construction
Fig. 3: Overall framework of the PCM-PPO method.
Formulation
Dynamic targets
Fixed action size
Feasible action constraint
Coupled constraints
Uncertainty response
Low online cost
Standard PPO
△
✓
×
△
✓
✓
Safe / Constrained RL
△
✓
✓
△
✓
✓
Receding-horizon optimization
✓
△
✓
✓
✓
△
Metaheuristic scheduling
△
△
×
△
△
×
PCM-PPO
✓
✓
✓
✓
✓
✓
✓ : supported; △ : supported only with additional problem-specific design or repeated re-planning; × : not naturally supported.
Table 1: Functional comparison of representative planning and reinforcement-learning formulations for Titan autonomous mission planning.
Fig. 4: Schematic illustration of probe attitude maneuvers for targets in different maneuver-demand bands
Parameter
Symbol
Value
Titan gravitational parameter
μT
8978.138km3/s2
Titan mean radius
RT
2575.0km
Saturn mean radius
RS
60268km
Titan zonal harmonic coefficient
J2,T
31.81×10−6
Titan rotation rate
ωT
4.560×10−6rad/s
Saturn gravitational parameter
μS
3.79312×107km3/s2
Table 2: Physical and simulation parameters of the Titan mission environment.
Hyperparameter
Symbol / Setting
Value
Target feature dimension
dx
8
Global state dimension
dg
7
Target encoder hidden layers
gψ
[256,256]
Target embedding dimension
hi,k
256
Context network hidden layers
ρ
[256,256]
Mission context dimension
ck
256
Table 3: Training hyperparameters, network settings, and reward coefficients of the PCM-PPO model.
Item
Value
Trainable parameters
Full actor–critic: 0.335–0.434 M; actor-only deployment path: 0.170–0.224 M
HRH=36 steps; Δt=60 s; time limit =0.25 s; relative MIP gap =0.01 ; horizon discount =0.995
Explicit
SA
Metaheuristic
Weight vector, 16-D
2 restarts; 80 iter./restart;
Initial temperature =2.5 ; minimum temperature =0.05 ; cooling rate =0.96 ; proposal scale =0.9 ; 6 eval. eps./candidate
Repair / penalty
Table 5: Baseline configurations used in the comparative experiments.
Target load
Method
Target completion rate (mean ± std)
Cumulative reward (mean ± std)
95% CI of reward
20
PCM-PPO
0.705 ± 0.026
6217.5 ± 185.6
[6130.6, 6304.4]
PB-PPO
0.593 ± 0.029
5428.4 ± 224.7
[5323.2, 5533.6]
RH-MILP
0.594 ± 0.027
5450.6 ± 206.8
[5353.8, 5547.4]
PPO
0.586 ± 0.035
5365.9 ± 243.1
[5252.1, 5479.7]
AC
0.569 ± 0.033
4927.2 ± 205.6
[4831.0, 5023.4]
DQN
0.539 ± 0.032
4622.2 ± 211.3
[4523.3, 4721.1]
Table 6: Quantitative comparison of different methods under varying target loads over a 10-day evaluation horizon. Both target completion rate and cumulative reward are reported as mean ± standard deviation over 20 independent Monte Carlo trials, and the 95% confidence interval is reported for the mean cumulative reward.
Algorithm
Targets 20
Targets 30
Targets 40
Targets 50
PCM-PPO
0.802
0.758
0.704
0.660
PB-PPO
0.764
0.717
0.668
0.608
RH-MILP
0.769
0.701
0.645
0.589
PPO
0.718
0.682
0.642
0.593
AC
0.693
0.655
0.614
0.567
Table 7: Opportunity-target completion rate of representative methods over the 10-day evaluation horizon.
Fig. 7: Average daily reward comparison under different daily target loads over the 10-day evaluation horizon.
Fig. 8: Box plots of average daily rewards over 20 independent Monte Carlo trials under the 10-day evaluation horizon.
Target load
Algorithm
Target completion rate (mean ± std)
Cumulative reward (mean ± std)
95% CI of reward
20
PCM-PPO
0.671 ± 0.011
58480.6 ± 965.4
[58028.8, 58932.4]
PB-PPO
0.584 ± 0.013
51986.7 ± 1056.4
[51492.3, 52481.1]
RH-MILP
0.575 ± 0.012
51320.8 ± 1034.6
[50836.6, 51805.0]
PPO
0.572 ± 0.014
50642.3 ± 1158.9
[50099.9, 51184.7]
AC
0.564 ± 0.013
47170.9 ± 1048.5
[46680.2, 47661.6]
DQN
0.532 ± 0.013
45982.1 ± 1015.6
[45506.8, 46457.4]
Table 8: Quantitative comparison of different methods under varying target loads over a 100-day evaluation horizon. Both target completion rate and cumulative reward are reported as mean ± standard deviation over 20 independent Monte Carlo trials, and the 95% confidence interval is reported for the mean cumulative reward.
Fig. 9: Average daily reward comparison under different daily target loads over the 100-day evaluation horizon.
Fig. 10: Box plots of average daily rewards over 20 independent Monte Carlo trials under the 100-day evaluation horizon.
Method
Training seeds
Cumulative reward mean ± std
Target completion rate mean ± std
PCM-PPO
5
81272.6 ± 686.3
0.619 ± 0.009
PB-PPO
5
74591.8 ± 842.7
0.558 ± 0.010
PPO
5
71584.4 ± 976.5
0.541 ± 0.012
Table 9: Training-seed stability of PPO-based methods under target load 30 over the 100-day evaluation horizon.
Fig. 11: Average daily reward comparison under different ablation settings over the 10-day evaluation horizon.
Fig. 12: Box plots of average daily rewards under different ablation settings over the 10-day evaluation horizon.
Fig. 13: Average daily reward comparison under different ablation settings over the 100-day evaluation horizon.
Fig. 14: Box plots of average daily rewards under different ablation settings over the 100-day evaluation horizon.
Fig. 15: Average daily reward comparison under different training strategy settings at daily target load 50.
Reward setting
Nominal avg. reward
Target completion rate
Downlink ratio
Avg. peak storage occupancy
Min. energy margin (Wh)
Nominal
1324.5
0.607
0.821
0.86
246
λobs−20%
1289.8
0.590
0.836
0.81
269
λobs+20%
1336.2
0.617
0.803
0.92
224
λdl−20%
1328.7
0.611
0.769
0.93
235
λdl+20%
1316.8
0.598
0.861
0.77
244
(λE,λS)−20%
1331.4
0.613
0.814
0.93
210
Table 10: Planning-quality sensitivity of PCM-PPO to reward-coefficient perturbations at daily target load 50 over the 10-day evaluation horizon.
Method variant
Raw-policy infeasible ratio
Final selected infeasible ratio
PCM-PPO
0.107
0.000
Maskable PPO
0.123
0.000
w/o Phys. mask
0.052
0.052
vanilla PPO
0.078
0.078
Table 11: Infeasible-action ratios relative to the modeled admissible action set before and after masking at a daily target load of 50 over the 10-day evaluation horizon.
Receding-horizon length, HRH
Time limit
Relative
Cumulative
Planning time
(s)
MIP gap
reward
(ms)
18
0.25
1%
10534.8
2.17
36
0.25
1%
10690.2
4.79
72
0.25
1%
10738.5
11.60
36
1.00
1%
10722.8
6.35
36
0.25
0.1%
10701.4
4.68
Table 12: RH-MILP configuration sensitivity at a daily target load of 50.
Task ID
Maneuver demand
Reward level
PCM-PPO
Maskable PPO
PPO
4
Low
Opportunity
Success
Success
Success
9
Low
Opportunity
Success
Success
Success
27
Low
Opportunity
Success
Success
Success
11
Medium
Opportunity
Success
Success
Success
5
Low
High
Success
Success
Success
10
Low
High
Success
Success
Success
Table 13: Illustrative observation results of PCM-PPO, Maskable PPO, and PPO under Scenario 1.
Fig. 16: Average daily reward comparison under energy budget degradation in Scenario 2.
Task ID
Maneuver demand
Reward level
PCM-PPO
Maskable PPO
PPO
9
Low
Opportunity
Success
Success
Success
27
Low
Opportunity
Success
Success
Success
11
Medium
Opportunity
Success
Success
Success
5
Low
High
Success
Success
Success
10
Low
High
Success
Success
Success
14
Low
High
Success
Success
Success
Table 14: Illustrative observation results under Scenario 2 with an 80% energy budget.
Task ID
Maneuver demand
Reward level
PCM-PPO
Maskable PPO
PPO
9
Low
Opportunity
Success
Success
Success
27
Low
Opportunity
Success
Success
Success
5
Low
High
Success
Success
Success
10
Low
High
Success
Success
Success
14
Low
High
Success
Success
Success
21
Low
High
Success
Success
Success
Table 15: Illustrative observation results under Scenario 2 with a 60% energy budget.
Fig. 17: Average daily reward comparison under communication degradation in Scenario 3.
Task ID
Maneuver demand
Reward level
PCM-PPO
Maskable PPO
PPO
9
Low
Opportunity
Success
Success
Success
27
Low
Opportunity
Success
Success
Success
11
Medium
Opportunity
Success
Success
Success
5
Low
High
Success
Success
Success
10
Low
High
Success
Success
Success
14
Low
High
Success
Success
Success
Table 16: Illustrative observation results under Scenario 3 with a 6 h DSN contact window.
Task ID
Maneuver demand
Reward level
PCM-PPO
Maskable PPO
PPO
9
Low
Opportunity
Success
Success
Success
27
Low
Opportunity
Success
Success
Success
5
Low
High
Success
Success
Success
10
Low
High
Success
Success
Success
14
Low
High
Success
Success
Success
21
Low
High
Success
Success
Success
Table 17: Illustrative observation results under Scenario 3 with a 4 h DSN contact window.
Fig. 18: Average daily reward comparison under coupled energy-communication degradation in Scenario 4.
Task ID
Maneuver demand
Reward level
PCM-PPO
Maskable PPO
PPO
9
Low
Opportunity
Success
Success
Success
27
Low
Opportunity
Success
Success
Success
11
Medium
Opportunity
Success
Success
Success
5
Low
High
Success
Success
Success
10
Low
High
Success
Success
Success
14
Low
High
Success
Success
Success
Table 18: Illustrative observation results under Scenario 4 with an 80% energy budget and a 6 h DSN contact window.
Task ID
Maneuver demand
Reward level
PCM-PPO
Maskable PPO
PPO
9
Low
Opportunity
Success
Success
Success
27
Low
Opportunity
Success
Success
Success
5
Low
High
Success
Success
Success
10
Low
High
Success
Success
Success
14
Low
High
Success
Success
Success
21
Low
High
Success
Success
Success
Table 19: Illustrative observation results under Scenario 4 with a 60% energy budget and a 4 h DSN contact window.
Method
Mean daily reward (95% CI)
Mean opportunity-target completion rate
PCM-PPO
872.7±12.4
0.771
Maskable PPO
751.1±14.9
0.574
PPO
726.1±16.8
0.548
Table 20: Monte Carlo aggregate results under Scenario 5.
Task ID
Maneuver demand
Reward level
PCM-PPO
Maskable PPO
PPO
9
Low
Opportunity
Success
Success
Success
27
Low
Opportunity
Success
Success
Success
11
Medium
Opportunity
Success
Success
Success
2
Medium
Opportunity
Success
Success
Success
18
Low
Opportunity
Success
Success
Success
5
Low
High
Success
Success
Success
Table 21: Illustrative observation results under Scenario 5 with target-priority variation.
Spacecraft operations scheduling is a highly constrained, long-horizon combinatorial optimization problem that traditionally relies on heuristics, constraint programming, or manual planning. We present a scalable deep reinforcement learning framework developed and deployed for NASA's Carruthers Geocorona Observatory mission. Our framework introduces a macro-action abstraction known as activity blocks coupled with dynamic action-masking to navigate the intractably large search space and strictly enforce complex power, thermal, and instrument constraints. The resulting architecture generates globally feasible schedules with overwhelming probability, establishes operational trust, and executes a full training cycle in under six hours, circumventing the need for policy robustness by enabling rapid, on-demand retraining. Further, resulting schedules outperform baseline heuristics in scheduled science quality. The deep reinforcement learning framework was deployed as the default operational scheduler for the Carruthers Geocorona Observatory mission from the outset of the mission, demonstrating that deep reinforcement learning can be trusted for real spacecraft operations under complex, evolving constraints.
Alex Zhang, Jackson Craig, Lara Waldrop
Department of Electrical and Computer Engineering, University of Illinois Urbana-Champaign, IL, USA
The deployment of reinforcement learning (RL)-based controllers on physical systems is often limited by poor generalization to real-world scenarios, known as the simulation-to-reality (sim-to-real) gap. This gap is particularly challenging in spaceflight, where real-world training data are scarce due to high cost and limited planetary exploration data. Traditional approaches, such as system identification and synthetic data generation, depend on sufficient data and often fail due to modeling assumptions or lack of physics-based constraints. We propose addressing this data scarcity by introducing physics-based learning bias in a generative model. Specifically, we develop the Mutual Information-based Split Variational Autoencoder (MI-VAE), a physics-informed VAE that learns differences between observed system trajectories and those predicted by physics-based models. The latent space of the MI-VAE enables generation of synthetic datasets that respect physical constraints. We evaluate MI-VAE on a planetary lander problem, focusing on limited real-world data and offline RL training. Results show that augmenting datasets with MI-VAE samples significantly improves downstream RL performance, outperforming standard VAEs in statistical fidelity, sample diversity, and policy success rate. This work demonstrates a scalable strategy for enhancing autonomous controller robustness in complex, data-constrained environments.
Alex E. Ballentine, Nachiket U. Bapat, Raghvendra V. Cowlagi
Graduate Research Assistant, Aerospace Engineering Department, Worcester Polytechnic Institute, MA. · Associate Professor, Aerospace Engineering Department. AIAA Senior Member. Corresponding Author.
Autonomous inspection of a target spacecraft is typically the initial step in on-orbit servicing. Planning close-proximity observation viewpoints involves a complex combinatorial problem constrained by coverage, fuel, and safety considerations. This paper models the viewpoint selection as a Constrained Markov Decision Process (CMDP) and applies Proximal Policy Optimization with Lagrangian relaxation to solve it. The system combines a high-level planner, which sequences camera viewpoints on an observation sphere around the target, with a low-level model that assesses each transfer using the Clohessy--Wiltshire (CW) equations for relative motion. The deterministic constraints on single-impulse reachability and Δv cap are enforced directly on the action set. Safety margins are affected by navigation and manoeuvre errors, making the actual clearance stochastic. To address this, a chance constraint on the realised trajectory is implemented, with a Lagrangian multiplier governed by PID control to adapt the penalty during training. The approach is tested on a Galileo-class navigation satellite in medium Earth orbit, modelled as a point cloud where visibility from each viewpoint considers field-of-view and line-of-sight occlusion. Across various risk budgets, the trained policy maintains the keep-out violation rate within the specified limit, achieving coverage close to its maximum and staying within the mission Δv budget. Compared to the usual practice of applying a fixed margin to the keep-out radius and planning deterministically, the chance-constrained policy attains less than half the violation rate at comparable coverage. This makes the collision risk an explicit design parameter, eliminating the need for re-tuning per case. The learned policy selects a viewpoint without the repeated Monte Carlo risk estimates required by a greedy, chance-constrained planner at every decision point.