A physics-constrained deep reinforcement learning method for autonomous mission planning of Titan deep space probes
Organizations: School of Intelligent Systems Engineering, Sun Yat-sen University, Shenzhen, 518107, China · Deep Space Exploration Laboratory, Hefei, 230031, China
Abstract
Titan is a compelling target for future deep space exploration, while its extreme Earth-Titan distance and complex environment make continuous ground-in-the-loop planning impractical. Existing planning methods, primarily developed for Earth-observation scenarios, have limited generalizability to uncertain deep space conditions and adaptability to time-varying mission states. To address these challenges, this paper proposes a physics-constrained maskable proximal policy optimization (PCM-PPO) method for Titan autonomous mission planning. A Titan-specific discrete-time Markov decision process is established to characterize mission feasibility and resource constraints. A system dynamics and geometric model is incorporated to enhance physical representation, including central gravity, J2 perturbation, third-body attraction, and downlink occultation. To accommodate opportunity targets under a bounded action dimension, PCM-PPO organizes predefined and opportunity targets through a fixed-capacity target pool and represents the mission state with a target-wise state matrix and a global state vector. A physics-constrained action mask constructs an admissible decision domain from current mission states, guiding policy learning toward physically executable decisions and avoiding infeasible exploration. An integrated training strategy facilitates policy optimization under coupled constraints. Comparative, ablation, and uncertainty experiments validate the effectiveness of PCM-PPO. PCM-PPO provides a feasible and interpretable scheme for autonomous mission planning of Titan deep space probes.
Figures & tables
| Formulation | Dynamic targets | Fixed action size | Feasible action constraint | Coupled constraints | Uncertainty response | Low online cost |
| Standard PPO | ||||||
| Safe / Constrained RL | ||||||
| Receding-horizon optimization | ||||||
| Metaheuristic scheduling | ||||||
| PCM-PPO | ||||||
| : supported; : supported only with additional problem-specific design or repeated re-planning; : not naturally supported. | ||||||
| Parameter | Symbol | Value |
| Titan gravitational parameter | ||
| Titan mean radius | ||
| Saturn mean radius | ||
| Titan zonal harmonic coefficient | ||
| Titan rotation rate | ||
| Saturn gravitational parameter |
| Hyperparameter | Symbol / Setting | Value |
| Target feature dimension | ||
| Global state dimension | ||
| Target encoder hidden layers | ||
| Target embedding dimension | ||
| Context network hidden layers | ||
| Mission context dimension |
| Item | Value |
| Trainable parameters | Full actor–critic: 0.335–0.434 M; actor-only deployment path: 0.170–0.224 M |
| Model size, FP32 / FP16 | Full model: 1.28–1.66 MB / 0.64–0.83 MB; actor-only path: 0.65–0.85 MB / 0.32–0.43 MB |
| Mask construction time | 0.92–2.01 ms mean; 2.52 ms at |
| Actor inference time | 0.39–0.42 ms mean; 0.55 ms at |
| Total online decision time | 1.32–2.43 ms mean; 2.97 ms at |
| Online computation type | Neural network inference and mask evaluation |
| Method | Type | Model / encoding | Budget | Key parameters | Constraint |
| PB-PPO | Policy-based RL | Actor–critic MLP, , Tanh | steps | ; clip ; GAE ; ; actor lr ; critic lr ; epochs | Penalty |
| PPO | Policy-based RL | Actor–critic MLP, , ReLU | steps | ; clip ; GAE ; ; batch size | Penalty |
| AC | Actor–critic RL | Actor–critic MLP, , ReLU | steps | ; ; value-loss coefficient ; learning rate to | Penalty |
| DQN | Value-based RL | Q-network MLP, , ReLU | steps | ; replay buffer ; batch size ; target update interval ; : 1.0 to 0.05 | Penalty |
| RH-MILP | Receding-horizon optimization | MILP binary variables | Online re-solving | steps; s; time limit s; relative MIP gap ; horizon discount | Explicit |
| SA | Metaheuristic | Weight vector, 16-D | 2 restarts; 80 iter./restart; | Initial temperature ; minimum temperature ; cooling rate ; proposal scale ; 6 eval. eps./candidate | Repair / penalty |
| Target load | Method | Target completion rate (mean std) | Cumulative reward (mean std) | 95% CI of reward |
| 20 | PCM-PPO | 0.705 0.026 | 6217.5 185.6 | [6130.6, 6304.4] |
| PB-PPO | 0.593 0.029 | 5428.4 224.7 | [5323.2, 5533.6] | |
| RH-MILP | 0.594 0.027 | 5450.6 206.8 | [5353.8, 5547.4] | |
| PPO | 0.586 0.035 | 5365.9 243.1 | [5252.1, 5479.7] | |
| AC | 0.569 0.033 | 4927.2 205.6 | [4831.0, 5023.4] | |
| DQN | 0.539 0.032 | 4622.2 211.3 | [4523.3, 4721.1] |
| Algorithm | Targets 20 | Targets 30 | Targets 40 | Targets 50 |
| PCM-PPO | 0.802 | 0.758 | 0.704 | 0.660 |
| PB-PPO | 0.764 | 0.717 | 0.668 | 0.608 |
| RH-MILP | 0.769 | 0.701 | 0.645 | 0.589 |
| PPO | 0.718 | 0.682 | 0.642 | 0.593 |
| AC | 0.693 | 0.655 | 0.614 | 0.567 |
| Target load | Algorithm | Target completion rate (mean std) | Cumulative reward (mean std) | 95% CI of reward |
| 20 | PCM-PPO | 0.671 0.011 | 58480.6 965.4 | [58028.8, 58932.4] |
| PB-PPO | 0.584 0.013 | 51986.7 1056.4 | [51492.3, 52481.1] | |
| RH-MILP | 0.575 0.012 | 51320.8 1034.6 | [50836.6, 51805.0] | |
| PPO | 0.572 0.014 | 50642.3 1158.9 | [50099.9, 51184.7] | |
| AC | 0.564 0.013 | 47170.9 1048.5 | [46680.2, 47661.6] | |
| DQN | 0.532 0.013 | 45982.1 1015.6 | [45506.8, 46457.4] |
| Method | Training seeds | Cumulative reward mean std | Target completion rate mean std |
| PCM-PPO | 5 | 81272.6 686.3 | 0.619 0.009 |
| PB-PPO | 5 | 74591.8 842.7 | 0.558 0.010 |
| PPO | 5 | 71584.4 976.5 | 0.541 0.012 |
| Reward setting | Nominal avg. reward | Target completion rate | Downlink ratio | Avg. peak storage occupancy | Min. energy margin (Wh) |
| Nominal | 1324.5 | 0.607 | 0.821 | 0.86 | 246 |
| 1289.8 | 0.590 | 0.836 | 0.81 | 269 | |
| 1336.2 | 0.617 | 0.803 | 0.92 | 224 | |
| 1328.7 | 0.611 | 0.769 | 0.93 | 235 | |
| 1316.8 | 0.598 | 0.861 | 0.77 | 244 | |
| 1331.4 | 0.613 | 0.814 | 0.93 | 210 |
| Method variant | Raw-policy infeasible ratio | Final selected infeasible ratio |
| PCM-PPO | 0.107 | 0.000 |
| Maskable PPO | 0.123 | 0.000 |
| w/o Phys. mask | 0.052 | 0.052 |
| vanilla PPO | 0.078 | 0.078 |
| Receding-horizon length, | Time limit | Relative | Cumulative | Planning time |
| (s) | MIP gap | reward | (ms) | |
| 18 | 0.25 | 10534.8 | 2.17 | |
| 36 | 0.25 | 10690.2 | 4.79 | |
| 72 | 0.25 | 10738.5 | 11.60 | |
| 36 | 1.00 | 10722.8 | 6.35 | |
| 36 | 0.25 | 10701.4 | 4.68 |
| Task ID | Maneuver demand | Reward level | PCM-PPO | Maskable PPO | PPO |
| 4 | Low | Opportunity | Success | Success | Success |
| 9 | Low | Opportunity | Success | Success | Success |
| 27 | Low | Opportunity | Success | Success | Success |
| 11 | Medium | Opportunity | Success | Success | Success |
| 5 | Low | High | Success | Success | Success |
| 10 | Low | High | Success | Success | Success |
| Task ID | Maneuver demand | Reward level | PCM-PPO | Maskable PPO | PPO |
| 9 | Low | Opportunity | Success | Success | Success |
| 27 | Low | Opportunity | Success | Success | Success |
| 11 | Medium | Opportunity | Success | Success | Success |
| 5 | Low | High | Success | Success | Success |
| 10 | Low | High | Success | Success | Success |
| 14 | Low | High | Success | Success | Success |
| Task ID | Maneuver demand | Reward level | PCM-PPO | Maskable PPO | PPO |
| 9 | Low | Opportunity | Success | Success | Success |
| 27 | Low | Opportunity | Success | Success | Success |
| 5 | Low | High | Success | Success | Success |
| 10 | Low | High | Success | Success | Success |
| 14 | Low | High | Success | Success | Success |
| 21 | Low | High | Success | Success | Success |
| Task ID | Maneuver demand | Reward level | PCM-PPO | Maskable PPO | PPO |
| 9 | Low | Opportunity | Success | Success | Success |
| 27 | Low | Opportunity | Success | Success | Success |
| 11 | Medium | Opportunity | Success | Success | Success |
| 5 | Low | High | Success | Success | Success |
| 10 | Low | High | Success | Success | Success |
| 14 | Low | High | Success | Success | Success |
| Task ID | Maneuver demand | Reward level | PCM-PPO | Maskable PPO | PPO |
| 9 | Low | Opportunity | Success | Success | Success |
| 27 | Low | Opportunity | Success | Success | Success |
| 5 | Low | High | Success | Success | Success |
| 10 | Low | High | Success | Success | Success |
| 14 | Low | High | Success | Success | Success |
| 21 | Low | High | Success | Success | Success |
| Task ID | Maneuver demand | Reward level | PCM-PPO | Maskable PPO | PPO |
| 9 | Low | Opportunity | Success | Success | Success |
| 27 | Low | Opportunity | Success | Success | Success |
| 11 | Medium | Opportunity | Success | Success | Success |
| 5 | Low | High | Success | Success | Success |
| 10 | Low | High | Success | Success | Success |
| 14 | Low | High | Success | Success | Success |
| Task ID | Maneuver demand | Reward level | PCM-PPO | Maskable PPO | PPO |
| 9 | Low | Opportunity | Success | Success | Success |
| 27 | Low | Opportunity | Success | Success | Success |
| 5 | Low | High | Success | Success | Success |
| 10 | Low | High | Success | Success | Success |
| 14 | Low | High | Success | Success | Success |
| 21 | Low | High | Success | Success | Success |
| Method | Mean daily reward (95% CI) | Mean opportunity-target completion rate |
| PCM-PPO | ||
| Maskable PPO | ||
| PPO |
| Task ID | Maneuver demand | Reward level | PCM-PPO | Maskable PPO | PPO |
| 9 | Low | Opportunity | Success | Success | Success |
| 27 | Low | Opportunity | Success | Success | Success |
| 11 | Medium | Opportunity | Success | Success | Success |
| 2 | Medium | Opportunity | Success | Success | Success |
| 18 | Low | Opportunity | Success | Success | Success |
| 5 | Low | High | Success | Success | Success |
Constrained RL
Constrained reinforcement learning optimizes task performance while keeping safety risks, resource use, or other costs within explicit limits.
See all 242 papers