Reachability-Informed Reinforcement Learning for Multi-Impulse Interplanetary Transfers

Yashdeep Chaudhary; Roberto Armellin; Harry Holt

math.OCOct 1, 2026

Reachability-Informed Reinforcement Learning for Multi-Impulse Interplanetary Transfers

Authors: Yashdeep Chaudhary, Roberto Armellin, Harry Holt

Organizations: Faculty of Engineering Waipapa Taumata Rau– University of Auckland Auckland 1010, New Zealand · Advanced Concepts Team ESTEC, European Space Agency Noordwijk 2201 AZ The Netherlands

Abstract

Reinforcement learning offers the prospect of a reusable sequential decision-making mechanism for spacecraft trajectory design, motivating policy interfaces that connect learned decisions to the underlying maneuver geometry. This paper develops Reachability Analysis-Informed Reinforcement Learning (RARL) for deterministic multi-impulse interplanetary transfers, placing intermediate waypoint selection at the center of the learned decision process. Local first-order reachability maps bounded velocity perturbations into an ellipsoidal set of next-node positions, within which the policy selects its waypoint. Lambert reconstruction then determines the corresponding maneuver to reach this selected waypoint along a dynamically consistent ballistic arc, coupling learned transfer-geometry selection with classical astrodynamics. A terminal two-impulse reconstruction completes the rendezvous, supported by a linear maneuver-demand assessment used for reward shaping. Numerical studies characterize this interface on a two-body Earth-Mars benchmark. Across three independent training runs, RARL achieves a mean maneuver cost of 10.23 km/s, 1.72% above a validated local sequential convex programming reference. Training over dispersed initial states extends policy reuse across a departure family with fixed target state and transfer duration. Each of the three independently trained multi-state policies completes all 10,000 held-out Monte Carlo departures without impulse-cap violations, compared with a mean feasibility rate of 6.49% for single-state policies. This broader sampled feasibility is accompanied by a 0.61% increase in mean nominal maneuver cost, without further training across departures. These results demonstrate that a reachability-informed decision interface supports benchmark-quality trajectory construction and policy reuse across dispersed departure conditions.

Figures & tables

Panel of Figure 1 — Figure 1: Two-dimensional schematic of the local first-order map. The position-from-velocity STM block maps guarded unit-ball coordinates to an ellipsoidal waypoint set centered on the nominal coast position, with orientation and semiaxis lengths set by local segment geometry.

Figure 2: Action-mapping comparison for $10^{4}$ uniform cube inputs. (a) Projected waypoints at the initial Earth–Mars node. (b) Radial exceedance probabilities for $r_{u}=\|\bm{u}\|_{2}$ . Dashed curves show analytical values, and annotations report empirical boundary probabilities.

Figure 3: Linearization adequacy of the first-order reachability map. (a) Relative error versus impulse fraction. (b) Segment-wise error for

\alpha=1

(maximum impulse).

Figure 4: Terminal two-impulse proxy versus normalized Lambert demand. Hexbins aggregate

10^{4}

infeasible samples. Dashed and dotted lines mark activation and total-demand thresholds, respectively.

Quantity	Value	Unit
$\mu_{\odot}$	$132{,}712{,}440{,}018$	$\mathrm{km^{3}/s^{2}}$
$t_{f}-t_{0}$	$348.79$	days
$N$	$20$	–
$\Delta t_{k}$	$17.4395$	days
$\Delta V_{\max}$	$0.7600$	$\mathrm{km/s}$
$\bm{r}_{0}^{-}$	$[-140{,}699{,}693,\,-51{,}614{,}428,\,980]^{\mathsf{T}}$	km

Table 1: Earth–Mars benchmark parameters and numerical scales.

Input	Lower bound	Upper bound
$\bm{r}_{k}^{-}/r_{\mathrm{scale}}$	$[-2.3,-2.4,-0.3]$	$[2.3,2.4,0.3]$
$\bm{v}_{k}^{-}/v_{\mathrm{scale}}$	$[-2.1,-2.2,-0.25]$	$[2.1,2.2,0.25]$
$\Delta\bm{r}_{k}^{\mathrm{RTN}}/r_{\mathrm{scale}}$	$[-3.5,-2.7,-0.4]$	$[3.5,2.7,0.4]$
$\Delta\bm{v}_{k}^{\mathrm{RTN}}/v_{\mathrm{scale}}$	$[-1.6,-2.5,-0.2]$	$[1.6,2.5,0.2]$
$\tau_{k}$	$0$	$1$
$\rho_{k-1}$	$0$	$2$

Table 2: Bounds for fixed observation scaling.

Parameter	Symbol	Value
Actor hidden sizes	–	$[70,32,15]$
Critic hidden sizes	–	$[56,14,4]$
Initial action log-std	$\log\sigma_{\pi,0}$	$0$
Initial learning rate	$\alpha_{0}$	$3.0\times 10^{-4}$
Final learning rate	$\alpha_{\mathrm{final}}$	$1.0\times 10^{-5}$
Discount factor	$\gamma$	$0.9999$

Table 3: Policy architecture and PPO training configuration.

Set	States	Construction and purpose
Training	$100$	Uniform grid sampled during training
Interleaved evaluation	$10$	Separate grid between training states for checkpoint selection
MC validation	$10^{4}$	Uniformly sampled offsets for assessment after selection

Table 4: State sets for training, evaluation, and MC validation.

Transfer performance
Quantity	Unit	SCP	RARL (single-state)
			Seed 1	Seed 2	Seed 3	Mean $\pm$ SD a
Total maneuver cost	km/s	$10.0585$	$10.2069$	$10.2469$	$10.2416$	$10.2318\pm 0.0218$
Cost above SCP b	km/s	$0.0000$	$0.1484$	$0.1884$	$0.1831$	$0.1733\pm 0.0218$
Gap to SCP b	%	$0.00$	$1.47$	$1.87$	$1.82$	$1.72\pm 0.22$
Impulse allocation

Table 5: Nominal single-state performance and impulse allocation.

Figure 5: Nominal impulse allocation for (a) RARL and (b) SCP. RARL bars and whiskers show the mean and sample standard deviation across three independent runs.

Figure 6: Nominal Earth–Mars transfer for the representative single-state RARL policy. Panel (a) shows the

XY

projection, with a terminal inset showing the projected waypoint set. Panels (b)–(c) show the

YZ

and

XZ

projections. Crosses mark selected intermediate waypoints.

Figure 7: Local $XY$ projections of the waypoint sets along the representative nominal trajectory. Crosses mark policy-selected waypoints at nodes 1–19, and the diamond marks the terminal target at node 20. Annotations give the normalized projected radius $\eta_{xy}$ and the preceding reconstructed impulse magnitude. All panels use the same spatial scale.

MC validation b
Quantity	RARL (single-state)	RARL (multi-state)
	Mean $\pm$ SD a	Seed 1	Seed 2	Seed 3	Mean $\pm$ SD a
Feasibility rate	$(6.49\pm 2.49)\%$	$100.00\%$	$100.00\%$	$100.00\%$	$100.00\%$
Mean maneuver cost	$10.3320\pm 0.0420$	$10.3362$	$10.3091$	$10.2831$	$10.3095\pm 0.0266$
MC cost spread, $\mathrm{SD}_{\mathrm{MC}}$ c	$0.0829\pm 0.0115$	$0.1235$	$0.1114$	$0.1337$	$0.1229\pm 0.0112$
95th-percentile cost	$10.4868\pm 0.0443$	$10.5506$	$10.4978$	$10.5074$	$10.5186\pm 0.0281$

Table 6: Multi-state performance under launch-time dispersion. All quantities except feasibility rate and percentage gaps are in km/s.

Figure 8: MC cost comparison of single- and multi-state RARL seeds.

Figure 9: Earth–Mars trajectory ensemble for the representative multi-state RARL policy, showing 201 MC departures and nominal waypoints. MC deviations from the nominal trajectory are visually magnified $\times 7.5$ in the XY plane (a) and $\times 2.5$ in the YZ and XZ projections (b)–(c). The terminal inset shows the nominal waypoint set and MC trajectories at true scale.

Figure 10: MC impulse and cost distributions for RARL (multi-state) policy seeds, with 5th–95th percentile whiskers and

0.01\,\mathrm{km/s}

bins.

Explore similar work

Jun 11, 2026math.OC

Distribution-Agnostic Robust Trajectory Optimization via Chance-Constrained Reinforcement Learning

This paper presents a distribution-agnostic robust trajectory-optimization framework based on chance-constrained reinforcement learning. The uncertainty is represented here through initial conditions and process noise, with the only requirement being that it can be sampled. A deterministic nominal trajectory is first computed offline, and reinforcement learning is then used only to robustify that baseline through a structured affine closed-loop correction law comprising a feedforward control adjustment and time-varying feedback gains. Probabilistic feasibility is enforced empirically through rollout-based upper-tail quantiles, while terminal dispersion is regulated through covariance-feasibility penalties. The framework is assessed on two materially different trajectory design problems. The flagship case study is a three-dimensional multi-impulse Earth-Mars transfer, where the learned policy is benchmarked against a recent robust trajectory-optimization reference under Gaussian uncertainty and then evaluated under bounded uniform uncertainty and under process disturbances not seen during training. The second case study is a stochastic atmospheric pinpoint rocket landing problem, used to assess portability to a short-horizon continuous-thrust setting with drag, mass depletion, and glide-slope constraints. The results show that the proposed framework can remain competitive in upper-tail fuel cost while preserving probabilistic feasibility, and that the same robustification scaffold can be carried across heterogeneous spacecraft trajectory planning problems without redesign of its core stochastic-control structure.

Yashdeep Chaudhary, Roberto Armellin, Harry Holt +1

Faculty of Engineering, Waipapa Taumata Rau – University of Auckland, 20 Symonds Street, Auckland, 1010, New Zealand · Advanced Concepts Team, European Space Research and Technology Centre, ESA, Noordwijk, 2201 AZ, The Netherlands · Department of Industrial Engineering, Alma Mater Studiorum - Università di Bologna, Via Zamboni 33, Bologna, 40126, Italy

Intent-Aligned Autonomous Spacecraft Guidance Robust Trajectory Distributionally-Robust Optimization Bidirectional Thrust

Ask PDF

May 23, 2026cs.RO

Towards Low-Gravity Planetary Exploration using Reinforcement Learning for Walking, Jumping, and In-flight Attitude Control

This paper presents reinforcement learning (RL) policies for dynamic quadrupedal locomotion in planetary exploration scenarios. Building on a taskoptimized quadruped with a 5-bar leg design, we develop RL policies for walking, vertical jumping, forward jumping, and in-flight attitude control, explicitly tailored to the reduced gravity on Mars. These policies jointly enable such robots to overcome obstacles larger than themselves through coordinated jumping and precise in-flight reorientation for safe landings. We demonstrate Sim2Real transfer of the attitude control policy on the Olympus quadruped through single-axis reorientation tests, while all locomotion policies are validated in simulation. A complete Mars exploration mission scenario demonstrates coordinated policy deployment across challenging terrain. Experimental results show 90° attitude reorientation in 2.6 seconds, with simulations demonstrating 3.1 meter vertical jumps and 3.9 meter forward jumps under Martian gravity conditions. - Supplementary video: https://www.youtube.com/watch?v=qlSJ3P87A4A

Jørgen Anker Olsen, Kostas Alexis

The authors are with the Autonomous Robots Lab, NTNU, O.S. Bragstads Plass 2D, 7034, Trondheim, Norway.

Planetary Exploration Reinforcement Learning Control Active Flow Control Exploration

Ask PDF

Jul 1, 2026cs.MA

Simulation Based Reward Function Validation for Multi-Agent On Orbit Inspection

A proposed method for the control of groups of inspection spacecraft is Multi-Agent Reinforcement Learning (MARL). While MARL has already been employed for this purpose in previous work, the reward functions used focus on reaching a finite set of predetermined inspection points around the target. In this work, we study and develop a generalized reward function for the MARL inspection task informed by the analysis of 3D reconstructions of inspected objects in orbit. Because the reward function is generalized such that any number of images at arbitrary locations may evaluated, we also allow trained agents to have complete control over when images are collected. With this approach, we gather insights into best practices for not only the specific MARL inspection task, but also gain key takeaways informative to the broader inspection task outside of a MARL context.

Patrick Quinn, Bala Prenith Reddy Gopu, George M. Nehma +1

Florida Institute of Technology, Melbourne, FL., 32901

Multi-Agent Reinforcement Learning Spacecraft Reward Functions 3D Reconstruction

Ask PDF