From Orbit Matching to Docking: A Hierarchical Reinforcement-Learning Approach to Multiphase Rendezvous Trajectory Design

Mariko Storey-Matsutani; Anatoly Zlotnik; Lee Holguin; Zigfried Hampel-Arias; Richard Linares

cs.ROOct 12, 2026

From Orbit Matching to Docking: A Hierarchical Reinforcement-Learning Approach to Multiphase Rendezvous Trajectory Design

Authors: Mariko Storey-Matsutani, Anatoly Zlotnik, Lee Holguin, Zigfried Hampel-Arias, Richard Linares

Organizations: Department of Aeronautics and Astronautics, Massachusetts Institute of Technology, Cambridge, MA 02139, USA · Los Alamos National Laboratory, Los Alamos, NM, USA

Abstract

As the complexity of space missions increases, autonomous spacecraft guidance must accommodate transitions between mission phases with distinct dynamics, objectives, and constraints. Rendezvous and proximity operations (RPO) are particularly challenging because they span multiple operating regimes, from orbital-scale matching and far-range approach to close-range maneuvering and terminal docking. Existing reinforcement-learning approaches generally address only a subset of this sequence or rely on manually defined switching conditions between controllers. This study presents a hierarchical reinforcement-learning framework for autonomous, three-degree-of-freedom RPO that spans the complete trajectory from orbit matching to docking. Five specialized policies are trained for orbit matching, far-range approach, mid-range approach, close-range approach, and docking. A learned high-level manager policy then selects among these phase-specific controllers based on the evolving spacecraft state. The hierarchical approach is evaluated over the full end-to-end rendezvous sequence and compared with a fixed sequential-invocation heuristic to assess trajectory performance and transition behavior. Training and inference metrics are also analyzed for the individual phase policies and the complete hierarchical system.

Figures & tables

Notation/Variable	Definition
$r$	Position
$v$	Velocity
$F$	Thrust force (control)
$m$	Mass
$t$	Time
$[B,C,T]$	[Body, Chaser, Target]

TABLE I : Notation and Variable Definitions

Phase	Initial Condition	Terminal Condition	Model Parameters
Orbit Match	Phase: [-90.0, 90.0] ∘	$\rho_{\mathrm{tol}}=10{,}000$ m	$\Delta v_{\max}=10{,}000$ m/s
	SMA: [-20k, 20k] m	$v_{\mathrm{tol}}=50$ m/s	$\Delta v_{\mathrm{k,max}}=1200$ m/s
	Ecc.: [0.0, 0.01]		$\rho_{\max}=25{,}000$ km
	Inc.: [-45.0, 45.0] ∘		$k_{\max}=$ 180
	RAAN: [-45.0, 45.0] ∘
Far Range	$\\|r_{\mathrm{rel},0}^{\mathcal{L}}\\|$ : [3000, 10000] m	$\rho_{\mathrm{tol}}=1000$ m	$\Delta v_{\max}=180$ m/s

TABLE II : Rendezvous Approach Phases

Reward	Definition
Position Penalty	$\phi=\begin{cases}r_{rel}/10&r_{rel}<r_{\text{prox}}\\ 0&\text{otherwise}\end{cases}$
Velocity Penalty	Constant: $\phi=-\|\|\mathbf{v}_{\text{rel}}\|\|$
	Distance-Scaled: $\phi=-\dfrac{\|\|\mathbf{v}_{\text{rel}}\|\|}{\max(r_{rel},r_{\min})}$
Progress	$\phi=\dfrac{\epsilon_{\text{prev}}-\epsilon_{\text{curr}}}{s}$ , where
	$\epsilon\in$ {altitude difference $\|h_{\text{tgt}}-h\|$ , (s=8M);
	eccentricity difference $\|e_{\text{tgt}}-e\|$ , (s=0.005);

TABLE III : Reward Component Definitions

	Single-Phase Models
Parameter	Orbit Match	Far Range	Mid Range	Close Range	Dock	Multi Phase
Reward Weights
Position Penalty	-	-	-	-1.0	-1.0	-
Velocity Penalty
Distance-Scaled	-	1.0	1.0	-	-	1.0
Constant	-	-	-	1.0	1.0	-

TABLE IV : Reward Weights Across All Controllers

Parameter	Orbit Match	Far Range	Mid Range	Close Range	Docking	Multi Phase
Network Size	[128, 128]					[128, 128]
Activation	tanh					tanh
Learning Rate	0.0003					0.0003
Batch size	256					256
Discount	0.998	0.99	0.99	0.99	0.99	0.99
GAE $\lambda$	0.95					0.95

TABLE V : RL Model Parameters

Phase	Success	Min. Range	Final Range	Final Vel.	$\Delta v$
	Rate	(m)	(m)	(m/s)	(m/s)
Orbit Match	97.0%	5380.10	5383.52	2.005	5660.64
Far Range	100.0%	249.96	269.97	0.478	26.83
Mid Range	100.0%	53.52	64.48	0.296	8.99
Close Range	100.0%	9.80	9.95	0.023	2.66
Docking	100.0%	0.98	0.98	0.009	0.25

TABLE VI : Single-phase policy Evaluation (

N=1000

episodes each, deterministic policy, held-out seed, all except success rate are median values).

Fig. 1 : Relative Position and Velocity Magnitude

Panel of Figure 8 — Fig. 2 : Trajectory Visualization from Learned Manager (Left: Earth-Centric, Right: Target-Centric)

Metric	Learned manager	Sequential Heuristic
Success rate	99.4%	97.0%
Final Range (m)	0.976	0.977
	( $3.76e4\pm 5.13e5$ )	( $5.25e4\pm 6.37e5$ )
Final Velocity (m/s)	0.00930	0.00930
	( $66.6\pm 947$ )	( $40.4\pm 525$ )
$\Delta v$ (m/s)	5706.86	5684.86

TABLE VII : Comparison of learned manager vs. heuristic baseline performance metrics with N=1000 (Format: Median (Mean

\pm

Std))

Fig. 3 : Comparison of learned manager vs. heuristic baseline for relative position and velocity through each trajectory. Only successful trajectories are displayed.

Phase	Learned Manager	Heuristic
Orbit Match	97.141% $\pm$ 1.052%	97.491% $\pm$ 0.803%
	( $25.8836\pm 7.4155$ h)	( $25.9421\pm 6.3255$ h)
Far Range	1.161% $\pm$ 0.380%	0.970% $\pm$ 0.303%
	( $0.2868\pm 0.0275$ h)	( $0.2456\pm 0.0448$ h)
Mid Range	0.484% $\pm$ 0.154%	0.375% $\pm$ 0.121%
	( $0.1194\pm 0.0109$ h)	( $0.0945\pm 0.0174$ h)

TABLE VIII : Time breakdown of learned manager and heuristic baseline by phase (percentages and hours reported as mean

\pm

std across episodes)

	Timesteps	Training	Throughput
Model		Time (h)	(fps)
Orbit Match
Stage 1	491,520	0.06	2,296
Stage 2	430,848	0.05	2,256
Stage 3	2,179,072	0.29	2,072
Total	3,101,440	0.40	2,129

TABLE IX : Training metrics per model

	Params	Checkpoint	Latency	Control Step
Model		(KB)	(ms)	Margin
Docking	35,847	462	$0.095\pm 0.032$	$1.05\times 10^{4}$
Close Range	35,847	464	$0.117\pm 0.072$	$8.53\times 10^{3}$
Mid Range	35,847	464	$0.093\pm 0.037$	$1.08\times 10^{4}$
Far Range	35,847	464	$0.096\pm 0.053$	$3.75\times 10^{4}$
Orbit Matching	36,871	476	$0.110\pm 0.053$	$9.35\times 10^{6}$

TABLE X : Model size and inference latency (N = 1,000)

Metrics	Learned	Heuristic
Success rate	0.994	0.970
Mean episode wall-clock (s)	$0.961\pm 0.152$	$0.917\pm 0.221$
Median episode wall-clock (s)	0.932	0.905
Expected wall-clock per success (s)	0.967	0.945
Computational Breakdown
Manager Inference (%)	0.21	-

TABLE XI : Computational breakdown for the learned and heuristic hierarchical managers (N = 1,000 paired episodes)

Spacecraft Rendezvous and Proximity Operations

Spacecraft rendezvous and proximity operations cover the navigation, guidance, and control needed to approach, inspect, dock with, or capture spacecraft safely.

See all 26 papers

AI-Augmented Model Predictive Control for Safe and Adaptive Rendezvous and Proximity Operations
Jul 18, 2026Luca Sportelli, Tyler Barr, Cagri Kilic +1Spacecraft Rendezvous and Proximity Operations Multi-Agent Control
Department of Aerospace Engineering, Embry–Riddle Aeronautical University, Daytona Beach, FL 32114
Spacecraft Rendezvous Trajectory Generation with Modular Constraints via Diffusion Model Composition
Oct 5, 2026Mariko A. Storey-Matsutani, Richard LinaresConstrained Generative Modeling Trajectory Generation
Graduate Student, Department of Aeronautics and Astronautics, Massachusetts Inst. of Technology, Cambridge MA · Associate Professor, Department of Aeronautics and Astronautics, Massachusetts Inst. of Technology, Cambridge MA
Curriculum-Driven Deep Reinforcement Learning for Autonomous Far-Range Rocket Landing
Oct 24, 2026Atin Chowdhury, Piyush JoshiRL Control Curriculum RL
Department of Computer Science and Engineering, Indian Institute of Information Technology, Sri City, Andhra Pradesh, India

Jul 18, 2026J/K move · Enter open · S save

AI-Augmented Model Predictive Control for Safe and Adaptive Rendezvous and Proximity Operations

Luca Sportelli, Tyler Barr, Cagri Kilic, Di Wu

Department of Aerospace Engineering, Embry–Riddle Aeronautical University, Daytona Beach, FL 32114

Spacecraft Rendezvous and Proximity Operations Multi-Agent Control Model Predictive Control Adaptive Control

Open paper AskPDF arXiv

Autonomous rendezvous and proximity operations (RPO) in adversarial orbital environments require guidance architectures balancing target pursuit, safety preservation, and real-time adaptability under dynamically evolving interaction conditions. Although learning-based approaches show promise, their application to safety-critical orbital robotics remains limited by concerns regarding interpretability, robustness, and constraint awareness. This work presents an adaptive Model Predictive Control (MPC) framework for autonomous spacecraft RPO in multi-agent adversarial scenarios. The proposed architecture combines a constrained receding-horizon MPC formulation with a data-driven supervisory tuning layer that adjusts controller parameters from offline closed-loop evaluation and online interaction geometry. Relative motion follows Clohessy-Wiltshire (CW) dynamics, enabling computationally efficient finite-horizon prediction and real-time quadratic optimization. The MPC formulation incorporates actuator limits, predictive keep-out-zone constraints, slack-variable feasibility handling, and optional Control Barrier Function (CBF) safety filtering. Rather than generating thrust commands directly, the adaptive layer modifies interpretable MPC parameters, including tracking weights, safety penalties, minimum-separation objectives, and keep-out-zone objectives. The framework was evaluated in the official Kerbal Space Program Differential Game (KSPDG) Capture-the-Satellite environment through Monte Carlo simulations. Results demonstrate improved closed-loop robustness, adaptive maneuvering behavior, and rendezvous performance compared with fixed-parameter MPC while preserving safety-aware operation and real-time feasibility, providing a modular, interpretable foundation for adaptive spacecraft RPO.

Abstract

Spacecraft Rendezvous and Proximity Operations

Explore similar work

AI-Augmented Model Predictive Control for Safe and Adaptive Rendezvous and Proximity Operations

Spacecraft Rendezvous Trajectory Generation with Modular Constraints via Diffusion Model Composition

Curriculum-Driven Deep Reinforcement Learning for Autonomous Far-Range Rocket Landing