Curriculum-Driven Deep Reinforcement Learning for Autonomous Far-Range Rocket Landing

Atin Chowdhury; Piyush Joshi

cs.LGOct 24, 2026

Curriculum-Driven Deep Reinforcement Learning for Autonomous Far-Range Rocket Landing

Authors: Atin Chowdhury, Piyush Joshi

Organizations: Department of Computer Science and Engineering, Indian Institute of Information Technology, Sri City, Andhra Pradesh, India

Abstract

Autonomous vertical landing of reusable launch vehicles is a critical yet formidable challenge in aerospace engineering, primarily due to highly nonlinear dynamics, stringent precision require- ments, and inherent environmental uncertainties. While reinforcement learning (RL) has emerged as a promising alternative to traditional model-based control, existing approaches frequently rely on flat training paradigms and highly simplified initial conditions, drastically limiting their real-world applicability. This paper proposes a comprehensive curriculum-driven deep reinforcement learning framework designed to master both far-range navigation and precision landing. By employing a three-stage curriculum augmented with a critic warm-up procedure and a rehearsal mechanism to prevent catastrophic forgetting, alongside a high-dimensional observation vector utilizing a full 3 X 3 rotation-matrix, the agent is trained to navigate from an initial horizontal distance of 1,000 to 1,500 units vastly farther than prior benchmarks before executing a soft touchdown. Crucially, we intro- duce a redesigned reward function that reduces unnecessary computational overhead and addresses limitations of earlier formulations, including poorly calibrated reward components and potentially conflicting optimization objectives. The redesigned function bounds the per-step reward magnitude by (O(1)), while its progress term accumulates the total distance closed over the trajectory. The available 100,000-episode results report a 99.98% success rate for the proposed policy and 68.31% for the baseline under the evaluation conditions. In the evaluation, the proposed policy retained 69.1% of its initial fuel capacity and achieved a mean final distance from the pad centre of 7.07 units, compared with 13.10 units for the baseline. Landing visualizations are available in Video 1 and Video 2.

Figures & tables

Symbol	Meaning	Value
$w_{s}$	soft-descent reference tracking	$0.4$
$w_{\text{slow}}$	near-ground slow-speed shaping	$0.5$
$w_{o}$	tilt penalty (stage two / stage three)	$0.3$ / $0.0$
$w_{o,\text{low}}$	near-ground tilt penalty (stage two / stage three)	$0.3$ / $0.0$
$w_{\omega}$	angular-rate penalty	$0.05$
$w_{p,\text{low}}$	near-ground pad-distance penalty	$0.1$

Table 1: Landing reward weights used in stages two and three. Stage two includes tilt penalties to enforce uprightness, while stage three disables them (

w_{o}=w_{o,\text{low}}=0

) to focus on composite navigation.

Concept	Baseline of [ 1 ]	Present work
Number of terms	eight shaping plus one compound terminal term	eight shaping plus three explicit terminal terms
Hand-tuned constants	fifteen	eight
Per-step clipping	interval $[-500,\,+500]$	none, but per-episode return scaled by $0.1$
Reward magnitude	$O(10^{2})$ per step	$O(1)$ per step
Horizontal progress term	additive term with weight $0.5$ on normalised change of horizontal distance	explicit progress reward $w_{\text{prog}}(d_{t-1}-d_{t})$ , combined with directional alignment shaping for thrust
Angular-rate penalty	based on $\sin$ of decoupled pitch and yaw angles	true body-frame norm $\sqrt{\omega_{P}^{2}+\omega_{R}^{2}}$

Table 2: Term-by-term comparison between the reward of Chowdhury et al. [ 1 ] and the present reward.

Figure 1: Architecture of the curriculum-driven PPO actor-critic agent interacting with the 6-DoF rocket flight simulator. The top block (purple) depicts the neural network topology, action selection, and PPO update mechanism. The bottom block (green) depicts the physics environment, feature computation, and stage-specific reward evaluation. Solid arrows denote data flow (interaction), dashed arrows denote gradient/parameter updates, and dotted arrows denote rollout data.

Hyperparameter	Stage one	Stage two	Stage three
Learning rate	$3\times 10^{-4}$	$1.5\times 10^{-4}$	$1\times 10^{-4}$
Discount factor	$0.995$	$0.995$	$0.995$
Entropy coefficient	$0.010$	$0.005$	$0.003$
GAE lambda	$0.95$	$0.95$	$0.95$
Clipping range	$0.2$	$0.2$	$0.2$
Rollout per environment	$512$	$512$	$512$

Table 3: Per-stage PPO hyperparameters.

Panel of Figure 2 — Figure 2: Stage one (far approach) training curves. Left: episode reward rises steadily as the agent learns to navigate the $1{,}000$ – $1{,}500$ unit gap to the pad. Right: episode length in physics frames initially spikes as the agent explores the large workspace, then settles as it discovers an efficient trajectory to the hover zone.

Panel of Figure 3 — Figure 3: Stage one critic conditioning. Left: value loss drops sharply within the first million timesteps. Right: explained variance rapidly reaches $1.0$ , showing the critic achieves an essentially perfect fit of the value function. This guarantees the actor receives clean, well-conditioned gradient updates at the transfer point.

Panel of Figure 4 — Figure 4: Stage two (soft descent) training curves. Left: reward initially dips as the reward function changes, then recovers and rises after the $100{,}000$ -timestep critic warm-up. Right: episode length stabilises as the agent learns the vertical braking profile.

Panel of Figure 5 — Figure 5: Stage two critic conditioning. Left: the value loss exhibits initial spikes because the critic is freshly initialised for the new reward function. This justifies the critic warm-up. Right: explained variance recovers quickly as the critic adapts to the landing objective.

Panel of Figure 6 — Figure 6: Stage three (full task) training curves. Left: reward stabilises as the agent learns the composite navigation and landing task. Right: episode length becomes highly variable as the agent alternates between the long approach and the soft descent, reflecting the rehearsal mechanism.

Panel of Figure 7 — Figure 7: Stage three critic conditioning. No warm-up is required because stage three inherits the same reward and a well-conditioned critic from stage two. The value loss remains extremely low and the explained variance remains near $1.0$ , allowing the policy to focus on the composite manoeuvre.

Field	Mean $\pm$ std	Count
Success	$0.99980\pm 0.01414$	$100{,}000$
Frames	$2955.91\pm 261.02$	$100{,}000$
Within pad radius	$0.99980\pm 0.01414$	$100{,}000$
Vertical-speed condition satisfied	$1.00000\pm 0.00000$	$100{,}000$
Horizontal-speed condition satisfied	$1.00000\pm 0.00000$	$100{,}000$
Initial $x$ -distance, $\|x_{0}\|$	$795.996\pm 397.713$	$100{,}000$

Table 4: Evaluation statistics over

100{,}000

episodes. Values are sample mean

\pm

sample standard deviation. The initial

x

- and

y

-distance rows report absolute coordinate offsets,

|x_{0}|

and

|y_{0}|

; the initial horizontal distance row reports radial distance from the pad centre. Lengths are in simulator units and frames.

Metric	Baseline of [ 1 ]	Proposed work
Training paradigm	flat single-stage	three-stage curriculum with rehearsal
Attitude representation	quaternion plus angular-rate proxies	full $3\times 3$ rotation matrix
Observation dimension	$13$	$29$
Action space	$6$	$18$ , joint $2\times 3\times 3$
Reward terms	$8$ shaping plus $1$ compound terminal	$8$ shaping plus $3$ explicit terminal
Reward constants	$15$	$8$

Table 5: Design-level comparison with the study of Chowdhury et al. [ 1 ] .

Metric	Baseline of [ 1 ]	Proposed work	Improvement
Success rate	$68.31\%$	$99.98\%$	$+31.67$ pp
Touchdown tilt angle $\theta$	$11.2^{\circ}$	$0.396^{\circ}\pm 0.196^{\circ}$	$\approx 28.3\times$ lower
Touchdown vertical speed $\|v_{z}\|$	$0.190\pm 0.045$	$0.106\pm 0.014$	$\approx 1.8\times$ lower
Distance from pad centre	$13.10\pm 5.1$	$7.07\pm 1.99$	$\approx 1.9\times$ lower
Touchdown horizontal speed	$0.152\pm 0.021$	$0.0126\pm 0.0068$	$\approx 12\times$ lower

Table 6: Quantitative comparison of reported landing outcomes between the baseline of Chowdhury et al. [ 1 ] and the proposed method over the

100{,}000

-episode test set.

Quantity	Mean $\pm$ standard deviation	Limit	Fraction of budget used
Touchdown tilt angle $\theta$	$0.396^{\circ}\pm 0.196^{\circ}$	$\leq 10^{\circ}$	$\approx 3.95\%$
Touchdown vertical speed $\|v_{z}\|$	$0.106\pm 0.014$	$\leq 0.15$	$\approx 70.4\%$
Distance from pad centre	$7.07\pm 1.99$	$\leq 16$	$\approx 44.2\%$
Touchdown horizontal speed	$0.0126\pm 0.0068$	$\leq 0.25$	$\approx 5.0\%$
Fuel remaining	$691.15\pm 26.83$	$1{,}000$ max	$\approx 69.1\%$ of capacity

Table 7: Average measured landing parameters for the proposed method across the evaluation episodes. For quantities with a stated operating limit, the last column gives the fraction of that limit represented by the reported mean.

Touchdown condition	Episodes satisfying condition	Rate
Tilt angle $\theta\leq 10^{\circ}$	$100{,}000$	$100.00\%$
Distance from pad centre $d_{h}\leq 16$	$99{,}980$	$99.98\%$
Vertical velocity $v_{z}\geq-0.15$ units/frame	$100{,}000$	$100.00\%$
Horizontal speed $v_{h}\leq 0.25$ units/frame	$100{,}000$	$100.00\%$
All four conditions simultaneously	$99{,}980$	$99.98\%$

Table 8: Four-condition touchdown assessment for the

100{,}000

-episode evaluation. Each condition is checked from the touchdown state.

RL Control

Reinforcement learning control uses reward-driven learning to make decisions in dynamic systems, from robots and vehicles to energy and infrastructure.

See all 272 papers

Oct 2, 2026

Barrier-Shaped Recurrent Reinforcement Learning for Autonomous Landing on a Heaving Ship Deck

This paper addresses autonomous landing of unmanned aerial vehicle (UAV) rotorcrafts on a heaving ship deck using a recurrent policy trained via asymmetric actor-critic reinforcement learning: the critic sees 6 s of future deck height during training, while the actor sees only what the onboard sensors provide in flight, a 17-dimensional state made of the vehicle's position, velocity and attitude relative to the deck and outputs world-frame velocity and yaw-rate commands. Training is performed across 4096 parallel simulated environments, followed by fine-tuning in 16 environments with an onboard vision pipeline in the loop. A control-barrier-function (CBF) stopping margin on the deck-relative vertical state is incorporated at two stages: as a reward term during training, where it halves the median simulated contact speed relative to a policy trained without it, and as a runtime safety filter at deployment, evaluated at every control step to abort and retry the descent when the margin is violated. Because the autopilot's disarm logic cannot detect the vehicle resting on a moving deck, proximity-based thrust cutoff at touchdown is commanded directly in the landing pipeline. The proposed approach is validated using a parallel-manipulator-platform-based deck emulator that reproduces the heaving motion of the ship deck (scaled to 0.70 m peak-to-peak, 7.5 s mean period) and a quadcopter UAV with an onboard camera. Across 31 motion-capture and 20 vision-based trials, the UAV landed every time, with median times to contact of 6.5 and 7.3 s; 77% and 50% landed on the first attempt, with mean deck-relative speeds of 0.29 and 0.27 m/s, respectively, at the instant of thrust cutoff, which is the last speed under the policy's control. Supplementary video: https://youtu.be/S5hDkrSZJt4

Ritwik Shankar, Chiranjeev Prachand, Abhishek +1

Department of Aerospace Engineering, Indian Institute of Technology Kanpur, Kanpur, India · Department of Electrical Engineering, Indian Institute of Technology Kanpur, Kanpur, India

UAV Landing on Moving Platforms Aerial Robotics Quadrotor Control RL for Robotics Reinforcement Learning

Ask PDF

Jul 1, 2026

WaveLander: A Generalizable Hierarchical Control Framework for UAV Landing on Wave-Disturbed Platforms via Reinforcement Learning

Autonomous landing of unmanned aerial vehicles (UAVs) on wave-disturbed marine platforms remains challenging due to stochastic platform motion, time-varying platform attitude, and uncertain touchdown conditions. Existing model-based methods often require accurate motion prediction and online optimization, while end-to-end learning approaches may suffer from high training complexity and limited interpretability. This paper presents WaveLander, a hierarchical control framework via reinforcement learning (RL) that decouples vertical landing decision-making from low-level flight stabilization. The RL policy maps a compact platform-relative observation to a scalar vertical velocity reference, while a conventional low-level flight controller maintains attitude stability and lateral tracking. This formulation reduces dynamic platform landing to a low-dimensional, timing-aware control problem and enables smooth landing behavior without explicit switching rules. Simulation results under randomized wave-induced platform motions show that WaveLander achieves robust landing performance and generalizes to unseen disturbance conditions, demonstrating the potential of hierarchical learning-based control for marine UAV recovery.

Chun-Kit Li, Iok Long Sit, Ming Fung Siu +4

Department of Mathematics, The Hong Kong University of Science and Technology, Hong Kong SAR. · Cheng Kar-Shun Robotics Institute, The Hong Kong University of Science and Technology, Hong Kong SAR. · Division of Integrative Systems and Design, The Hong Kong University of Science and Technology, Hong Kong SAR. +2

RL Control Aerial Robotics Hierarchical RL UAV Landing on Moving Platforms

Ask PDF

Jun 30, 2026

Deep Reinforcement Learning for Spacecraft Attitude Control During Atmospheric Re-Entry

Deep reinforcement learning has the potential to solve attitude control problems more adaptively, precisely, and robustly by handling nonlinear dynamics, uncertainties, and failure cases more effectively than traditional attitude control approaches. We explore reinforcement learning (RL) for attitude control in spacecraft re-entry. An industry-standard proportional-integral-derivative controller with gain scheduling serves as a strong baseline for model-free RL and hybrid controllers that combine these two approaches. We formalize the application in the RL framework to apply continuous, off-policy RL. State-of-the-art RL achieves comparable performance to traditional control approaches in this domain. However, its out-of-distribution generalization is not sufficient. Hence, we use dynamics randomization to introduce challenging task variations during training and enforce generalization in a predefined operational envelope. Finally, we assess the best obtained RL-based controllers with application-specific metrics to show superior performance in comparison to traditional controllers in the operational envelope, that is, hybrid controllers are able to track the angle of attack better and are more robust under variations of mass, inertia tensor, and flap actuator bandwidth.

Alexander Fabisch, Melvin Laux, Mariela De Lucas Álvarez +2

Robotics Innovation Center, German Research Center for Artificial Intelligence (DFKI GmbH), Germany · 2Airbus Defence and Space GmbH, Germany

RL Control Spacecraft Attitude Determination Feedback Control Robust RL

Ask PDF

Abstract

RL Control

Explore similar work

Barrier-Shaped Recurrent Reinforcement Learning for Autonomous Landing on a Heaving Ship Deck

WaveLander: A Generalizable Hierarchical Control Framework for UAV Landing on Wave-Disturbed Platforms via Reinforcement Learning

Deep Reinforcement Learning for Spacecraft Attitude Control During Atmospheric Re-Entry