Curriculum-Based Deep Reinforcement Learning for Robust Autonomous Rocket Landing under Atmospheric Uncertainty

Atin Chowdhury; Piyush Joshi

cs.AIOct 28, 2026

Curriculum-Based Deep Reinforcement Learning for Robust Autonomous Rocket Landing under Atmospheric Uncertainty

Authors: Atin Chowdhury, Piyush Joshi

Organizations: Department of Computer Science and Engineering, Indian Institute of Information Technology, Sri City, Andhra Pradesh, India

Abstract

Autonomous far-range rocket landing is a challenging control problem because navigation, descent-rate regulation, attitude management, and fuel use must be coordinated in the presence of uncertain atmospheric and vehicle dynamics. This paper presents a curriculum-driven proximal policy optimization (PPO) framework for three-dimensional rocket landing in a stochastic environment incorporating altitude-dependent aerodynamic drag, wind shear, temporally correlated turbulence, discrete gusts, uncertain propulsion and aerodynamic parameters, thrust-vector misalignment, engine lag, actuator-gain variation, noisy observations, and command latency. Learning is organised into far-range approach, soft descent, and integrated landing stages. Policy transfer, critic warm-up after reward changes, rehearsal of earlier tasks, and progressive expansion of weather severity are used to support learning across these coupled phases. Across 100,000 evaluation episodes, the policy achieves 96,656 successful landings, corresponding to a success rate of 96.656% and a 95% Wilson confidence interval of 96.543--96.766%. Success decreases from 98.84% under zero-disturbance conditions to 88.07% in the highest weather-severity band. Successful touchdowns exhibit small miss distances and low residual horizontal and vertical velocities. An exploratory logistic-regression analysis identifies mean wind speed as the strongest measured correlate of success, with initial distance, initial altitude, initial vertical velocity, and wind-shear exponent also associated with outcomes. The results quantify both the effectiveness of the learned policy over the evaluated disturbance distribution and the degradation that occurs under severe compound atmospheric conditions.

Figures & tables

Quantity	Range at $\ell=1$	Implementation role
Reference horizontal wind	$[0.4,1.0]\times 1.5$ units/frame	Random direction and altitude shear
Horizontal turbulence SD	$[0.4,1.0]\times 0.25$ units/frame	Correlated per-frame perturbation
Vertical turbulence innovation	Half the horizontal scale	Vertical atmospheric fluctuation
Gust amplitude parameter	$[0.4,0.9]\times 1.5$ units/frame	Intermittent 60–160-frame gusts
Gust initiation probability	$[0,0.0015]$ per frame	Probability of starting a gust
Density scale $s_{\rho}$	$[0.85,1.15]$	Scales air density

Table 1: Principal uncertainty ranges in the stochastic environment. Ranges describe the simulator’s engineering model and are not flight-calibrated distributions.

Figure 1: Overall architecture of the curriculum-trained PPO rocket-landing system. The diagram shows the 29-dimensional noisy observation, separate actor and critic networks, factorised MultiDiscrete action head, rollout buffer, PPO update, disturbance sampler, actuator latency, flight dynamics, reward calculation, and episode manager. Although the schematic includes the label “6-DoF”, the simulation model used in this study approximates three-dimensional translation and commanded pitch/roll attitude dynamics; it is not a validated full six-degree-of-freedom aerodynamic vehicle model.

Reward parameter	Stage 2	Stage 3
$w_{s}$	0.4	0.4
$w_{\mathrm{slow}}$	0.5	0.5
$w_{o}$	0.3	0
$w_{o,\mathrm{low}}$	0.3	0
$w_{\omega}$	0.05	0.05
$w_{p,\mathrm{low}}$	0.10	0.12

Table 2: Reward weights for the stage-two and stage-three landing tasks.

Parameter	Stage 1	Stage 2	Stage 3
Task	Far approach / hover	Soft descent	Integrated landing
Timesteps	$4.0\times 10^{6}$	$4.0\times 10^{6}$	$8.0\times 10^{6}$
Initial horizontal radius	1,000–1,500	0–15	1,000–1,500
Initial altitude	900–1,300	500–800	900–1,300
Initial vertical velocity	$[-0.8,-0.1]$	$[-2.5,-0.5]$	$[-1.5,-0.2]$
Initial tilt offsets	$\pm 20^{\circ}$	$\pm 15^{\circ}$	$\pm 20^{\circ}$

Table 3: Stage-specific configuration and common PPO settings from the training configuration.

Panel of Figure 2 — Figure 2: Stage 1 training behaviour. Left: individual episode returns and their rolling mean. Right: episode lengths in agent decisions and their rolling mean. Rewards improve rapidly early in training and remain mostly positive later, while the mean episode length eventually falls from prolonged capped episodes to a shorter range.

Panel of Figure 3 — Figure 3: Stage 2 training behaviour. Left: individual episode returns and rolling mean. Right: individual episode lengths and rolling mean. Both statistics are non-monotonic; the late negative reward trend and increase in mean episode length warrant checkpoint-level evaluation rather than a blanket convergence claim.

Panel of Figure 4 — Figure 4: Stage 3 training behaviour. Left: individual episode returns and rolling mean. Right: episode lengths and rolling mean. Following the initial transient, average returns remain comparatively stable and mean episode length fluctuates within a narrower band than in the early Stage 2 trace.

Panel of Figure 5 — Figure 5: Critic explained variance during the three curriculum stages. Values close to one indicate that the critic explains a large fraction of return-target variance within the logged samples. Stage 2 contains more pronounced dips; in all stages the metric should be interpreted as a critic diagnostic rather than a direct measure of landing success.

Outcome	Episodes	Share of all episodes
Successful touchdown	96,656	96.656%
Touchdown failing one or more criteria	3,237	3.237%
Timeout	107	0.107%
Left workspace	0	0.000%
Total	100,000	100.000%

Table 4: Outcomes over the 100,000-episode stochastic-atmosphere evaluation.

Weather level	$n$	Successful	Crashed	Timeout + lost	Success
0 (calm)	15,218	15,041	177	0	98.84%
$(0,0.2]$	17,059	16,857	202	0	98.82%
$(0.2,0.4]$	16,936	16,780	156	0	99.08%
$(0.4,0.6]$	16,825	16,658	167	0	99.01%
$(0.6,0.8]$	17,097	16,467	628	2	96.32%
$(0.8,1.0]$	16,865	14,853	1,907	105	88.07%

Table 5: Outcome counts and success rate by weather severity. Rates are calculated from the episode counts.

Figure 6: Success rate versus weather level using the six summary bins in Table 5 . Error bars show Wilson 95% intervals calculated from each bin’s counts. The nonzero weather bins contain mixed values within their stated intervals.

Metric	Mean $\pm$ SD	Median	P95	Criterion / note
Miss distance	$6.896\pm 3.439$	6.634	13.054	$\leq 16$ units
Horizontal speed	$0.0573\pm 0.0310$	0.0529	0.1142	$\leq 0.25$
Vertical speed $\|v_{z}\|$	$0.1085\pm 0.0252$	0.1101	0.1470	$v_{z}\geq-0.2$
Tilt (descriptive)	$4.932\pm 2.806$	4.390	10.149	Not in success rule
Fuel used	$168.93\pm 47.80$	159.18	226.53	Initial fuel: 1,000
Flight duration	$26.015\pm 9.118$	24.117	36.517	Descriptive (seconds)

Table 6: Landing-quality statistics for the 96,656 successful episodes. Velocities are in units/frame, lengths are in simulator units, and all reported means include sample standard deviations.

Violated criteria	Episodes	Share of crashes
Missed pad only	2,677	82.7%
Missed pad, too fast vertically, and too fast horizontally	366	11.3%
Missed pad and too fast horizontally	69	2.1%
Missed pad and too fast vertically	67	2.1%
Too fast vertically only	41	1.3%
Too fast vertically and too fast horizontally	17	0.5%

Table 7: Mutually exclusive combinations of violated touchdown criteria among the 3,237 unsuccessful touchdowns.

Factor	Bin	Episodes	Successful	Success
Mean wind speed	$0$	15,218	15,041	98.84%
	$(0,0.3]$	25,937	25,654	98.91%
	$(0.3,0.6]$	25,938	25,678	99.00%
	$(0.6,0.9]$	19,771	19,264	97.44%
	$(0.9,1.2]$	10,145	8,982	88.54%
	$(1.2,1.5]$	2,991	2,037	68.10%

Table 8: Marginal success rates for selected disturbance values.

Figure 7: Marginal success rates by mean wind speed, turbulence intensity, and command latency. The bins are defined over the observed ranges of the corresponding disturbance variables. The curves are descriptive and do not isolate causal effects.

Predictor	OR	95% CI for OR	Reported $p$ -value
Mean wind speed	0.244	[0.228, 0.260]	$<10^{-16}$
Wind-shear exponent	1.453	[1.399, 1.508]	$<10^{-16}$
Initial height	1.210	[1.167, 1.255]	$<10^{-16}$
Initial vertical speed	1.199	[1.156, 1.243]	$<10^{-16}$
Initial distance	0.839	[0.809, 0.871]	$<10^{-16}$
Gust amplitude	1.172	[1.097, 1.252]	$2.78\times 10^{-6}$

Table 9: Selected results from the in-sample logistic regression. OR is the odds ratio per one-standard-deviation increase in the named predictor.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,742 papers

Curriculum-Driven Deep Reinforcement Learning for Autonomous Far-Range Rocket Landing
Oct 24, 2026Atin Chowdhury, Piyush JoshiRL Control Curriculum RL
Department of Computer Science and Engineering, Indian Institute of Information Technology, Sri City, Andhra Pradesh, India
Adversarial Reinforcement Learning for Robust Planetary Landing
Oct 14, 2026Miguel Leiva-Vélez, Nicolas Gaston-Rozado, Andrea Scorsoglio +3Adversarial MARL Reinforcement Learning
ETSI Aeronáutica y del Espacio, Universidad Politécnica de Madrid, Madrid, Spain · Indra Sistemas, Madrid, Spain · Department of Systems and Industrial Engineering, University of Arizona, Tucson, USA +2
Deep Reinforcement Learning for Spacecraft Attitude Control During Atmospheric Re-Entry
Jun 30, 2026Alexander Fabisch, Melvin Laux, Mariela De Lucas Álvarez +2RL Control Spacecraft Attitude Determination
Robotics Innovation Center, German Research Center for Artificial Intelligence (DFKI GmbH), Germany · 2Airbus Defence and Space GmbH, Germany

Oct 24, 2026J/K move · Enter open · S save

Curriculum-Driven Deep Reinforcement Learning for Autonomous Far-Range Rocket Landing

Atin Chowdhury, Piyush Joshi

Department of Computer Science and Engineering, Indian Institute of Information Technology, Sri City, Andhra Pradesh, India

RL Control Curriculum RL

Open paper AskPDF arXiv

Autonomous vertical landing of reusable launch vehicles is a critical yet formidable challenge in aerospace engineering, primarily due to highly nonlinear dynamics, stringent precision require- ments, and inherent environmental uncertainties. While reinforcement learning (RL) has emerged as a promising alternative to traditional model-based control, existing approaches frequently rely on flat training paradigms and highly simplified initial conditions, drastically limiting their real-world applicability. This paper proposes a comprehensive curriculum-driven deep reinforcement learning framework designed to master both far-range navigation and precision landing. By employing a three-stage curriculum augmented with a critic warm-up procedure and a rehearsal mechanism to prevent catastrophic forgetting, alongside a high-dimensional observation vector utilizing a full 3 X 3 rotation-matrix, the agent is trained to navigate from an initial horizontal distance of 1,000 to 1,500 units vastly farther than prior benchmarks before executing a soft touchdown. Crucially, we intro- duce a redesigned reward function that reduces unnecessary computational overhead and addresses limitations of earlier formulations, including poorly calibrated reward components and potentially conflicting optimization objectives. The redesigned function bounds the per-step reward magnitude by (O(1)), while its progress term accumulates the total distance closed over the trajectory. The available 100,000-episode results report a 99.98% success rate for the proposed policy and 68.31% for the baseline under the evaluation conditions. In the evaluation, the proposed policy retained 69.1% of its initial fuel capacity and achieved a mean final distance from the pad centre of 7.07 units, compared with 13.10 units for the baseline. Landing visualizations are available in Video 1 and Video 2.

Abstract

Reinforcement Learning

Explore similar work

Curriculum-Driven Deep Reinforcement Learning for Autonomous Far-Range Rocket Landing

Adversarial Reinforcement Learning for Robust Planetary Landing

Deep Reinforcement Learning for Spacecraft Attitude Control During Atmospheric Re-Entry