Mitigating Data Scarcity in Spaceflight Applications for Offline Reinforcement Learning Using Physics-Informed Deep Generative Models

Alex E. Ballentine; Nachiket U. Bapat; Raghvendra V. Cowlagi

cs.LGApr 2, 2026

Mitigating Data Scarcity in Spaceflight Applications for Offline Reinforcement Learning Using Physics-Informed Deep Generative Models

Authors: Alex E. Ballentine, Nachiket U. Bapat, Raghvendra V. Cowlagi

Organizations: Graduate Research Assistant, Aerospace Engineering Department, Worcester Polytechnic Institute, MA. · Associate Professor, Aerospace Engineering Department. AIAA Senior Member. Corresponding Author.

Abstract

The deployment of reinforcement learning (RL)-based controllers on physical systems is often limited by poor generalization to real-world scenarios, known as the simulation-to-reality (sim-to-real) gap. This gap is particularly challenging in spaceflight, where real-world training data are scarce due to high cost and limited planetary exploration data. Traditional approaches, such as system identification and synthetic data generation, depend on sufficient data and often fail due to modeling assumptions or lack of physics-based constraints. We propose addressing this data scarcity by introducing physics-based learning bias in a generative model. Specifically, we develop the Mutual Information-based Split Variational Autoencoder (MI-VAE), a physics-informed VAE that learns differences between observed system trajectories and those predicted by physics-based models. The latent space of the MI-VAE enables generation of synthetic datasets that respect physical constraints. We evaluate MI-VAE on a planetary lander problem, focusing on limited real-world data and offline RL training. Results show that augmenting datasets with MI-VAE samples significantly improves downstream RL performance, outperforming standard VAEs in statistical fidelity, sample diversity, and policy success rate. This work demonstrates a scalable strategy for enhancing autonomous controller robustness in complex, data-constrained environments.

Figures & tables

Panel of Figure 1 — Figure 1: Schematic overview of the overall working pipeline

Panel of Figure 2 — Figure 2: Forces acting on the Mars Lander vehicle

Panel of Figure 3 — Figure 3: Architecture of a S-VAE

Panel of Figure 4 — Figure 4: Architecture of MI-VAE

Parameter Set	${m}$ [ $\mathrm{m}$ ]	${g}$ [ $\mathrm{\nicefrac{{m}}{{s^{2}}}}$ ]	${l}$ [ $\mathrm{m}$ ]	${c}$	$\Delta t$ [ $\mathrm{s}$ ]	$u_{1}(t)^{\mathrm{max}}$ [ $\mathrm{kN}$ ]	$u_{2}(t)^{\mathrm{max}}$ [ $\mathrm{kN}$ ]	$u_{3}(t)^{\mathrm{max}}$ [ $\mathrm{kN}$ ]
PA	$500$	$3.728$	$10$	$0.2$	$0.05$	$15$	$2$	$2$
PB	$450$	$3.65$	$11$	$0.4$	$0.05$	$10$	$5$	$5$

Table 1: True and estimated vehicle parameters

	$x_{1}$	$x_{2}$	$x_{3}$	$x_{4}$	$x_{5}$	$x_{6}$	$w_{x}$	$w_{y}$
$\mathbf{x}_{0}$ Lower Bound	$-50$	$150$	$-\frac{\pi}{6}$	$-5$	$-20$	$-0.5$	$-4$	$-1$
$\mathbf{x}_{0}$ Upper Bound	$50$	$200$	$\frac{\pi}{6}$	$5$	$-2.5$	$0.5$	$4$	$1$
$\mathbf{x}_{f}$ Lower Bound	$-4$	$0$	$-\frac{\pi}{18}$	$-3-w_{x}$	$-3-w_{y}$	$-$	$-$	$-$
$\mathbf{x}_{f}$ Upper Bound	$4$	$1$	$\frac{\pi}{18}$	$3-w_{x}$	$-w_{y}$	$-$	$-$	$-$
$\mathbf{x}_{f}$ Target Value	$0$	$<1$	$0$	$-w_{x}$	$-w_{y}$	$-$	$-$	$-$

Table 2: Trajectory Characteristics

Model	Average State Deviation
	$x_{1}$	$x_{2}$	$x_{3}$	$x_{4}$	$x_{5}$	$x_{6}$
S-VAE-25	$3.20\pm 1.70$	$5.08\pm 3.07$	$0.12\pm 0.08$	$1.69\pm 0.97$	$1.60\pm 0.81$	$0.029\pm 0.017$
MI-VAE-25	$\mathbf{1.27\pm 1.31}$	$\mathbf{1.95\pm 2.22}$	$\mathbf{0.03\pm 0.04}$	$\mathbf{0.59\pm 0.51}$	$\mathbf{0.67\pm 0.71}$	$\mathbf{0.008\pm 0.008}$
S-VAE-1000	$2.26\pm 2.18$	$2.95\pm 3.18$	$0.09\pm 0.09$	$1.49\pm 1.37$	$0.96\pm 0.99$	$0.024\pm 0.021$
MI-VAE-1000	$\mathbf{1.72\pm 1.54}$	$\mathbf{2.92\pm 3.69}$	$\mathbf{0.07\pm 0.06}$	$\mathbf{1.12\pm 0.98}$	$\mathbf{0.88\pm 1.05}$	$\mathbf{0.018\pm 0.014}$

Table 3: Mean and standard deviation of average state deviation for

\mathbf{x}_{1}

\mathbf{x}_{6}

over 1000 generated trajectories based on predicted state and integrated state using predicted control input.

Panel of Figure 5 — Figure 5: Sample outputs of the S-VAE for 25 real-world samples

Panel of Figure 6 — Figure 6: Sample outputs of the S-VAE for 1000 real-world samples

Panel of Figure 7 — Figure 7: Scatter plot of first three principal components (P.C.) of data points in the OTD and generated datasets for S-VAE and MI-VAE with $N_{\mathrm{D}}=25$ and $N_{\mathrm{S}}=1000$ .

Panel of Figure 8 — Figure 8: Scatter plot of first three principal components (P.C.) of data points in the OTD and generated datasets for S-VAE and MI-VAE with $N_{\mathrm{D}}=1000$ and $N_{\mathrm{S}}=1000$ .

	Mean			Variance			Skewness			Kurtosis
OTD	$4.598\textsc{~e}{4}$	$-5.427\textsc{~e}{2}$	$2.142\textsc{~e}{2}$	$2.165\textsc{~e}{7}$	$1.401\textsc{~e}{8}$	$7.319\textsc{~e}{7}$	$-0.6578$	$-0.05233$	$-1.086$	$3.752$	$2.173$	$4.831$
S-VAE	$-4.650\textsc{~e}{4}$	$1.247\textsc{~e}{2}$	$1.440\textsc{~e}{2}$	$5.573\textsc{~e}{6}$	$3.721\textsc{~e}{7}$	$1.779\textsc{~e}{7}$	$0.1665$	$-0.4627$	$-0.1275$	$2.802$	$2.936$	$2.468$
MI-VAE	$4.640\textsc{~e}{4}$	$2.310\textsc{~e}{2}$	$3.101\textsc{~e}{2}$	$1.108\textsc{~e}{7}$	$8.290\textsc{~e}{7}$	$3.460\textsc{~e}{7}$	$-0.2651$	$-0.6804$	$-0.1179$	$2.662$	$3.109$	$1.970$

Table 4: Statistical moments of datasets generated by the VAE models for

N_{\mathrm{D}}=25

	Mean			Variance			Skewness			Kurtosis
OTD	$4.598\textsc{~e}{4}$	$-5.427\textsc{~e}{2}$	$2.142\textsc{~e}{2}$	$2.165\textsc{~e}{7}$	$1.401\textsc{~e}{8}$	$7.319\textsc{~e}{7}$	$-0.6578$	$-0.05233$	$-1.086$	$3.752$	$2.173$	$4.831$
S-VAE	$4.583\textsc{~e}{4}$	$4.464\textsc{~e}{2}$	$-1.223\textsc{~e}{2}$	$1.901\textsc{~e}{7}$	$1.044\textsc{~e}{8}$	$6.786\textsc{~e}{7}$	$-0.4046$	$-0.09715$	$0.7414$	$2.8917$	$2.222$	$3.160$
MI-VAE	$4.628\textsc{~e}{4}$	$4.858\textsc{~e}{2}$	$-1.765\textsc{~e}{2}$	$1.954\textsc{~e}{7}$	$1.029\textsc{~e}{8}$	$6.055\textsc{~e}{7}$	$-0.4377$	$-0.07340$	$0.7093$	$3.011$	$2.221$	$3.342$

Table 5: Statistical moments of datasets generated by the VAE models for

N_{\mathrm{D}}=1000

Panel of Figure 9 — Figure 9: Flow of training data to generate BC and BPPO policies.

Panel of Figure 10 — Figure 10: Sample outputs of the MI-VAE for 25 real-world samples

Panel of Figure 11 — Figure 11: Sample outputs of the MI-VAE for 1000 real-world samples

Dataset	Dataset Metrics		BC		BPPO
	$\Delta t\sum\|{u}\|$	$\sum\|\mathbf{x}_{\mathrm{f}}^{\mathrm{err}}\|$	$\Delta t\sum\|{u}\|$	$\sum\|\mathbf{x}_{\mathrm{f}}^{\mathrm{err}}\|$	$\Delta t\sum\|u\|$	$\sum\|\mathbf{x}_{\mathrm{f}}^{\mathrm{err}}\|$
RL-25	$3.99\textsc{~e}{4}\pm 4.12\textsc{~e}{3}$	$4.8\pm 4.5$	$3.48\textsc{~e}{4}\pm 9.54\textsc{~e}{3}$	$19.4\pm 17.0$	$4.00\textsc{~e}{4}\pm 9.98\textsc{~e}{3}$	$14.4\pm 16.5$
RL-Hybrid-25	$5.69\textsc{~e}{4}\pm 6.21\textsc{~e}{3}$	$3.0\pm 0.9$	$5.30\textsc{~e}{4}\pm 2.85\textsc{~e}{4}$	$50.0\pm 27.3$	$5.37\textsc{~e}{4}\pm 2.88\textsc{~e}{4}$	$50.2\pm 26.9$
S-VAE-25	$3.36\textsc{~e}{4}\pm 3.57\textsc{~e}{3}$	$3.6\pm 0.6$	$\mathbf{3.45\textsc{~e}{4}\pm 5.12\textsc{~e}{3}}$	$15.3\pm 11.7$	$\mathbf{3.35\textsc{~e}{4}\pm 5.02\textsc{~e}{3}}$	$15.0\pm 11.1$
MI-VAE-25	$3.53\textsc{~e}{4}\pm 5.38\textsc{~e}{3}$	$3.8\pm 0.8$	$3.57\textsc{~e}{4}\pm 5.43\textsc{~e}{3}$	$\mathbf{10.2\pm 21.2}$	$3.69\textsc{~e}{4}\pm 5.47\textsc{~e}{3}$	$\mathbf{7.6\pm 14.1}$
RL-1000	$3.87\textsc{~e}{4}\pm 5.95\textsc{~e}{3}$	$4.6\pm 2.0$	$4.04\textsc{~e}{4}\pm 5.35\textsc{~e}{3}$	$\mathbf{3.4\pm 2.1}$	$4.22\textsc{~e}{4}\pm 5.62\textsc{~e}{3}$	$\mathbf{2.8\pm 1.6}$

Table 6: Average cost and final state deviation for 200 trajectories generated using BC and BPPO policies.

Panel of Figure 12 — Figure 12: Reward with Std. Dev. (T-lines) and Success Rate across datasets for $N_{\mathrm{D}}=25$ .

Panel of Figure 13 — Figure 13: Reward with Std. Dev. and Success Rate for $N_{\mathrm{D}}=1000$ .

Physics-Informed Generative Modeling

Physics-informed generative modeling uses physical laws and data-driven generators to create plausible samples and solve scientific inverse problems.

See all 107 papers

Towards Reinforcement Learning for Space Robotics: Fast Training and Reliable Real-World Transfer
Oct 16, 2026Abhishek Naik, Michael Wu, Michael O'Sullivan +2RL for Robotics RL for Legged Locomotion
National Research Council Canada · University of Waterloo · University of Ottawa +1
Reward-Free Continual Adaptation for Resilient Space Robots
Aug 24, 2026Andrej Orsula, Miguel Olivares-Mendez, Carol MartinezReinforcement Learning Continual Robot Learning
University of Luxembourg
Improving Generalization and Robustness in Offline Reinforcement Learning via Boundary-Aware Data Augmentation
Aug 22, 2026Gong Gao, Weidong Zhao, Xianhui LiuOffline RL Synthetic Data Augmentation
School of Computer Science, Tongji University, China

Oct 16, 2026J/K move · Enter open · S save

Towards Reinforcement Learning for Space Robotics: Fast Training and Reliable Real-World Transfer

Abhishek Naik, Michael Wu, Michael O'Sullivan, Colin Bellinger+1

National Research Council Canada · University of Waterloo · University of Ottawa+1

RL for Robotics RL for Legged Locomotion Sim-to-Real Transfer Data-Efficient Robot Learning

Open paper AskPDF arXiv

This work demonstrates the use of a fast simulation-based training framework to learn control policies using reinforcement learning. The hardware requirements are minimal and the simulation-learned policies transfer reliably to the real world. With this setup, we show that a quadruped robot can learn to traverse diverse real-world terrains reliably---including in a lunar-analog facility---after less than 20 minutes of training on a single consumer-grade GPU. These results provide a glimpse of a future where reinforcement learning is a crucial tool in developing more capable and robust space robotic systems. Video demonstrations are available at https://youtu.be/vUFJiZdJaQ0.

Abstract

Physics-Informed Generative Modeling

Explore similar work

Towards Reinforcement Learning for Space Robotics: Fast Training and Reliable Real-World Transfer

Reward-Free Continual Adaptation for Resilient Space Robots

Improving Generalization and Robustness in Offline Reinforcement Learning via Boundary-Aware Data Augmentation