cs.ROOct 13, 2026

Learning in Orbit: Onboard Training and Closed-Loop Attitude Control with a Thermal Segmentation Model on a Spacecraft

Authors: Elena Sorina Lupu, Hannah Grauer, Phaedrus Leeds, Nikhil Ranganathan, Connor Lee, Sulekha Kishore, Cailyn Smith, Shon Cortes, 

Organizations: California Institute of Technology, Pasadena, CA 91125, USA · The Aerospace Corporation, 2310 E. El Segundo Blvd., El Segundo, CA 90245-4609, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Deep space missions increasingly require spacecraft capable of autonomously operating in environments with communication delays and bandwidth limitations. A key enabler of such autonomy is the ability to run learning-based algorithms directly onboard. In this work, we present a flight deployment on a 6U CubeSat that demonstrates (1) onboard training and inference of a thermal segmentation model on an NVIDIA Jetson TX2 NX GPU in space and (2) closed-loop attitude control using the segmentation model on a real-time, thermal image feed to track the Moon. In addition to this demonstration, we implement ROS 2 Foxy as a data transport middleware and integrate an open-source file system that provides strong data integrity guarantees and self-healing features. Over the course of approximately one year, we performed 12 experiments on the spacecraft, advancing the state of the art in autonomous spacecraft systems by showing how modern learning-based pipelines, commercially available hardware, and open-source software can operate reliably under the compute and environmental constraints of Low Earth Orbit (LEO).

Figures & tables

Figure 1: The spacecraft, alongside images captured in flight. At the top, three Moon images captured by the Boson thermal camera are shown. On the left, one image taken of the dark sky exhibiting streak artifacts (see Section 6-6.4 ). In the middle, we show an example of a thermal image acquired in orbit, with the Earth’s thermal emission and the Moon as a distinct thermal signature. The spacecraft is illustrated on the right.
MissionPlatformLaunch DateHardwareCapability
Remote Agent (RA) on Deep Space 1 (DS1) [ 15 ]Medium-sized interplanetary spacecraftOct. 1998RAD6000 [ 25 ]“The first AI-based closed loop autonomous control system to take control of a spacecraft” [ 25 ]
NASA’s Earth Observing-1 (EO-1) [ 16 ]Medium-sized Earth-observation satelliteNov. 2000Mongoose-V [ 26 ]First ML in space [ 27 ] (used support vector machine (SVM) and later hyperspectral unmixing [ 17 ] )
CNES’s DEMETER [ 18 ]SmallSatJune 2004Analog Devices ADSP 21020 DSP“First onboard neural network dedicated to the scientific processing of wave data”, Time Delay Neural Network [ 19 ]
Intelligent Payload Experiment [ 20 ]CubesatDec. 2013Gumstix Earth StormML random decision forests to classify images onboard and CV visual salience software to extract interesting regions for downlink in acquired imagery [ 28 ]
OPS-SAT [ 11 ]CubeSatDec. 2019Altera Cyclone V SX SoC (FPGA)CNN image classification, k-means, FDIR model training [ 11 ] . “First re-training of an onboard AI model with live in-flight data”
Phi-Sat-1 [ 12 ]CubeSatSep. 2020Intel Movidius board with a Myriad II chip (VPU)Inference of a CNN in space [ 12 ]
Table 1: Flown missions with onboard machine learning or AI capabilities (based on publicly available information). Early missions without neural networks are included for historical context as pioneering AI-driven autonomy in space; certain claims are reported as stated in the literature.
Figure 2: Overview of our method. On Earth, we perform distillation of the VFM into the Fast-SCNN network using Algorithm 1 . As interfaces, we downlink ROS 2 bags and other logs from the satellite, as well as images, and uplinked labels after annotation. In space, we perform both training and inference of images captured by the onboard camera. We also close the loop on the segmented images to enable the spacecraft to point towards the Moon. The “snowflake” symbol represents the fact the DINO model is frozen (no gradients flow through the network).
Figure 3: Timeline of the different experiments run onboard the spacecraft since its launch. In total, we ran 12 experiments, the majority being inference and dynamic targeting experiments. We have run three training experiments on the TX2 NX’s GPU on Moon images with labels uploaded from the ground (Aug. 8th, 2025 and Dec. 15th, 2025). The weights from Aug. 2025 were used for the Sep., Nov., and Dec. experiments, and the ones from Dec. for the Jan. experiments. The Moon icon signifies in which phase the Moon was when the experiment was performed. See Table 2 for the parameters used in the Jan. 2026 tracking experiments. For discussion on these parameters, see Section 5-5.2 .
Figure 4: Performance of the training and inference experiments in space (August 8th). (top plot) Thermal results for the GPU. The experiment was run while the spacecraft was in eclipse. (second plot) GPU usage during training and inference. (third plot) Thermal results for the NVMe. For the first 3 plots, the data are sampled at 1 Hz. (bottom plot) Training running in two cycles for 60 epochs total.
Experiment DateBoson Frame RateKF Prediction RateControl GainLow-Pass FilterBoson Frame Buffer
Run 3: Jan. 6th, 20261 Hz10 Hz2None4
Run 4: Jan. 9th, 20261 Hz10 Hz2None1
Run 5: Jan. 10th, 20262 Hz2 Hz1Enabled1
Run 6: Jan. 26th, 20262 Hz2 Hz1/2Disabled1
Run 7: Jan. 30th, 20262 Hz2 Hz1/2Disabled1
Table 2: Parameters used for the closed-loop control experiments in the experiments performed in January 2026.
Experiment DateMeanFinalΔ\Delta errΔ\Delta XΔ\Delta Y
Run 2: Dec. 10, 2025167.8197.2127.0117.671.2
Run 3: Jan. 06, 2026199.7188.1612.9970.0278.9
Run 4: Jan. 09, 2026579.82356.62335.54444.171.0
Run 5: Jan. 10, 2026176.837.0726.81481.782.4
Run 6: Jan. 26, 20267.44.020.134.16.8
Run 7: Jan. 30, 202613.51.23.56.93.7
Table 3: Quantitative results of the pointing accuracy for dynamic targeting experiments. Values are reported in pixels.
Figure 5: Performance on the dynamic targeting task. (a) Results from the dynamic targeting experiment runs. (top) GPU temperature, (middle) CPU usage in total over the 4 cores, (lower) GPU load. The dots show when image inference was performed. For the runs on and after Jan. 10th, we increased the inference frequency from 1 Hz to 2 Hz. (b) Position of the Moon during the dynamic targeting phase (first row contains a sequence of segmented Moon images and the second row represents a sequence of true thermal images). The Moon was moving towards the center of the image, showing that Run 7 performs well. (c) Performance from two runs (image coordinates, quaternions, and reaction wheel commands). For a discussion on the cause of the oscillating behavior on Run 5, see Section 5-5.2 .
Figure 6: Example images from four datasets. The first dataset (a) consists of images taken inside our spacecraft simulator, while the second (b) is an open-source collection of spacecraft imagery [ 53 ] . The remaining two datasets (c, d) are thermal: one includes in-orbit observations of the Moon, and the other contains synthetically generated images based on Landsat data with overlaid spacecraft.
ModelAllBodySolar PanelAntennaBackground
Fast-SCNN0.6830.7050.7070.3470.974
DINO-distilled + Fast-SCNN (ours)0.7560.7640.7680.4800.982
Table 4: Spacecraft part segmentation performance (mIoU) on the Adelaide dataset.
ModelAllBodySolar PanelBackground
Fast-SCNN0.9710.9680.9580.986
DINO-distilled + Fast-SCNN (ours)0.9720.9690.9610.987
Table 5: Spacecraft part segmentation model performance (mIoU) on our custom spacecraft simulator dataset.
Model% of Training Data Used
 75%50%12.5%
Fast-SCNN0.9710.9660.933
DINO-distilled + Fast-SCNN (ours)0.9720.9690.947
Table 6: Effect of training set size (% of overall training set) on short-range spacecraft part segmentation (mIoU) for the custom dataset.
Model% of Training Data Used
 75%50%12.5%
Fast-SCNN0.6830.5980.537
DINO-distilled + Fast-SCNN (ours)0.7560.7220.636
Table 7: Effect of training set size (% of overall training set) on short-range spacecraft part segmentation (mIoU) for the Adelaide dataset.
ModelTaskInference Time (ms)Params (M)
  TX2Titan RTX
Fast-SCNNSemantic Segm.49.393.541.13
DINO-distilled + Fast-SCNN (ours)Semantic Segm.92.564.291.55
Table 8: Inference time (ground results) on two different platforms.
More in this topic

Spacecraft Attitude Determination

See all 10 papers

Explore similar work

CardsList
Sep 25, 2026

Resource-Aware Federated Mixture-of-Experts with Adaptive Pruning for Onboard Learning in LEO Satellite Constellations

Low-Earth-orbit (LEO) satellites are increasingly expected to perform onboard learning for applications such as disaster response and environmental monitoring. However, conventional federated learning (FL) is ill-suited to onboard satellite learning, as it assumes computational, memory, and communication resources beyond the capabilities of resource-constrained LEO platforms, often necessitating the transmission of raw imagery to ground stations. We present COSMIC-FL, a resource-aware FL framework for efficient onboard learning in LEO satellite constellations. COSMIC-FL introduces two complementary Mixture-of-Experts (MoE) architectures: a Sliced design that shares backbone representations while activating task-specific channel subsets, and a Modular design that employs lightweight gating to route inputs to physically separated expert networks. A semantic class-to-expert mapping enables each satellite to train, update, and communicate only the expert paths relevant to its local data. To further improve efficiency, COSMIC-FL integrates staged optimization with three structured pruning strategies: server-side pruning, client-side fixed-ratio pruning with mean-vote aggregation, and adaptive client-side per-layer pruning based on aggregated importance and a MAD-based gap criterion. Combined with semantic expert routing, these techniques jointly adapt computation and model sparsity to both data semantics and layer importance, yielding a favourable accuracy--efficiency trade-off for heterogeneous space platforms. Experiments on six image classification benchmarks under highly non-i.i.d. settings show that COSMIC-FL maintains competitive accuracy while reducing communication, computation, and energy consumption by up to 80% over SOTA FL methods. We further validate COSMIC-FL on an NVIDIA Jetson AGX Orin, confirming its efficiency gains under realistic embedded deployment constraints.

Mohamed Shaaban, Mohamed Elmahallawy, Marius Bernahrndt +1

School of Engineering and Applied Science, Washington State University, Richland, WA 99354, USA · Institute for Software Technology, German Aerospace Center (DLR), Porz, 51147 Cologne, Germany

Federated LearningMixture of ExpertsLEO Satellite NetworksNon-IID Federated LearningStructured Pruning
AskPDF
Feb 24, 2026

Onboard-Targeted Segmentation of Straylight in Space Camera Sensors

This study details an artificial intelligence (AI)-based methodology for the semantic segmentation of space camera faults. Specifically, we address the segmentation of straylight effects induced by solar presence around the camera's Field of View (FoV). Anomalous images are sourced from our published dataset. Our approach emphasizes generalization across diverse flare textures, leveraging pre-training on a public dataset (Flare7k++) including flares in various non-space contexts to mitigate the scarcity of realistic space-specific data. A DeepLabV3 model with MobileNetV3 backbone performs the segmentation task. The model design targets deployment in spacecraft resource-constrained hardware. Finally, based on a proposed interface between our model and the onboard navigation pipeline, we develop custom metrics to assess the model's performance in the system-level context.

Riccardo Gallon, Fabian Schiemenz, Alessandra Menicucci +1

Department of Space Systems Engineering, Faculty of Aerospace Engineering, Kluyverweg 1, HS Delft, 2629, Netherlands · Airbus Defence and Space GmbH, Claude-Dornier Straße, Immenstaad am Bodensee, 88090, Germany

Efficient Semantic SegmentationImage SegmentationSemantic Segmentation
AskPDF
Jun 5, 2026

NAVI-Orbital: First In-Orbit Demonstration of a Zero-Shot Vision-Language Model for Autonomous Earth Observation

As Earth Observation data generation outpaces downlink bandwidth and human-in-the-loop processing, a widening gap has emerged between onboard collection and actionable ground intelligence. This paper presents NAVI-Orbital, a software system deployed on a Low Earth Orbit (LEO) spacecraft. On April 16, 2026, NAVI-Orbital achieved what is, to the authors' knowledge, the first in-orbit demonstration of a vision-language model performing autonomous multi-modal inference entirely onboard. NAVI-Orbital uses a local vision-language model (Gemma 3) to classify each captured scene, produce a text description of its content and the relationships between its features, and respond to operator follow-up via natural-language dialogue. The system is re-tasked through plain-English prompts in place of conventional command sequences, and is orchestrated by a graph-based state machine (LangGraph) coordinating dedicated agents for detection and dialogue. Results across ground benchmarking (88.16% accuracy on the 7,960-image curated AID benchmark), Flatsat validation, and live in-orbit captures of newly acquired, previously unseen Earth imagery (including uncorrected YAM-9 imagery, processed onboard with hardware-accelerated GPU inference and no fine-tuning for the flight instrument) demonstrate the feasibility of running foundation models on satellite-class edge computers to invert the conventional acquire-then-downlink-everything bandwidth profile through semantic compression of Earth observations in-orbit.

Juan Manuel Delfa Victoria, Taran Cyriac John, Andrew W. Herson

Jet Propulsion Laboratory California Institute of Technology Pasadena, CA, USA · School of Engineering and Computer Science Victoria University of Wellington Wellington 6140, New Zealand · Loft Orbital 321 11th St, San Francisco, CA 94103

Remote Sensing Image UnderstandingVision-Language ModelsVLM Inference
AskPDF