Monocular Navigation Relative to Unknown Spacecraft Using a Transformer-Aided Kalman Filter

Pol Francesch Huc; Simone D'Amico

cs.ROOct 5, 2026

Monocular Navigation Relative to Unknown Spacecraft Using a Transformer-Aided Kalman Filter

Authors: Pol Francesch Huc, Simone D'Amico

Organizations: Department of Aeronautics and Astronautics Stanford University Stanford, CA 94305

Abstract

This work presents a novel learning-based pipeline for pose estimation of unknown spacecraft using only monocular images from a single servicer. The approach combines a transformer-based neural network with a Multi-State Constraint Kalman Filter (MSCKF) to estimate the pose (i.e., position and orientation of the target spacecraft relative to the camera) throughout rendezvous and proximity operations. Unlike existing vision-based methods that require prior knowledge of the target shape or inertia properties, rely on additional sensing modalities such as depth, lidar, or stereo, or only recover translation up to scale, the proposed pipeline generalizes to previously unseen spacecraft using a single monocular camera. The transformer network estimates the odometry, the change in pose between images up to scale, from SuperPoint features matched by LightGlue. The MSCKF uses these pseudo-measurements along with an orbit and attitude kinematics model to estimate the pose of the target. In particular, the relative orbit elements, the target's attitude with respect to the servicer's camera, and the associated angular velocity are estimated directly by the filter. Given the monocular approach and short distance to the target, the full observability of the range to the target is recovered via attitude maneuvers by the servicer. The method is trained and evaluated on a re-rendered high-resolution version of the SPE3R dataset, which includes synthetic images of 103 spacecraft. Eleven of these spacecraft are held out during training to evaluate the generalization to unseen targets. Monte Carlo simulations are then used to evaluate the navigation pipeline on rendered trajectories of the held out spacecraft. The results demonstrate that learned vision pipelines as a front-end for Kalman filters provide median errors of 3.7° in attitude and 2.2% of range in ROE when navigating about unknown targets.

Figures & tables

Fig. 1: Proposed pipeline for pose estimation where a servicer captures images of a target, and uses those images to generate three measurements. These measurements leverage feature extraction and matching to estimate odometry between the current image and the previous one, feature tracks along the sliding window, and loop closure between the current image and candidates in the selected keyframes. The odometry is estimated using the novel Spacecraft Odometry Network (SON). These three measurements provide short- and long-baseline relative information that is leveraged by a Multi-State Constraint Kalman Filter that then tracks the desired pose states. For every image in the sliding window and for every keyframe, the filter maintains a clone of the target attitude estimate.

Fig. 2: Features extracted by SuperPoint (top) and matched by LightGlue (bottom) from a pair of images of the ACRIMSAT satellite.

Fig. 3: Architecture of SON, where feature estimates from SuperPoint are matched with LightGlue and the descriptors $d$ and their positional encodings $p$ are used to regress the odometry between images $A$ and $B$ .

Fig. 4: Sample comparison images of the original SPE3R dataset (top) and the new re-rendered version SPE3R+ (bottom). Note that the re-rendered versions of the images are cropped in this figure to match the scale of the original dataset.

Group	Parameter	LightGlue	SON
Data	Crop	Margin 1.3; $768\times 768$ px	Same crop; cached matches
	Input	Frozen SuperPoint features; $\dim(\mathbf{d}_{i})=256$ ; $\leq 2048$ features	LightGlue matches; $\mathbf{d}_{i}$ , $\mathbf{p}_{i}$
	Pairs	120,000/epoch; total rotation change $\leq 120^{\circ}$ ; bins $\{0,2,4,6,8,12,20,30,45,60,75,90,120\}^{\circ}$	304,765 train pairs; viewing direction change $\leq 30^{\circ}$ ; $\{0,2,4,6,8,12,20,30\}^{\circ}$ rotation bins
Model	Backbone	Standard LightGlue	$N_{t}=9$ ; 4 heads; $N_{p}=4$
Model	Head	Match-assignment head	Two MLP heads; widths $512$ , $128$
Loss	Objective	Metric depth and pose; 3/5 px match thresholds; 5 px epipolar	Equations 6 – 7

TABLE I: LightGlue and Spacecraft Odometry Network (SON) training configuration.

Group	Quantity	Symbol	Value
Truth, imaging, and control
Orbit	Initial elements	$\boldsymbol{\alpha}_{S,0}$	$a=7078.135$ km, $e=0.001$ , $i=98.19^{\circ}$ , $\Omega=189.89086^{\circ}$ , $\omega=M=0^{\circ}$
Orbit	Epoch	$t_{0}$	2011-07-18 01:00:00
Truth	Dynamics	–	$120\times 120$ gravity field; drag; SRP; Sun/Moon gravity; gravity-gradient, drag, and SRP torques
Image	Cadence	$\Delta t$	$5$ s
Control	Boresight roll	–	$0.5^{\circ}/\mathrm{s}$

TABLE II: Simulation, control, and filter configuration. Noise entries are per-component process variances or measurement standard deviations, as defined in the text.

Group	Quantity	Symbol/criterion	Value
Campaign	Targets/trials	$(N_{T},N_{\mathrm{MC}})$	$(11,50)$
Campaign	Horizon/window	$(T_{\mathrm{MC}},\mathcal{I}_{\mathrm{eval}})$	$2P_{\mathrm{orb}}$ ; final $25\%$
Truth	Attitude	$\mathbf{q}_{T/C,0}$	$\mathcal{U}(\mathrm{SO}(3))$
	Angular velocity	${}^{T}\boldsymbol{\omega}_{T/C,0}$	Direction $\sim\mathcal{U}(\mathbb{S}^{2})$ ; magnitude $\sim\mathcal{U}(0.6,1.2)^{\circ}/\mathrm{s}$
	Along-track ROE	$a\delta\lambda_{0}$	$-\max(\mathcal{U}(0.5,1.5),1)\cdot\rho_{\text{nom}}$
	Remaining ROE	$a[\delta a,\delta e_{x},\delta e_{y},\delta i_{x},\delta i_{y}]_{0}$	$\mathcal{U}(-1.5,0.5)\%$ , $\mathcal{U}(-1.5,1.5)\%$ , $(1+\mathcal{U}(-1.5,1.5))\%$ , $\mathcal{U}(-1.5,1.5)\%$ , $(-1+\mathcal{U}(-1.5,1.5))\%$ of $\rho_{\text{nom}}$

TABLE III: Monte Carlo campaign and evaluation configuration.

	HPatches					MegaDepth1500			SPE3R+ (test)
	P	AUC - RANSAC		AUC - DLT		P	R	RANSAC AUC	P	R	#M	Inl	RANSAC AUC
Weights		@1px	@5px	@1px	@5px			5° / 10° / 20°					5° / 10° / 20°
Original	94.1	34.5	68.5	35.2	77.6	83.5	90.4	50.3 / 67.6 / 80.5	28.7	32.7	89	27	0.0 / 0.2 / 0.8
Original + BBox	94.1	34.5	68.5	35.2	77.6	83.5	90.4	50.3 / 67.6 / 80.5	32.9	15.7	346	87	0.5 / 1.6 / 4.0
Fine-Tuned + BBox	88.2	32.3	64.7	30.3	70.5	62.1	58.5	31.6 / 46.2 / 57.7	59.5	49.1	694	274	3.8 / 8.6 / 17.0

TABLE IV: The performance of the LightGlue feature matcher (using SuperPoint features) is compared with different weights: the original weights released by Lindenberger et al., and the ones fine-tuned using SPE3R+. On HPatches Precision (P) and homography estimation accuracy using RANSAC and DLT is compared. On MegaDepth1500 and SPE3R+ match Precision (P) and Recall (R) as well as the pose accuracy from RANSAC is compared. On SPE3R+ the number of matches (#M) and the number of RANSAC inliers (Inl) are also reported.

Fig. 5: Performance of fine-tuned LightGlue and the classical odometry estimator on SPE3R+ data splits.

		Train		Val		Test
		$\epsilon_{rot}$ $[^{\circ}]$	$\epsilon_{\hat{u}}$ $[^{\circ}]$	$\epsilon_{rot}$ $[^{\circ}]$	$\epsilon_{\hat{u}}$ $[^{\circ}]$	$\epsilon_{rot}$ $[^{\circ}]$	$\epsilon_{\hat{u}}$ $[^{\circ}]$
Baseline		$10.1^{+7.8}_{-4.8}$	$44.8^{+35.7}_{-25.1}$	$12.2^{+8.7}_{-5.9}$	$43.2^{+35.6}_{-24.8}$	$9.9^{+7.7}_{-4.7}$	$45.2^{+35.2}_{-24.7}$
SON	MLP Only	$12.5^{+6.2}_{-4.5}$	$44.0^{+26.9}_{-18.2}$	$16.2^{+5.4}_{-5.0}$	$44.0^{+28.1}_{-18.7}$	$11.7^{+6.6}_{-4.3}$	$46.6^{+27.3}_{-19.0}$
	1 Layer, Rand Init.	$9.6^{+4.9}_{-3.4}$	$23.8^{+13.4}_{-9.3}$	$12.7^{+4.9}_{-4.0}$	$26.4^{+17.5}_{-10.8}$	$10.4^{+5.4}_{-3.6}$	$32.4^{+21.9}_{-13.4}$
	1 Layer	$7.1^{+3.9}_{-2.6}$	$19.6^{+10.6}_{-7.5}$	$9.5^{+4.5}_{-3.3}$	$22.0^{+14.1}_{-8.9}$	$8.0^{+4.7}_{-3.0}$	$26.8^{+18.2}_{-10.9}$
	9 Layers	$\mathbf{2.1^{+1.7}_{-0.9}}$	$\mathbf{3.1^{+2.0}_{-1.3}}$	$\mathbf{2.8^{+2.1}_{-1.2}}$	$\mathbf{3.4^{+2.8}_{-1.5}}$	$\mathbf{2.6^{+2.2}_{-1.2}}$	$\mathbf{4.4^{+3.9}_{-2.0}}$

TABLE V: Median rotation and translation unit vector errors for the different data splits of SPE3R+ from the baseline and different SON architectures. Interquartile range shown through the super and subscript.

Fig. 6: Performance of fine-tuned LightGlue and SON on SPE3R+ data splits.

		Pairs [%]	Baseline	SON		Baseline
Trajectory	Domain	$N_{m}\geq 8$	fails [%]	$\epsilon_{rot}$ $[^{\circ}]$	$\epsilon_{\hat{u}}$ $[^{\circ}]$	$\epsilon_{rot}$ $[^{\circ}]$	$\epsilon_{\hat{u}}$ $[^{\circ}]$
ROE1	Noise-free	100	0	$\mathbf{1.2^{+0.6}_{-0.4}}$	$\mathbf{3.6^{+1.4}_{-1.3}}$	$\mathbf{1.2^{+0.7}_{-0.5}}$	$10.9^{+10.4}_{-5.9}$
	Synthetic	69	38	$\mathbf{3.3^{+5.2}_{-2.0}}$	$\mathbf{7.4^{+29.5}_{-3.6}}$	$\mathbf{3.3^{+1.6}_{-1.8}}$	$45.6^{+40.4}_{-25.7}$
	Lightbox	50	17	$\mathbf{3.0^{+5.2}_{-1.4}}$	$\mathbf{11.4^{+145.7}_{-6.7}}$	$4.0^{+1.0}_{-1.8}$	$58.9^{+31.4}_{-27.3}$
ROE2	Noise-free	100	1	$\mathbf{0.8^{+0.4}_{-0.3}}$	$\mathbf{5.6^{+4.4}_{-2.5}}$	$0.9^{+0.7}_{-0.4}$	$24.4^{+20.4}_{-12.5}$
	Synthetic	91	14	$1.5^{+1.0}_{-0.6}$	$\mathbf{7.0^{+7.1}_{-3.1}}$	$\mathbf{1.3^{+1.1}_{-0.6}}$	$36.1^{+30.7}_{-20.3}$

TABLE VI: Performance of SON and the classical baseline on sequential images from the SHIRT [ 8 ] dataset. Noise-free image domain is a re-rendered version of SHIRT using the same renderer as SPE3R+. Errors (median with IQR) are reported for image pairs for which there are at least 8 feature matches, below which neither estimator provides a measurement. The proportion of failures of the baseline method on the pairs with at least 8 feature matches is also reported (SON always provides a measurement).

					Attitude $[^{\circ}]$			ROE $[m]$			$a\delta\lambda$ $[m]$
Satellite name	Converged $[\%]$	Meets Spec. $[\%]$	$\rho_{nom}$ $[m]$	$\bar{\rho}$ $[m]$	p50	p95	NEES	p50	p95	NEES	p50	p95	NEES
AIM	100	100	10.0	10.0	3.7	8.0	0.8	0.14	0.24	17.8	0.14	0.24	2.8
Aquarius	100	98	10.0	12.0	2.5	7.8	0.3	0.23	0.36	18.6	0.21	0.35	2.3
CYGNSS	96	94	5.0	5.6	5.8	16.0	3.6	0.06	0.12	15.5	0.05	0.11	1.8
EO-1	100	92	10.0	12.0	4.5	18.1	1.6	0.28	0.48	29.6	0.26	0.47	4.8
Europa Orbiter	100	100	20.0	22.0	3.1	5.8	0.6	0.49	0.87	17.6	0.49	0.86	3.6

TABLE VII: Performance of the whole pipeline on each of the SPE3R+ test-spacecraft on the Monte Carlo trajectories.

\rho_{nom}

is the designed standoff distance, and

\bar{\rho}

is the mean separation over the entire trajectory over the 50 trials.

Panel of Figure 14 — Fig. 7: View within Blender of SMART-1 (top) and a sample image from SPE3R+ of this spacecraft (bottom). A red bounding box is used to highlight the target in the image.

			Attitude $[^{\circ}]$			ROE			$a\delta\lambda$
Variant	Converged $[\%]$	Meets Spec. $[\%]$	p50	p95	NEES	p50	p95	NEES	p50	p95	NEES
Full pipeline	98	96	3.9	11.1	1.1	2.3%	7.4%	20.6	2.2%	7.2%	3.8
SON $\rightarrow$ Baseline	72	64	8.3	93.5	7.4	2.7%	35.0%	25.0	2.6%	34.9%	5.2
No roll	77	11	3.6	8.4	0.8	30.6%	71.7%	49.2	30.6%	71.7%	19.0
No feature tracks	12	0	20.0	69.0	15.6	73.7%	90.0%	600.4	73.6%	90.0%	315.7
No loop closure	72	44	15.6	64.6	10.4	2.3%	6.8%	20.9	2.2%	6.8%	4.1

TABLE VIII: Leave-one-out ablation. Each row removes the stated portion from the finished filter in Figure 1 . Statistics are given for the 200 trials per row (20 trials for the 10 targets, with SMART-1 removed).

Fig. 8: Error of the nominal state along with the $\pm 1\sigma$ uncertainty for one sample Monte Carlo experiment with the CYGNSS spacecraft. Boxed quantity represents the mean and standard deviation of the error for the last quarter of the trajectory.

Fig. 9: Error of the nominal state along with the $\pm 1\sigma$ uncertainty for one sample Monte Carlo experiment with the Galileo spacecraft. Boxed quantity represents the mean and standard deviation of the error for the last quarter of the trajectory.

Stage	Calls/step	Cost $[ms]$	RAM $[MB]$	VRAM $[MB]$
SuperPoint	1.00	113.3	308.9	304.8
LightGlue	2.77	172.2	272.6	303.3
SON odom.	1.00	81.5	4.2	35.0
SON loop	1.77	51.2	16.8	18.7
MSCKF update	1.00	229.0	287.7	–
Filter step		992	2518	–

TABLE IX: Inference cost measured as compute time, host memory (RAM), and GPU memory (VRAM) for the MSCKF filter. Cost is measured per call.

Explore similar work

Jul 15, 2026cs.CV

DreamSat-Pose: Spacecraft Pose Estimation from Single-View 3D Reconstructions and Learned 2D-3D Feature Matching

6-DoF pose estimation is a critical task in autonomous rendezvous and proximity operations. In the case of an unknown target, this task becomes challenging as it shall be paired with the reconstruction of the target shape model. In this article, we propose a novel framework for single-shot shape and pose estimation of unknown spacecraft objects. Given a single image, we first reconstruct a 3D shape model of the target, then estimate the relative six-degrees-of-freedom pose by learning dense 2D-3D correspondences. The image features are extracted using a frozen DINOv3 vision transformer, while the geometric features are computed from the reconstructed point cloud using a trainable dynamic graph convolutional neural network encoder. A dual-stream transformer matcher refines descriptors through alternating self- and cross-attention, producing soft correspondences that are passed to a Perspective-

n

-Point solver for pose recovery. We evaluate the method on the SPE3R dataset and consider FoundationPose as a representative baseline for current state-of-the-art capabilities. Results show reliable pose estimates achieving 0.157 degrees mean pointing error using only a single image and reconstructed geometry, demonstrating strong generalization to unseen spacecraft.

Josiane Uwumukiza, Jocelyn Zhao, Giovanni Lavezzi +5

Undergraduate Student, Wellesley College, Wellesley, MA 02481, USA · Undergraduate Student, Massachusetts Institute of Technology, Cambridge, MA 02139, USA · Research Scientist, Massachusetts Institute of Technology, Cambridge, MA 02139, USA +5

Category-Level Object Pose Estimation 3D Reconstruction Spacecraft Dinov3 Shapes Constraint Language

Ask PDF

May 29, 2026cs.CV

TALON: Token-Aligned Lightweight Adapters for 6-DoF Spacecraft Pose Estimation

Monocular 6-DoF spacecraft pose estimation methods predominantly process individual frames, discarding the temporal information present in an image sequence acquired during spacecraft manoeuvres. Few temporal approaches require full backbone fine-tuning or auxiliary optical flow networks, risking catastrophic forgetting or increasing computational cost, respectively. We propose TALON (Token-Aligned Lightweight adapters for Orbital Navigation): spatiotemporal 3D adapters injected before the self-attention layers of a frozen ViT vision transformer, combined with a patch-token alignment loss that geometrically grounds the adapted features to keypoint structure through a prototype-conditioned KL-divergence objective. Pre-attention placement allows the frozen attention to reason over temporally enriched tokens, achieving stronger performance with a single adapter per block than post-attention alternatives. The alignment loss shapes the intermediate representations so that each keypoint induces a spatially precise activation in the token field, while the framework adds less than 5% parameters to the frozen backbone. On SPADES dataset, TALON reduces the pose error by 50% over the prior state-of-the-art, and on SwissCube dataset it surpasses the prior best by 21.8% in ADD-0.1d accuracy. Zero-shot cross-domain evaluation from sim-to-real on SPARK real data reduces pose error by 4.7x, and ablations characterise the role of adapter depth across in-domain and cross-domain settings.

Abid Ali, Arunkumar Rathinam, Djamila Aouada

Category-Level Object Pose Estimation Representation-Alignment Loss Frozen Vision-Language Models

Ask PDF

Jun 10, 2026cs.CV

Precision-Aware Illumination-Disentangled Vision Transformer for Spacecraft 6D Pose Estimation

Vision sensors provide a lightweight solution for spacecraft proximity operations, but monocular spacecraft 6D pose estimation remains difficult under illumination variation, specular reflection, shadowing, weak texture, and background interference. These factors make local visual evidence spatially unreliable and can destabilize pose regression. This article proposes a Precision-Aware Illumination-Disentangled Vision Transformer (PAID-ViT) for robust spacecraft pose estimation.The proposed model separates pose-relevant structure tokens from illumination-sensitive appearance tokens, estimates patch reliability before pose aggregation, and uses foreground mask supervision to preserve silhouette cues. A parameter-free geometric recovery module converts normalized crop coordinates, log-depth, and a continuous 6D rotation representation into camera-frame rotation and translation. Experiments on SPEED+ V2, the SPEED+ validation/lightbox/sunlamp evaluation configuration used in this study, suggest that PAID-ViT reduces translation error and improves robustness in the challenging sunlamp domain, while ablation studies support the complementary roles of illumination disentanglement, reliability-aware token aggregation, mask supervision, and training-side regularization.

Zongwu Xie, Yifan Yang, Yonglong Zhang +3

School of Mechatronics Engineering, Harbin Institute of Technology, Harbin 150001, China

Self-Supervised Vision Transformers Vision Sensors Vision Transformer Transformer Architectures

Ask PDF