As on-orbit operations grow, so does the need for precise navigation around unknown resident space objects (RSOs), such as spacecraft, debris, and asteroids. Simultaneous Localization and Mapping (SLAM) algorithms are often studied for mapping an RSO and estimating the inspector's relative pose, but conventional cameras struggle during eclipse, while lidar, though illumination-robust, has higher SWaP. Thermal-infrared cameras track targets through difficult lighting but lack the resolution and feature-richness of visible cameras. Images of a target satellite in low Earth orbit are simulated in both visible and thermal bands, and fused at the pixel level to combine each sensor's strengths for front-end tracking. Thermal imagery is further exploited by a novel silhouette-based visual place recognition (VPR) method that detects loop closures without training, runs entirely on CPU, and outperforms all baselines. Integrated SLAM performance is compared using visible, thermal-infrared, and fused imagery across lighting conditions, trajectories, and place recognition methods; using fused imagery for front-end tracking and our thermal-silhouette place recognition method performs best. The results show that visible and thermal modalities each provide unique information that is exploited differently at various stages of navigation. These low-SWaP sensors could ultimately enable reliable proximity operations on smaller, lower-cost spacecraft.
Figures & tables
Figure 1: A target and inspector orbiting around a parent body, with the inspector characterizing the target. The Hill coordinate frame is centered at the target, with the x axis pointing radially outwards, the y axis pointing in the target’s direction of motion, and the z axis pointing in the direction of the target’s orbital angular momentum vector.
Figure 2: The SpaceX Dragon spacecraft seen during its approach to the ISS for the Crew-8 mission [ 5 ] . These two images show the visibility of the spacecraft during illumination by the Sun and during eclipse. These two images were taken approximately twenty minutes apart.
Figure 3: The ISS as seen from the TIR imager of the TriDAR vision system during rendezvous operations on STS-131 [ 55 ] .
Figure 4: Visualization of the β angle and its effect on illumination.
Figure 5: The rendering framework, showing the effects of each simulation component on the CloudSat model. Camera artifacts include lens glare, bloom, and noise.
Figure 6: Earth as simulated using Blender in the VIS and TIR bands.
VIS Camera
TIR Camera
Resolution, pixels
1024×1024
640×512
FoV (S1-S4)
25∘×25∘
31.25∘×25∘
FoV (S5)
8∘×8∘
10∘×8∘
FoV (S6)
2∘×2∘
2.5∘×2∘
Table 1: VIS and TIR camera properties used in the simulations. The field of view is changed depending on the scenario to keep the target large in the frame.
Scenario
Range
Illumination
Target Attitude
S1: CloudSat 15 m
15 m
Nominal illumination with eclipse
Inertially fixed
S2: CloudSat 15 m, Fully Lit
15 m
Perfectly illuminated from the inspector
Inertially fixed
S3: CloudSat 15 m, Rotating
15 m
Nominal illumination with eclipse
Spins opposite the inspector’s motion about the axis parallel to Earth’s rotation axis at 5\times10−4rad/s
S4: CloudSat 15–25 m
15–25 m
Nominal illumination with eclipse
Inertially fixed
S5: CloudSat 50 m
50 m
Nominal illumination with eclipse
Inertially fixed
S6: CubeSat 15 m, Rotating
15 m
Nominal illumination with eclipse
Spins opposite the inspector’s motion about the axis parallel to Earth rotation axis at 5\times10−4rad/s
Table 2: The six simulated scenarios used to render visible and thermal-infrared images.
Figure 7: Three unique relative trajectories are generated to create the scenarios. All trajectories are shown in the Hill frame over three orbits.
Relative Acceleration
Relative Angular Velocity
Noise Density
0
1\times10−6rad/s/Hz
Bias Instability
1\times10−5m/s2
1\times10−6rad/s
Random Walk
0
1\times10−6rad/s⋅Hz
Table 3: Error parameters used for the generation of simulated relative acceleration and relative angular velocity measurements. These are generated at a rate of 10 Hz.
Figure 9: Simulated imagery of the target spacecraft at specified times on the inspector’s trajectory for the various sensor types. This sequence shows the first orbit of Scenario 1 with CloudSat as the target. The full scenario has a similar sequence repeated twice more, for a total of three orbits. The first orbit of Scenario 6 is also shown with a 1U CubeSat as the target. Unlike in Scenario 1, the CubeSat is spinning in the inertial frame, causing the images in the following two orbits to be different from this first orbit.
Figure 10: Kimera tracks features on sequential images of the target spacecraft. The feature tracks are used to create a 3D map of the target and estimate the trajectory of the inspector.
Figure 11: The full pipeline used for dynamics propagation, rendering, fusion, and SLAM.
Figure 12: Pairs of frames from each of the 12 loop closure opportunities in both visible and thermal-infrared bands of Scenario 3. The frames from the visible camera are often in eclipse. Drastic illumination/temperature changes are also seen between the frames in each pair in both visible and thermal-infrared.
Figure 13: A visualization of the silhouette-based descriptor calculation process. In the right-most image, there are 16 rays shown, representing d=16 . The intersection of each ray and the silhouette boundary are found, represented by the circular markers. b=8 in this case, representing 8 annuli separated by the dashed green circles. With the center-most circle having an index of 0 and starting from the right-most ray moving counterclockwise, each circular marker falls into the annulus indexed 3232122333321123. We then find the lexicographically minimal rotation of these values to find the final descriptor: 1123323212233332.
Figure 14: The calculation of the descriptors with d=16 , b=8 , and n=3 . Since n=3 , three descriptors are found for this one silhouette. The red rays and circular markers represent the first descriptor, the turquoise represent the second, and the purple represent the third. This results in three unique descriptors found for this silhouette: 1123323212233332, 1123332211333322, and 1113333211233322.
Figure 15: A visualization of how frames are matched using their descriptors, borrowing ideas from locality-sensitive hashing. Each frame has its frame ID added to a hash table, with the key being its descriptor. When a frame is added to a row with previous frame IDs present, a match is identified as shown in red. This enables efficient matching without requiring a comparison against all previous frames. This example shows the case where n=1 , meaning each image has one descriptor calculated. When n>1 , each frame is added to the hash table n times—once for each descriptor calculated.
Figure 16: An example of two frames with a 10 degree angle between relative position vectors. These are taken from Scenario 3 in the infrared band. This difference in view is small enough that similar sides and components are visible on the target, but not large enough that unique parts of the target’s surface are seen in each image.
Figure 17: Precision-recall curves comparing the performance of our silhouette-based VPR method run on Scenario 3 using various parameter values (a), sensor modalities (b), and compared to baseline VPR methods (c).
VPR Method
CPU Runtime, ms
GPU Runtime, ms
Silhouette Matching ( d=16,n=16 )
23.0
N/A
Bag of Words* [ 28 ]
65.0
N/A
NetVLAD [ 4 ]
432.5
12.9
CosPlace [ 14 ]
210.6
8.3
MixVPR [ 3 ]
61.1
8.7
EigenPlaces [ 16 ]
164.5
54.0
Table 4: Runtime of VPR methods on the thermal-infrared imagery of Scenario 3. Silhouette Matching and Bag of Words are not designed for GPU, so no timing results are available for those cases.
Tracking/Mapping Success Rate
Inlier Feature Matches (Min / Avg / Max)
Scenario
VIS
TIR
Fused
VIS
TIR
Fused
S1: CloudSat 15 m
0 64.0%
100.0%
100.0%
0 0 / 0 89 / 272
15 / 0 63 / 133
0 9 / 120 / 278
S2: CloudSat 15 m, Fully Lit
100.0%
100.0%
100.0%
44 / 169 / 289
15 / 0 63 / 133
36 / 169 / 288
S3: CloudSat 15 m, Rotating
0 63.9%
100.0%
100.0%
0 0 / 0 71 / 269
13 / 0 47 / 0 88
0 9 / 0 79 / 249
S4: CloudSat 15–25 m
0 63.9%
100.0%
100.0%
0 0 / 0 65 / 285
17 / 0 61 / 169
11 / 0 77 / 278
S5: CloudSat 50 m
0 63.2%
0 99.7%
100.0%
0 0 / 0 93 / 292
43 / 101 / 271
15 / 112 / 286
Table 5: Tracking/mapping success rate and minimum, average, and maximum inlier feature matches per frame over each run. The success rate is calculated by counting the number of frames in which valid features are detected and mapped as landmarks, divided by the total number of frames.
Bag of Words LCs
Silhouette-Based LCs
Scenario
VIS
TIR
Fused
TIR
S1: CloudSat 15 m
116 / 147
23 / 40
233 / 270
1044 / 1044
S2: CloudSat 15 m, Fully Lit
400 / 465
23 / 40
401 / 472
1044 / 1044
S3: CloudSat 15 m, Rotating
00 0 / 0 13
0 3 / 15
00 1 / 00 4
00 61 / 00 61
S4: CloudSat 15–25 m
0 62 / 0 86
19 / 53
0 32 / 0 42
1073 / 1073
S5: CloudSat 50 m
0 73 / 0 97
0 1 / 0 7
187 / 225
0 971 / 1098
Table 6: Number of loop closures in each run, reported as inliers / total. Each loop closure represents a detected instance when an earlier view of the target spacecraft is revisited. Scenarios 3 and 6 only consist of 12 loop closure opportunities, each made up of many frames that match a previous frame. All other scenarios have the inspector retracing its relative trajectory, resulting in many more loop closure opportunities. These values are after geometric verification. Inlier loop closures are those found to be consistent with VIO estimates using GNC [ 66 ] .
VIO
VIO + Bag of Words LCs
VIO + TIR Silhouette-Based LCs
Scenario
VIS
TIR
Fused
VIS
TIR
Fused
VIS
TIR
Fused
S1: CloudSat 15 m
0 7.32
0 25.93
0 0.86
0 4.90
0 20.18
0.79
0 4.55
00 8.98
0.65
S2: CloudSat 15 m, Fully Lit
0 0.80
0 25.93
0 1.80
0 0.97
0 20.18
1.97
0 0.75
00 8.98
0.92
S3: CloudSat 15 m, Rotating
0 5.78
0 12.67
0 5.21
0 5.78
0 12.67
5.20
0 5.03
00 9.45
5.14
S4: CloudSat 15–25 m
0 7.55
0 24.79
0 3.05
0 6.83
0 15.63
2.68
0 6.51
00 9.56
1.97
S5: CloudSat 50 m
19.65
0 49.35
12.13
18.19
0 49.42
5.94
16.32
0 50.50
5.70
Table 7: Average position errors (meters) over each run. Errors are between Kimera’s estimate of the inspector’s relative pose and the ground truth relative pose.
VIO
VIO + Bag of Words LCs
VIO + TIR Silhouette-Based LCs
Scenario
VIS
TIR
Fused
VIS
TIR
Fused
VIS
TIR
Fused
S1: CloudSat 15 m
15.86
0 57.10
0 1.76
11.19
0 48.13
0 1.71
0 9.78
0 21.29
0 1.36
S2: CloudSat 15 m, Fully Lit
0 1.37
0 57.10
0 3.34
0 4.00
0 48.13
0 3.62
0 1.69
0 21.29
0 1.63
S3: CloudSat 15 m, Rotating
10.33
0 34.59
10.64
10.33
0 36.16
11.53
10.04
0 21.18
10.65
S4: CloudSat 15–25 m
23.05
0 53.85
0 6.56
14.01
0 37.76
0 4.71
12.71
0 20.09
0 3.46
S5: CloudSat 50 m
59.60
116.50
29.10
55.92
117.06
14.62
49.95
106.11
13.41
Table 8: Maximum position errors (meters) over each run. Errors are between Kimera’s estimate of the inspector’s relative pose and the ground truth relative pose.
VIO
VIO + Bag of Words LCs
VIO + TIR Silhouette-Based LCs
Scenario
VIS
TIR
Fused
VIS
TIR
Fused
VIS
TIR
Fused
S1: CloudSat 15 m
22.55
89.50
0 3.36
0 8.51
63.24
0 2.59
0 3.81
15.92
0 2.03
S2: CloudSat 15 m, Fully Lit
0 2.29
89.50
0 5.05
0 2.86
63.24
0 2.81
0 1.91
15.92
0 2.46
S3: CloudSat 15 m, Rotating
21.08
60.72
16.86
21.08
64.13
16.69
11.16
37.25
10.32
S4: CloudSat 15–25 m
20.89
72.75
11.00
12.55
37.61
0 4.73
0 7.62
13.79
0 3.52
S5: CloudSat 50 m
17.74
21.65
15.08
12.23
22.19
0 4.92
0 5.08
0 4.99
0 4.32
Table 9: Average rotation errors (degrees) over each run. Errors are between Kimera’s estimate of the inspector’s relative pose and the ground truth relative pose.
VIO
VIO + Bag of Words LCs
VIO + TIR Silhouette-Based LCs
Scenario
VIS
TIR
Fused
VIS
TIR
Fused
VIS
TIR
Fused
S1: CloudSat 15 m
39.39
180.00
0 7.84
19.78
131.17
0 6.01
10.23
66.82
0 3.89
S2: CloudSat 15 m, Fully Lit
0 3.50
180.00
0 9.18
15.54
131.17
0 6.55
0 4.34
66.82
0 7.04
S3: CloudSat 15 m, Rotating
45.73
129.22
30.11
45.73
147.10
49.87
31.95
80.29
27.27
S4: CloudSat 15–25 m
51.22
171.03
26.79
39.38
0 84.98
10.80
16.17
26.87
0 8.32
S5: CloudSat 50 m
48.14
0 36.59
44.65
35.34
0 45.30
12.19
10.55
18.88
0 8.09
Table 10: Maximum rotation errors (degrees) over each run. Errors are between Kimera’s estimate of the inspector’s relative pose and the ground truth relative pose.
Automatic Control Group (RAT), Paderborn University, 33098 Paderborn, Germany · Department of Electrical and Computer Engineering, Aarhus University, 8000 Aarhus C, Denmark, and also with Upteko ApS, Denmark