We present a modular, full-stack autonomy system for lunar surface navigation and mapping developed for the Lunar Autonomy Challenge. Operating in a GNSS-denied, visually challenging environment, our pipeline integrates semantic segmentation, stereo visual odometry, pose graph SLAM with loop closures, and layered planning and control. We leverage lightweight learning-based perception models for real-time segmentation and feature tracking and use a factor-graph backend to maintain globally consistent localization. High-level waypoint planning is designed to promote mapping coverage while encouraging frequent loop closures, and local motion planning uses arc sampling with geometric obstacle checks for efficient, reactive control. We evaluate our approach in the competition's high-fidelity lunar simulator, demonstrating centimeter-level localization accuracy, high-fidelity map generation, and strong repeatability across random seeds and rock distributions. Our solution achieved first place in the final competition evaluation.
Figures & tables
Figure 1: Key aspects of the Lunar Autonomy Challenge: (a) realistic image rendering, (b) realistic rover design with actuation, and (c) competition mapping objective. Images are adapted from the Lunar Autonomy Challenge simulator and documentation ( Johns Hopkins University Applied Physics Laboratory, 2025 )
Figure 2: Block diagram of our overall approach. Stereo images are processed by the front-end for semantic segmentation, rock detection, and stereo visual odometry. Detected rocks are passed to the motion planner, while tracked 3D features and odometry estimates are used in the SLAM backend for pose graph construction and loop closure. Optimized poses and semantic landmarks are projected into a global frame to generate geometric and rock maps. High-level path design generates waypoints that encourage loop closures and full map coverage, while the motion planner selects safe arcs using geometric rock avoidance.
Figure 3: Semantic segmentation example. Given input image, each pixel is labeled with a semantic class of sky, ground, rock, lander, or fiducials.
Figure 4: Extracted keypoints. Points in green have a score greater than 0.5, while points in red have score less than 0.5.
Figure 5: LightGlue feature matching between left and right stereo images.
Figure 6: LightGlue feature matching on the lander between different viewpoints.
Figure 7: Visualization of the feature matching used to perform visual odometry. Stereo matched features are extracted from left and right frames, and the matched left image features in the previous frame are matched with that of the current frame. The PnP problem of triangulated 3D points in previous frame and matched 2D points in current frame is solved to obtain relative pose of the camera from previous to current frame.
Figure 8: Overview of the SLAM frontend. Front and back stereo images and fed through feature tracking and visual odometry to produce an odometry estimate and tracked points. Semantic segmentation is used to extract semantic masks used to label tracked points and detect rocks for collision avoidance.
Figure 9: Progressive execution of the high-level path. The rover begins with a small loop and expands outward in overlapping square patterns to ensure complete coverage and frequent loop closures.
Figure 10: Example of detected rocks from stereo segmentation and depth estimates on the left, with planned arc and rocks shown in rover local frame on the right.
Preset
RMSE (m)
Geometric Map Score
Rock Map Score
Total Score
1
0.0434
269.6
153.6
823.3
2
0.0379
272.3
155.2
827.5
3
0.0605
200.8
146.2
746.9
4
0.0612
190.2
154.8
745.0
5
0.0510
224.7
150.6
775.3
Table 1: Performance across five rock distribution presets on Map 1. Our system consistently achieves strong localization and mapping scores across diverse terrain configurations.
Preset
RMSE (m)
Geometric Map Score
Rock Map Score
Total Score
1
0.0628
195.7
150.4
746.1
1
0.0472
260.2
158.1
818.3
1
0.0671
221.0
144.7
765.7
1
0.0434
269.6
153.6
823.3
1
0.0510
262.8
151.8
814.6
Table 2: Performance across different random seeds using the same rock preset. Our solution shows strong repeatability and robustness to initial rover placement.
Figure 11: Progression of a sample run visualized using Rerun. The rover’s trajectory (estimated in light blue, ground truth in black) is overlaid with the semantic point cloud. Ground points are shown in gray, rocks in red, and the lander in gold. The system maintains accurate localization and semantic consistency over the course of the mission.
Figure 12: 2D trajectories from two different runs, with ground truth trajectory shown in black and final estimated SLAM trajectory shown in orange. In the right plot, multiple backup maneuvers were triggered as the planner disengaged from local obstacles. Despite this, our SLAM maintains low localization error through the entire trajectory.
Figure 13: Final geometric and rock maps from a representative run.
Model
Params
FPS ↑
IoU [%] ↑
Ground
Rocks
Sky
Mean
U-Net
14.3M
114.4
95.5
89.2
99.8
96.1
U-Net++
16.0M
78.0
91.1
91.6
99.8
96.1
MA-Net
21.7M
71.7
70.9
90.1
99.8
91.5
Linknet
11.7M
104.5
96.4
91.1
99.8
96.7
FPN
13.0M
108.7
97.7
86.8
99.7
96.2
Table 3: Comparison of semantic segmentation performance across different models.
Figure 14: 3D trajectory plot, with ground truth trajectory shown in black and final estimated SLAM trajectory shown in orange.
Figure 15: XYZ position errors over time (meters). Errors periodically drop due to loop closures
Figure 16: Final competition leaderboard. Our solution (NAV Lab) placed first with highest geometric and rock map scores.
Future lunar missions will require autonomous rovers capable of traversing tens of kilometers across challenging terrain while maintaining accurate localization and producing globally consistent maps. However, the absence of global positioning systems, extreme illumination, and low-texture regolith make long-range navigation on the Moon particularly difficult, as visual-inertial odometry pipelines accumulate drift over extended traverses. To address this challenge, we present a stereo visual simultaneous localization and mapping (SLAM) system that integrates learned feature detection and matching with global constraints from digital elevation models (DEMs). Our front-end employs learning-based feature extraction and matching to achieve robustness to illumination extremes and repetitive terrain, while the back-end incorporates DEM-derived height and surface-normal factors into a pose graph, providing absolute surface constraints that mitigate long-term drift. We validate our approach using both simulated lunar traverse data generated in Unreal Engine and real Moon/Mars analog data collected from Mt. Etna. Results demonstrate that DEM anchoring consistently reduces absolute trajectory error compared to baseline SLAM methods, lowering drift in long-range navigation even in repetitive or visually aliased terrain.
Adam Dai, Guillem Casadesus Vila, Grace Gao
Stanford University William F. Durand Building 496 Lomita Mall Stanford, CA 94305
Navigation and mapping on the lunar surface require robust perception under challenging conditions, including poorly textured environments, high-contrast lighting, and limited computational resources. This paper presents a real-time mapping framework that integrates dense perception models with a 3D Gaussian Splatting (3DGS) representation. We first benchmark several models on synthetic datasets generated with the LuPNT simulator, selecting a stereo dense depth estimation model based on Gated Recurrent Units for its balance of speed and accuracy in depth estimation, and a convolutional neural network for its superior performance in detecting semantic segments. Using ground truth poses to decouple the local scene understanding from the global state estimation, our pipeline reconstructs a 120-meter traverse with a geometric height accuracy of approximately 3 cm, outperforming a traditional point cloud baseline without LiDAR. The resulting 3DGS map enables novel view synthesis and serves as a foundation for a full SLAM system, where its capacity for joint map and pose optimization would offer significant advantages. Our results demonstrate that combining semantic segmentation and dense depth estimation with learned map representations is an effective approach for creating detailed, large-scale maps to support future lunar surface missions.
Guillem Casadesus Vila, Adam Dai, Grace Gao
Department of Aeronautics and Astronautics, Stanford University, Stanford, CA, United States
Rovers rely on perception to maintain spatial maps that encode both objects and sensor quality (e.g., range reliability, lighting artifacts, data density), guiding data fusion, embedding updates, and navigation under partial observability. To study these coupled perception-navigation processes, we present CrossMaps, a real-time confidence-aware open-vocabulary semantic mapping pipeline that constructs language-queryable maps from RGB-D data. Building on VLMaps-style approaches, CrossMaps integrates multi-scale CLIP embeddings with confidence-aware fusion and a dual-memory architecture consisting of Short-Term Memory (STM) and Long-Term Memory (LTM). The STM aggregates noisy visual observations using geometric, semantic, and temporal confidence cues, while confident and coherent cells are promoted to the LTM as persistent semantic landmarks. Designed for deployment with a Jetson Orin-powered UGV alongside SLAM, CrossMaps runs in real time and produces semantic heatmaps that can be queried with natural language to guide rover navigation.
Jan-Niklas Klein, Sona Ghahremani, Christian Medeiros Adriano +1
Hasso Plattner Institute for Digital Engineering, Potsdam, Germany.