cs.ROMar 18, 2026
SaveFull Stack Navigation, Mapping, and Planning for the Lunar Autonomy Challenge
Organizations: Stanford University
Abstract
We present a modular, full-stack autonomy system for lunar surface navigation and mapping developed for the Lunar Autonomy Challenge. Operating in a GNSS-denied, visually challenging environment, our pipeline integrates semantic segmentation, stereo visual odometry, pose graph SLAM with loop closures, and layered planning and control. We leverage lightweight learning-based perception models for real-time segmentation and feature tracking and use a factor-graph backend to maintain globally consistent localization. High-level waypoint planning is designed to promote mapping coverage while encouraging frequent loop closures, and local motion planning uses arc sampling with geometric obstacle checks for efficient, reactive control. We evaluate our approach in the competition's high-fidelity lunar simulator, demonstrating centimeter-level localization accuracy, high-fidelity map generation, and strong repeatability across random seeds and rock distributions. Our solution achieved first place in the final competition evaluation.
Figures & tables
Figure 1: Key aspects of the Lunar Autonomy Challenge: (a) realistic image rendering, (b) realistic rover design with actuation, and (c) competition mapping objective. Images are adapted from the Lunar Autonomy Challenge simulator and documentation ( Johns Hopkins University Applied Physics Laboratory, 2025 )
Figure 2: Block diagram of our overall approach. Stereo images are processed by the front-end for semantic segmentation, rock detection, and stereo visual odometry. Detected rocks are passed to the motion planner, while tracked 3D features and odometry estimates are used in the SLAM backend for pose graph construction and loop closure. Optimized poses and semantic landmarks are projected into a global frame to generate geometric and rock maps. High-level path design generates waypoints that encourage loop closures and full map coverage, while the motion planner selects safe arcs using geometric rock avoidance.
Figure 3: Semantic segmentation example. Given input image, each pixel is labeled with a semantic class of sky, ground, rock, lander, or fiducials.
Figure 4: Extracted keypoints. Points in green have a score greater than 0.5, while points in red have score less than 0.5.
Figure 5: LightGlue feature matching between left and right stereo images.
Figure 6: LightGlue feature matching on the lander between different viewpoints.
Figure 7: Visualization of the feature matching used to perform visual odometry. Stereo matched features are extracted from left and right frames, and the matched left image features in the previous frame are matched with that of the current frame. The PnP problem of triangulated 3D points in previous frame and matched 2D points in current frame is solved to obtain relative pose of the camera from previous to current frame.
Figure 8: Overview of the SLAM frontend. Front and back stereo images and fed through feature tracking and visual odometry to produce an odometry estimate and tracked points. Semantic segmentation is used to extract semantic masks used to label tracked points and detect rocks for collision avoidance.
Figure 9: Progressive execution of the high-level path. The rover begins with a small loop and expands outward in overlapping square patterns to ensure complete coverage and frequent loop closures.
Figure 10: Example of detected rocks from stereo segmentation and depth estimates on the left, with planned arc and rocks shown in rover local frame on the right.
| Preset | RMSE (m) | Geometric Map Score | Rock Map Score | Total Score |
|---|---|---|---|---|
| 1 | 0.0434 | 269.6 | 153.6 | 823.3 |
| 2 | 0.0379 | 272.3 | 155.2 | 827.5 |
| 3 | 0.0605 | 200.8 | 146.2 | 746.9 |
| 4 | 0.0612 | 190.2 | 154.8 | 745.0 |
| 5 | 0.0510 | 224.7 | 150.6 | 775.3 |
Table 1: Performance across five rock distribution presets on Map 1. Our system consistently achieves strong localization and mapping scores across diverse terrain configurations.
| Preset | RMSE (m) | Geometric Map Score | Rock Map Score | Total Score |
|---|---|---|---|---|
| 1 | 0.0628 | 195.7 | 150.4 | 746.1 |
| 1 | 0.0472 | 260.2 | 158.1 | 818.3 |
| 1 | 0.0671 | 221.0 | 144.7 | 765.7 |
| 1 | 0.0434 | 269.6 | 153.6 | 823.3 |
| 1 | 0.0510 | 262.8 | 151.8 | 814.6 |
Table 2: Performance across different random seeds using the same rock preset. Our solution shows strong repeatability and robustness to initial rover placement.
Figure 11: Progression of a sample run visualized using Rerun. The rover’s trajectory (estimated in light blue, ground truth in black) is overlaid with the semantic point cloud. Ground points are shown in gray, rocks in red, and the lander in gold. The system maintains accurate localization and semantic consistency over the course of the mission.
Figure 12: 2D trajectories from two different runs, with ground truth trajectory shown in black and final estimated SLAM trajectory shown in orange. In the right plot, multiple backup maneuvers were triggered as the planner disengaged from local obstacles. Despite this, our SLAM maintains low localization error through the entire trajectory.
Figure 13: Final geometric and rock maps from a representative run.
| Model | Params | FPS | IoU [%] | |||
|---|---|---|---|---|---|---|
| Ground | Rocks | Sky | Mean | |||
| U-Net | 14.3M | 114.4 | 95.5 | 89.2 | 99.8 | 96.1 |
| U-Net++ | 16.0M | 78.0 | 91.1 | 91.6 | 99.8 | 96.1 |
| MA-Net | 21.7M | 71.7 | 70.9 | 90.1 | 99.8 | 91.5 |
| Linknet | 11.7M | 104.5 | 96.4 | 91.1 | 99.8 | 96.7 |
| FPN | 13.0M | 108.7 | 97.7 | 86.8 | 99.7 | 96.2 |
Table 3: Comparison of semantic segmentation performance across different models.
Figure 14: 3D trajectory plot, with ground truth trajectory shown in black and final estimated SLAM trajectory shown in orange.
Figure 15: XYZ position errors over time (meters). Errors periodically drop due to loop closures
Figure 16: Final competition leaderboard. Our solution (NAV Lab) placed first with highest geometric and rock map scores.