Surface navigation on the Moon is subject to extreme illumination contrast and deployment-time perceptual disturbances. While modular navigation pipelines have long supported planetary rover navigation, robustness is compromised by measurement errors that propagate through hand-engineered components. End-to-end learned policies are an alternative approach to navigation, which maps observations directly to control actions without intermediate components. Vision-language-action models, in particular, have driven significant breakthroughs in robotics and autonomous vehicle research. By leveraging internet-scale vision-language pretraining, these models can be exploited for their commonsense reasoning and visual understanding to provide robust generalization with limited task-specific data. However, their applicability and performance have yet to be characterized for lunar surface operations. In this study, we design a vision-language-action policy for autonomous lunar rover docking and characterize its performance relative to a modular navigation pipeline. We develop a scalable training framework that generates safety-aware expert demonstrations in a high-fidelity lunar simulation environment. In closed-loop evaluations, our policy performs competitively relative to the modular baseline under perceptual disturbances unseen during training. To address the tradeoff between compute consumption and closed-loop safety, the policy adapts its execution horizon by determining how far the rover may safely travel in open loop. Adaptive-horizon execution reduces inference frequency while limiting hazard-contact rates relative to fixed horizons in the evaluated scenarios.
Figures & tables
Fig. 1 : Overview of the docking task. The ego rover must enter the final-approach corridor (blue strip) then align with the landing module’s docking port (red panel). Terrain, sun elevation and hazard landscape are randomized per scenario. Docking is successful when the rover makes contact within 350 s and with <15∘ bearing error relative to the docking-port axis.
Fig. 2 : Representative cost map for expert-trajectory generation. Voronoi safety field (blue) and shadow costs (orange) are visualized, with isocontours overlaid and darker regions denoting higher combined cost ρ(x,y) . The expert planner generates a trajectory (solid red line) to a waypoint labeled “Goal” at the entrance to the final-approach corridor. From the “Goal” waypoint, the rover tracks the docking-port axis (dashed red line) to complete docking.
Fig. 3 : Overview of the VLA policy and training. a) Prior to policy training, an RVQ-VAE is trained on expert trajectories from the policy’s training dataset, which then assigns each trajectory in the same dataset a primary and secondary code pair. b) The policy is trained to predict these primary and secondary codes from camera and BEV-map images. The quantized latent vector corresponding to predicted codes is decoded back to an RVQ-discretized trajectory, whose heading angles are refined by continuous offsets. An execution-horizon head predicts a categorical distribution over execution horizons for each curvature bound. At deployment, the maximum curvature demanded for closed-loop trajectory tracking selects the corresponding execution horizon.
Fig. 4 : Conditions evaluated under sensor disturbance.
Fig. 5 : Docking success rates under nominal and sensor disturbance conditions. The VLA policy demonstrates reduced sensitivity to sensor disturbances relative to the modular baseline.
Fig. 7 : Emergent closed-loop behaviors not present in the expert demonstrations.
Fig. 6 : Camera observations and derived perception estimates for the Nominal (left) and Low optical-effect-strength (right) conditions. Degradations in stereo depth estimates (bottom) and semantic segmentation (middle) are observed.
Fig. 8 : Docking-success and hazard-contact rates over mean inference calls per successful scenario. Squares denote fixed-horizon conditions (8, 6, 4, 3, and 2 m, from left to right); crosses denote the adaptive-horizon condition (mean distance traveled per inference call: 4.34 m). The adaptive-horizon condition achieves a similar docking-success rate (90%) and hazard-contact rate (10% versus 9%) compared with the fixed 2-m horizon while requiring 51% fewer inference calls per successful scenario.
Without BEV
With BEV
Execution horizon
Bin accuracy (%) by curvature ( m−1 ) ↑
0.125
83.39
84.92
0.250
79.46
82.24
0.375
75.15
80.02
0.500
73.84
80.61
TABLE I : Execution-horizon and trajectory prediction metrics with and without BEV maps on open-loop evaluations ( N=3203 ). BEV maps reduce hazard contact rates and waypoint heading MAE, and increase execution-horizon prediction accuracy, with larger improvements at higher curvatures.
Trajectory ensembling
Condition
Frequent inference
Within inference
Across inferences
Docking bearing error ( ∘ ) ↓
Baseline
7.24±6.04
A
X
5.98±4.39
B
X
X
6.67±4.16
C
X
X
5.83±3.92
GTE
X
X
X
4.73±3.27∗
TABLE II : Docking bearing errors (mean ± SD; N=50 per condition) across ablated components of GTE. GTE significantly improves docking accuracy relative to Baseline. * p=0.015
Rate (%)
Video-history length (frames)
Docking success
Hazard contact
Mean docking completion time (s)
5
87
11
114.7 ± 62.9
10
84
14
93.5 ± 40.7
20
90
10
81.4 ± 25.3
TABLE III : Closed-loop docking performance metrics evaluated in N=100 scenarios per condition. Successful docking completion time is reported in mean ± SD.
Component
Median
P95
Input preprocessing
41
56
Qwen3.5-4B backbone
183
189
Trajectory + execution-horizon heads
2
2
Total
225
247
TABLE IV : Policy inference latency (ms) data. P95 denotes the 95th percentile.
Recent advances in multimodal vision-language models (VLMs) have enabled joint reasoning over visual and textual information, yet their application to planetary science remains largely unexplored. A key hindrance is the absence of large-scale datasets that pair real planetary imagery with detailed scientific descriptions. In this work, we introduce LLaVA-LE (Large Language-and-Vision Assistant for Lunar Exploration), a vision-language model specialized for lunar surface and subsurface characterization. To enable this capability, we curate a new large-scale multimodal lunar dataset, LUCID (LUnar Caption Image Dataset) consisting of 96k high-resolution panchromatic images paired with detailed captions describing lunar terrain characteristics, and 81k question-answer (QA) pairs derived from approximately 20k images in the LUCID dataset. Leveraging this dataset, we fine-tune LLaVA using a two-stage training curriculum: (1) concept alignment for domain-specific terrain description, and (2) instruction-tuned visual question answering. We further design evaluation benchmarks spanning multiple levels of reasoning complexity relevant to lunar terrain analysis. Evaluated against GPT and Gemini judges, LLaVA-LE achieves a 3.3x overall performance gain over Base LLaVA and 2.1x over our Stage 1 model, with a reasoning score of 1.070, exceeding the judge's own reference score, highlighting the effectiveness of domain-specific multimodal data and instruction tuning to advance VLMs in planetary exploration. Code is available at https://github.com/OSUPCVLab/LLaVA-LE.
Vision-language models (VLMs) provide a compelling foundation for reasoning-driven mobile navigation, offering rich contextual understanding and strong generalization from large-scale pretraining. Most existing navigation frameworks rely on imitation learning and therefore require substantial labeled trajectory data, limiting their scalability and robustness. In this work, we propose a parameter-efficient approach to fine-tune a pretrained VLM for autonomous navigation using an Imperative Learning paradigm. By optimizing against differentiable geometric cost fields rather than labeled trajectories, our model learns to generate collision-free paths exclusively from stereoscopic depth observations. We introduce a unified end-to-end navigation pipeline for natural-language-driven robotic control. This system leverages a shared VLM backbone with task-specific Low-Rank Adaptation (LoRA) modules, effectively bridging the gap from semantic target selection to low-level trajectory planning. Our approach achieves competitive Success weighted by Path Length (SPL) in unseen environments while updating less than 1% of the model's total parameters. Qualitative real-world experiments validate sim-to-real generalization and stable path planning without fine-tuning on real-world data. These results highlight a practical approach for deploying VLM-based agents on mobile robots, enabling high-level semantic navigation without the prohibitive requirement for large-scale, labeled trajectory data.
Sebastian Berger, Katharina Winter, Fabian B. Flohr
Munich University of Applied Sciences, Intelligent Vehicles Lab (IVL), 80335 Munich, Germany
Visual Language Action (VLA) models offer unprecedented generalization for autonomous robots; however, their real-world deployment is frequently bottlenecked by unreliable execution and the prohibitive computational cost of fine-tuning for specific robot embodiments and tasks. To bridge this gap, we propose WayFinder, an end-to-end, closed-loop hierarchical VLA framework that circumvents the need for fine-tuning by decoupling high-level task reasoning from low-level kinematic control. WayFinder utilizes a zero-shot, offboard Multimodal Large Language Model (MLLM) policy to process linguistic context and state maps for strategic waypoint generation. Asynchronously, a lightweight, onboard policy executes real-time kinematic control at high frequency based on continuous sensor feedback. We evaluate WayFinder in Microsoft AirSim, testing on four environments of varying complexity and three MLLM scales to balance prediction efficacy with computational efficiency. Our results demonstrate that WayFinder achieves superior navigation reliability compared to baseline low-level policies. By querying the high-level MLLM only during navigation failures, WayFinder eliminates the need for fine-tuning, minimizes expensive inferences, and significantly increases navigation success rates by up to 27.45%.
Timothy K Johnsen, Marco Levorato
San Diego State University and The University of California, Irvine in CA, USA · University of California, Irvine in CA, USA