End-to-End Vision-Language-Action Policies for Autonomous Lunar Rover Docking

Sitan Huang; Grace Gao

cs.ROOct 13, 2026

End-to-End Vision-Language-Action Policies for Autonomous Lunar Rover Docking

Authors: Sitan Huang, Grace Gao

Organizations: Stanford University, William F. Durand Building, 496 Lomita Mall, Stanford, CA 94305

Abstract

Surface navigation on the Moon is subject to extreme illumination contrast and deployment-time perceptual disturbances. While modular navigation pipelines have long supported planetary rover navigation, robustness is compromised by measurement errors that propagate through hand-engineered components. End-to-end learned policies are an alternative approach to navigation, which maps observations directly to control actions without intermediate components. Vision-language-action models, in particular, have driven significant breakthroughs in robotics and autonomous vehicle research. By leveraging internet-scale vision-language pretraining, these models can be exploited for their commonsense reasoning and visual understanding to provide robust generalization with limited task-specific data. However, their applicability and performance have yet to be characterized for lunar surface operations. In this study, we design a vision-language-action policy for autonomous lunar rover docking and characterize its performance relative to a modular navigation pipeline. We develop a scalable training framework that generates safety-aware expert demonstrations in a high-fidelity lunar simulation environment. In closed-loop evaluations, our policy performs competitively relative to the modular baseline under perceptual disturbances unseen during training. To address the tradeoff between compute consumption and closed-loop safety, the policy adapts its execution horizon by determining how far the rover may safely travel in open loop. Adaptive-horizon execution reduces inference frequency while limiting hazard-contact rates relative to fixed horizons in the evaluated scenarios.

Figures & tables

Panel of Figure 1 — Fig. 1 : Overview of the docking task. The ego rover must enter the final-approach corridor (blue strip) then align with the landing module’s docking port (red panel). Terrain, sun elevation and hazard landscape are randomized per scenario. Docking is successful when the rover makes contact within 350 s and with $<15^{\circ}$ bearing error relative to the docking-port axis.

Fig. 2 : Representative cost map for expert-trajectory generation. Voronoi safety field (blue) and shadow costs (orange) are visualized, with isocontours overlaid and darker regions denoting higher combined cost $\rho(x,y)$ . The expert planner generates a trajectory (solid red line) to a waypoint labeled “Goal” at the entrance to the final-approach corridor. From the “Goal” waypoint, the rover tracks the docking-port axis (dashed red line) to complete docking.

Fig. 3 : Overview of the VLA policy and training. a) Prior to policy training, an RVQ-VAE is trained on expert trajectories from the policy’s training dataset, which then assigns each trajectory in the same dataset a primary and secondary code pair. b) The policy is trained to predict these primary and secondary codes from camera and BEV-map images. The quantized latent vector corresponding to predicted codes is decoded back to an RVQ-discretized trajectory, whose heading angles are refined by continuous offsets. An execution-horizon head predicts a categorical distribution over execution horizons for each curvature bound. At deployment, the maximum curvature demanded for closed-loop trajectory tracking selects the corresponding execution horizon.

Fig. 4 : Conditions evaluated under sensor disturbance.

Fig. 5 : Docking success rates under nominal and sensor disturbance conditions. The VLA policy demonstrates reduced sensitivity to sensor disturbances relative to the modular baseline.

Fig. 7 : Emergent closed-loop behaviors not present in the expert demonstrations.

Fig. 6 : Camera observations and derived perception estimates for the Nominal (left) and Low optical-effect-strength (right) conditions. Degradations in stereo depth estimates (bottom) and semantic segmentation (middle) are observed.

Fig. 8 : Docking-success and hazard-contact rates over mean inference calls per successful scenario. Squares denote fixed-horizon conditions (8, 6, 4, 3, and 2 m, from left to right); crosses denote the adaptive-horizon condition (mean distance traveled per inference call: 4.34 m). The adaptive-horizon condition achieves a similar docking-success rate (90%) and hazard-contact rate (10% versus 9%) compared with the fixed 2-m horizon while requiring 51% fewer inference calls per successful scenario.

Execution horizon
	Without BEV	With BEV
Bin accuracy (%) by curvature ( $\mathrm{m}^{-1}$ ) $\uparrow$
0.125	83.39	84.92
0.250	79.46	82.24
0.375	75.15	80.02
0.500	73.84	80.61

TABLE I : Execution-horizon and trajectory prediction metrics with and without BEV maps on open-loop evaluations (

\mathbf{N=3203}

). BEV maps reduce hazard contact rates and waypoint heading MAE, and increase execution-horizon prediction accuracy, with larger improvements at higher curvatures.

		Trajectory ensembling
Condition	Frequent inference	Within inference	Across inferences	Docking bearing error ( ∘ ) $\downarrow$
Baseline				$7.24\pm 6.04$
A	X			$5.98\pm 4.39$
B	X	X		$6.67\pm 4.16$
C	X		X	$5.83\pm 3.92$
GTE	X	X	X	$\mathbf{4.73\pm 3.27^{*}}$

TABLE II : Docking bearing errors (mean

\pm

SD;

N=50

per condition) across ablated components of GTE. GTE significantly improves docking accuracy relative to Baseline. *

p=0.015

	Rate (%)
Video-history length (frames)	Docking success	Hazard contact	Mean docking completion time (s)
5	87	11	114.7 $\pm$ 62.9
10	84	14	93.5 $\pm$ 40.7
20	90	10	81.4 $\pm$ 25.3

TABLE III : Closed-loop docking performance metrics evaluated in

N=100

scenarios per condition. Successful docking completion time is reported in mean

\pm

SD.

Component	Median	P95
Input preprocessing	41	56
Qwen3.5-4B backbone	183	189
Trajectory + execution-horizon heads	2	2
Total	225	247

TABLE IV : Policy inference latency (ms) data. P95 denotes the 95th percentile.

Space Robotics

Space robotics develops robots and autonomy for exploring, assembling, and maintaining spacecraft and planetary environments.

See all 71 papers

LLaVA-LE: Large Language-and-Vision Assistant for Lunar Exploration
Mar 25, 2026Gokce Inal, Pouyan Navard, Alper YilmazVision-Language Models Large Vision-Language Models
The Ohio State University Columbus, OH, USA
"Dear LLaVA, Please Drive": A Depth-Aware Vision-Language Agent for Closed-Loop Robotic Control
Sep 19, 2026Sebastian Berger, Katharina Winter, Fabian B. FlohrVision-Language Navigation Vision-Based Robot Control
Munich University of Applied Sciences, Intelligent Vehicles Lab (IVL), 80335 Munich, Germany
WayFinder: Hierarchical Visual-Language-Action for Zero-Shot Waypoint Generation and Low-Level Kinematic Control
Sep 29, 2026Timothy K Johnsen, Marco LevoratoRobotic Control Large Language Model-Based Robot Planning
San Diego State University and The University of California, Irvine in CA, USA · University of California, Irvine in CA, USA

Mar 25, 2026J/K move · Enter open · S save

LLaVA-LE: Large Language-and-Vision Assistant for Lunar Exploration

Gokce Inal, Pouyan Navard, Alper Yilmaz

The Ohio State University Columbus, OH, USA

Vision-Language Models Large Vision-Language Models Visual Question Answering Image Captioning

Open paper AskPDF arXiv

Recent advances in multimodal vision-language models (VLMs) have enabled joint reasoning over visual and textual information, yet their application to planetary science remains largely unexplored. A key hindrance is the absence of large-scale datasets that pair real planetary imagery with detailed scientific descriptions. In this work, we introduce LLaVA-LE (Large Language-and-Vision Assistant for Lunar Exploration), a vision-language model specialized for lunar surface and subsurface characterization. To enable this capability, we curate a new large-scale multimodal lunar dataset, LUCID (LUnar Caption Image Dataset) consisting of 96k high-resolution panchromatic images paired with detailed captions describing lunar terrain characteristics, and 81k question-answer (QA) pairs derived from approximately 20k images in the LUCID dataset. Leveraging this dataset, we fine-tune LLaVA using a two-stage training curriculum: (1) concept alignment for domain-specific terrain description, and (2) instruction-tuned visual question answering. We further design evaluation benchmarks spanning multiple levels of reasoning complexity relevant to lunar terrain analysis. Evaluated against GPT and Gemini judges, LLaVA-LE achieves a 3.3x overall performance gain over Base LLaVA and 2.1x over our Stage 1 model, with a reasoning score of 1.070, exceeding the judge's own reference score, highlighting the effectiveness of domain-specific multimodal data and instruction tuning to advance VLMs in planetary exploration. Code is available at https://github.com/OSUPCVLab/LLaVA-LE.

Abstract

Space Robotics

Explore similar work

LLaVA-LE: Large Language-and-Vision Assistant for Lunar Exploration

"Dear LLaVA, Please Drive": A Depth-Aware Vision-Language Agent for Closed-Loop Robotic Control

WayFinder: Hierarchical Visual-Language-Action for Zero-Shot Waypoint Generation and Low-Level Kinematic Control