LLaVA-LE: Large Language-and-Vision Assistant for Lunar Exploration

Gokce Inal; Pouyan Navard; Alper Yilmaz

cs.CVMar 25, 2026

LLaVA-LE: Large Language-and-Vision Assistant for Lunar Exploration

Authors: Gokce Inal, Pouyan Navard, Alper Yilmaz

Organizations: The Ohio State University Columbus, OH, USA

Abstract

Recent advances in multimodal vision-language models (VLMs) have enabled joint reasoning over visual and textual information, yet their application to planetary science remains largely unexplored. A key hindrance is the absence of large-scale datasets that pair real planetary imagery with detailed scientific descriptions. In this work, we introduce LLaVA-LE (Large Language-and-Vision Assistant for Lunar Exploration), a vision-language model specialized for lunar surface and subsurface characterization. To enable this capability, we curate a new large-scale multimodal lunar dataset, LUCID (LUnar Caption Image Dataset) consisting of 96k high-resolution panchromatic images paired with detailed captions describing lunar terrain characteristics, and 81k question-answer (QA) pairs derived from approximately 20k images in the LUCID dataset. Leveraging this dataset, we fine-tune LLaVA using a two-stage training curriculum: (1) concept alignment for domain-specific terrain description, and (2) instruction-tuned visual question answering. We further design evaluation benchmarks spanning multiple levels of reasoning complexity relevant to lunar terrain analysis. Evaluated against GPT and Gemini judges, LLaVA-LE achieves a 3.3x overall performance gain over Base LLaVA and 2.1x over our Stage 1 model, with a reasoning score of 1.070, exceeding the judge's own reference score, highlighting the effectiveness of domain-specific multimodal data and instruction tuning to advance VLMs in planetary exploration. Code is available at https://github.com/OSUPCVLab/LLaVA-LE.

Figures & tables

Panel of Figure 1 — Figure 1: Category-wise Visual Question Answering Performance Relative to Judge Scores . We evaluate Base LLaVA, LLaVA-LE Stage 1, and LLaVA-LE Stage 2 on a held-out evaluation set across three question categories: Detailed, Conversation, and Reasoning. Each axis shows the model’s score normalized by the judge’s score (Score / Judge Score), averaged across GPT-4 and Gemini judges. The Overall axis represents the mean across all three categories. Values greater than 1.0 indicate the model outperforms the judge’s reference score.

Figure 2: An Example of LUCID dataset : The top shows the the panchromatic image with the associated caption generated using GPT, while the below shows question-answer pairs generated from that caption.

Figure 3: Distribution of LUCID VQA dataset across question-answer pairs categories. This heatmap matrix shows how much each answer is contributing, in terms of percentage, to each one of the question categories across the entire LUCID visual question answering dataset. The heatmap is normalized by the row dimension and shows the reasoning capability captured in our dataset to answer each question.

Figure 4: Overview of LLaVA-LE Training and Inference. The right panel illustrates the two-stage training pipeline. Visual inputs $X_{\texttt{v}}$ are encoded by a frozen CLIP vision encoder $g$ to produce visual features $Z_{v}=g(X_{\texttt{v}})$ , which are then projected into the language embedding space via a trainable projection layer, yielding $H_{\texttt{v}}$ to be concatenated with the language token embeddings $H_{\texttt{q}}$ . The language model backbone $f_{\phi}$ autoregressively generates a response $X_{a}$ conditioned on the joint visual and language context. In Stage 1 (Concept Alignment) , caption supervision aligns lunar visual features with domain-specific geological language. In Stage 2 (Instruction Tuning) , the aligned model is further fine-tuned on multi-turn question-answer pairs to enable reasoning and conversational interaction. Throughout both stages, the CLIP vision encoder and the pretrained language model backbone $f_{\phi}$ remain frozen; only the projection layer and lightweight LoRA adaptation modules are updated. The left panel illustrates a representative inference example after Stage 2, where a user query about lunar surface observations is processed and the model generates a domain-grounded geological interpretation $X_{a}$ .

Figure 5: Qualitative comparison of LLaVA-LE visual chat and reasoning capabilities in instruction-tuning tasks. Comparison between the response of LLaVA-13B model and our LlaVA-LE model in during instruction-tuning stage evaluation. The base LLaVA model often produces fluent but generic answers, occasionally lacking domain-specific terminology or geological interpretation. In some cases, the model hallucinates features that are not clearly implied by the scene or shifts toward broad discussions of lunar history rather than addressing the detail in the question. The Gemini and GPT are considered as the performance upper bound because the ground truth caption is fed into both of these models as context to generate the ground truth response.

	GPT Judge				Gemini Judge
Model	Detailed	Conversation	Reasoning	Overall	Detailed	Conversation	Reasoning	Overall
LLaVA-LE Stage2	0.952	0.767	1.006	0.925	0.892	0.628	1.134	0.917
LLaVA-LE Stage1	0.433	0.419	0.480	0.450	0.367	0.362	0.536	0.436
Base LLaVA	0.319	0.314	0.344	0.329	0.220	0.205	0.246	0.227

Table 1: Performance Ratios Relative to GPT and Gemini Judges (Score / Judge Score)

Panel of Figure 6 — Figure 6: Category distribution of the evaluation set : Reasoning category is intentionally emphasized in the evaluation set because they pose a harder challenge beyond general visual description.

Explore similar work

Multimodal-Multiresolution Foundation Model for Lunar Remote Sensing
Sep 8, 2026Paolo Fraccaro, Gabby Nyirjesy, Daniela Szwarcman +19Multimodal Foundation Models Remote Sensing
IBM Research, Yorktown Heights, NY, USA · The University of Alabama in Huntsville, Huntsville, AL, USA · University Space Research Association (USRA), Science and Technology Institute, Huntsville, AL, USA +7
MoonAnything: A Vision Benchmark with Large-Scale Lunar Supervised Data
Apr 1, 2026Clémentine Grethen, Yuang Shi, Simone Gasparini +1Remote Sensing Benchmarks 3D Reconstruction
IRIT - Université de Toulouse, France · National University of Singapore, Singapore · IPAL, IRL2955, Singapore
LunarFM: A Shared Multimodal Representation of the Moon's Surface
Jul 24, 2026Marc Girona-Mata, Jakob Gawlikowski, Sumit Goski +6Geospatial Representation Learning Multimodal Pretraining
German Aerospace Center (DLR) · European Space Resources Innovation Center (ESRIC) · Universidad de Antioquia

Sep 8, 2026J/K move · Enter open · S save

Multimodal-Multiresolution Foundation Model for Lunar Remote Sensing

Paolo Fraccaro, Gabby Nyirjesy, Daniela Szwarcman, Himanshu Patil+18

IBM Research, Yorktown Heights, NY, USA · The University of Alabama in Huntsville, Huntsville, AL, USA · University Space Research Association (USRA), Science and Technology Institute, Huntsville, AL, USA+7

Multimodal Foundation Models Remote Sensing Multimodal Pretraining

Open paper AskPDF arXiv

We present a multimodal foundation model for lunar remote sensing, pretrained from scratch on SomBench, a geographically partitioned corpus of nearly two million co-registered tile bundles spanning 11 modalities at two spatial scales (1 m/pixel and 100 m/pixel). The model adapts the TerraMind masked-token architecture with two lunar-specific extensions: acquisition geometry is provided as explicit context, and meter- and hundred-meter-scale tiles are trained jointly so that a single set of weights covers both resolutions. FlexiViT patch embeddings allow adaptation to different patch sizes without retraining, while modality-wise inputs enable flexible multimodal fine-tuning. Qualitative generation experiments suggest the model learns meaningful cross-modal correspondences, including terrain derivatives from elevation and illumination-consistent reflectance from geometry. We evaluate on four benchmarks: crater detection at WAC and NAC scales, irregular mare patch (IMP) segmentation, and polar ice prospectivity regression. Across tasks, the pretrained model matches or outperforms ImageNet-pretrained baselines and an architecturally identical random-init control. On multimodal ice prospectivity regression, pretrained variants achieve the best results, while the random-init model outperforms most baselines, suggesting gains arise from both the architecture and pretraining. Label efficiency is notable for WAC crater detection, where the pretrained model trained on 50% of the data exceeds the strongest ImageNet baseline trained on the full dataset. Among adaptation strategies, LoRA matches or surpasses full fine-tuning on crater detection and IMP segmentation while using far fewer trainable parameters, whereas full fine-tuning performs best for ice prospectivity regression. We release the pretrained checkpoint, benchmark datasets, and fine-tuning code to support reproducible lunar AI research.

Abstract

Explore similar work

Multimodal-Multiresolution Foundation Model for Lunar Remote Sensing

MoonAnything: A Vision Benchmark with Large-Scale Lunar Supervised Data

LunarFM: A Shared Multimodal Representation of the Moon's Surface