EarthJev: Calibrated, Non-Generative Answers to Natural-Language Questions about Sentinel-2 Land Cover

Domenico Benfenati; Antonio Maria Rinaldi; Cristian Tommasino

cs.CVNov 1, 2026

EarthJev: Calibrated, Non-Generative Answers to Natural-Language Questions about Sentinel-2 Land Cover

Authors: Domenico Benfenati, Antonio Maria Rinaldi, Cristian Tommasino

Organizations: Department of Electrical Engineering and Information Technology (DIETI), University of Naples Federico II, Via Claudio, 21, Naples, 80125, Italy

Abstract

Vision-language models for remote sensing answer questions about satellite images by generating text: every answer costs a decoding loop, has to be parsed back to an option, and comes without a usable probability. We present EarthJev, a 0.8B-parameter model that answers typed questions about multispectral Sentinel-2 patches without generating any token. All 12 bands enter the vision encoder of Qwen3.5-0.8B through a widened input layer initialised from the RGB kernels, and, following the open OneJev recipe, the answer is read from the logits of the option letters at a single position and trained with cross-entropy plus Brier score. Under a pre-specified analysis plan, we compare it with the same backbone fine-tuned to generate the answer and with three multimodal language models fine-tuned with LoRA, all trained on the same 70,890 questions from 5% of BigEarthNet v2.0 and timed in one job on the same GPU; VQA models that classify over a fixed answer vocabulary are reported as a reference for the cost of answering only the questions seen in training. EarthJev answers a question in 19.9 ms, 4.3 to 8.7 times faster than the multimodal language models and 6.8 times faster than its own backbone used generatively; when all the questions about an image are asked together, it answers them in 42.8 ms, 3.7 to 10.1 times faster than the batched generative models. On 36,028 test questions it reaches 86.0% accuracy (three seeds), 0.6 and 1.2 points below the two 2B multimodal models and 3.5 points below the remote-sensing EarthDial-4B, with answers that are valid by construction and calibrated probabilities (ECE 0.006). Replacing the nine non-RGB bands with their mean costs 4.5 points, so the model uses the multispectral input, and in an ablation training the whole vision encoder raises accuracy to 88.4% at the same cost. Code and model: https://github.com/CosmoIknosLab/EarthJev

Figures & tables

Ours
		CORINE test (36,028 q.)		RSVQAxBEN	Latency per question
Model	Params	Acc. (%)	ECE	Acc. (%)	p50 (ms)	Slower
EarthJev (ours)	0.8B	86.0 ±0.1	0.006	81.0 ±0.6	19.9	–
Same backbone and input, generative fine-tuning
Qwen3.5-0.8B, generative	0.8B	86.8 ±0.2	0.005	80.5 ±0.6	134.7	6.8 $\times$
Multimodal LLMs, LoRA fine-tuning

Table 1: Speed against quality. CORINE: seed mean (standard deviation as subscript) of the accuracy on the 36,028 test questions, and mean of the per-seed ECE (15 bins, before temperature scaling). Latency per question at batch 1 and how many times slower than EarthJev each model is, from Table 2 . Fixed-vocabulary models answer only the questions and options seen in training.

			One question per request			All questions of an image
Model	Params	Input	p50 (ms)	p95 (ms)	Slower	Image (ms)	Per q. (ms)	Slower	Mem. (GB)
Ours
EarthJev (ours)	0.8B	12 bands	19.9	20.1	–	42.8	7.9	–	3.5
Same backbone and input, generative fine-tuning
Qwen3.5-0.8B, generative	0.8B	12 bands	134.7	144.4	6.8 $\times$	191.9	36.9	4.5 $\times$	2.2
Multimodal LLMs, LoRA fine-tuning

Table 2: Inference cost of every model, measured in one job on one H100 PCIe GPU (bfloat16; 100 test images after 10 warm-up images; each timer runs from the decoded image in memory to the answer, preprocessing and tokenisation included). Slower : p50 latency of the model divided by that of EarthJev in the same scenario (below 1: faster than EarthJev). Image : p50 latency of a request with all the questions of an image (6.1 on average); Per q. : mean latency per question in that scenario. In the second scenario EarthJev encodes the image once and answers all its questions as branches of a shared prefix; the generative models answer them in one batched generation and the fixed-vocabulary models in one batched forward pass.

Ours
			CORINE test (36,028 questions)							RSVQA-	BigEarthNet.txt
Model	Params	Input	All	Dom.	Pres.	Share	Loc.	Comp.	ECE	xBEN	Binary	MC
EarthJev (ours)	0.8B	12 bands	86.0 ±0.1	83.6	93.2	82.9	66.1	87.9	0.006	81.0	59.6	40.1
Same backbone and input, generative fine-tuning
Qwen3.5-0.8B, generative	0.8B	12 bands	86.8 ±0.2	85.0	93.6	83.7	66.6	88.5	0.005	80.5	59.3	17.1
Multimodal LLMs, LoRA fine-tuning

Table 3: All models trained on the same 70,890 CORINE questions (three seeds). CORINE: accuracy (%) overall (seed mean, standard deviation as subscript) and per question type, seed-averaged; ECE: mean of the per-seed ECEs, 15 bins, before temperature scaling. RSVQAxBEN: 29,501 presence questions. BigEarthNet.txt: 6,927 binary and 5,550 four-option questions; the fixed-vocabulary models cannot answer them (their answer vocabulary does not cover them). Transfer tests are never used for training.

	CORINE		RSVQAxBEN		BigEarthNet.txt
Variant	Acc.	$\Delta$ [95% CI]	Acc.	$\Delta$ [95% CI]	Acc.	$\Delta$ [95% CI]
EarthJev (seed 0)	85.9	–	80.3	–	50.8	–
whole vision encoder trained	88.4	+2.5 [+2.2, +2.8]	83.0	+2.7 [+2.2, +3.1]	51.1	+0.3 [-0.4, +1.1]
new channels start from the mean RGB kernel	84.3	-1.6 [-2.0, -1.3]	80.3	+0.0 [-0.4, +0.5]	47.5	-3.3 [-4.1, -2.5]
acquisition date and country in the state	86.4	+0.5 [+0.1, +0.8]	81.9	+1.5 [+1.2, +1.9]	50.7	-0.1 [-0.9, +0.6]
+ 81,000 EuroSAT-MS questions	86.5	+0.6 [+0.3, +0.9]	78.9	-1.5 [-1.9, -1.1]	49.2	-1.5 [-2.3, -0.8]

Table 4: One-factor ablations of EarthJev (seed 0).

\Delta

: accuracy of the variant minus that of EarthJev, in points, with the 95% paired bootstrap interval; positive values mean that the variant is more accurate. Pre-specified and descriptive: no correction for multiple comparisons and no claim rule.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Type	Template	Split
dominant	Which land-cover class covers most of this image?	seen
	What is the dominant land cover in this Sentinel-2 patch?	seen
	Which CORINE land-cover class occupies the largest area here?	seen
	Looking at this satellite image, which kind of land covers the biggest part of it?	unseen
presence	This image contains {c}.	seen
	Is there any {c} in this patch?	seen

Appendix

Table 5: Question templates. Seen templates are used for training, validation and the main test set; the unseen template of each type is used only for the second rendering of the test set.

Remote Sensing Visual Question Answering

Remote sensing visual question answering (VQA) studies how AI answers natural-language questions about satellite, aerial, and other Earth-observation imagery.

See all 39 papers

Sep 23, 2026

GeoNLI - A Natural Language Interpreter for Satellite Imagery

Multi-modal multitasking models have shown strong performance on remote sensing datasets. However, because these models are trained on heterogeneous data and vary across tasks, designing a unified model that performs well in captioning, visual question answering (VQA), and visual grounding remains challenging. In this work, we evaluate several models on the VRS Bench and NWPU-VHR-10 datasets. The EarthMind model demonstrates strong results in both captioning and VQA. For grounding, we propose multiple pipelines - RemoteSAM-SAM-v1, RemoteSAM-SAM-v2, and DiffuSAM - and ultimately adopt a majority-voting ensemble across EarthMind, RemoteSAM, SAM3, Falcon, RemoteSAM-SAM3-v1, RemoteSAM-SAM3-v2, and DiffuSAM predictions. Our unified, modular pipeline integrates advanced SAM variants with multimodal LLMs to jointly perform captioning, VQA, and grounding. It achieves 82% accuracy on captioning and 83.32% on VQA, with 90.94%, 52.04%, and 92.06% for binary, numeric, and semantic question types respectively. For grounding, it attains 64.94% accuracy. By combining diverse VLMs with our custom RemoteSAM-SAM3 models through ensemble majority voting, the system delivers more accurate and consistent remote-sensing understanding than task-specific approaches.

Ashutosh Gandhe, Anupam Rawat, Geet Sethi +5

Indian Institute of Technology, Bombay

Remote Sensing Visual Grounding Remote Sensing Visual Question Answering Remote Sensing Image Captioning Remote Sensing Image Understanding Unified Multimodal Models Visual Grounding+2

Ask PDF

Dec 12, 2025

VLM2GeoVec: Toward Universal Multimodal Embeddings for Remote Sensing

Satellite imagery differs from natural images in viewpoint, resolution, scale variation, and the prevalence of small objects -- demanding both region-level spatial reasoning and holistic scene understanding. Existing remote-sensing approaches are fragmented: dual-encoder retrieval models scale well but cannot interleave modalities, whereas generative assistants support grounding, yet are inefficient for retrieval. Benchmarks mirror this split: interleaved evaluations mainly target generative assistants, while cross-modal retrieval benchmarks target dual encoders. To bridge this gap, we introduce RSMEB, a unified remote sensing benchmark that evaluates cross-modal and interleaved retrieval across 21 tasks under a single ranking protocol, enabling comprehensive comparison of retrieval models on region- and geo-aware capabilities as well as conventional retrieval. As a strong reference baseline, we present VLM2GeoVec, an instruction-conditioned, single-encoder interleaving formulation tailored to remote sensing that packs image, text, bounding-box, and geo-coordinate tokens into one sequence and learns a unified embedding via contrastive training. Across RSMEB, VLM2GeoVec achieves

\textbf{26.6\%}

P@1 in region-caption retrieval (

\textbf{+25}

percentage points),

\textbf{32.5\%}

in referring-expression retrieval (

\textbf{+19}

), and

\textbf{17.8\%}

in semantic geo-aware retrieval (

\textbf{>3}$$\times

prior best), while remaining competitive in conventional scene classification and text--image retrieval in zero-shot settings. Together, the proposed suite and reference baseline standardize evaluation and deliver a unified embedder for scalable retrieval and region-/geo-aware grounding. The code, the model checkpoints, and the data are available at https://github.com/emasa/VLM2GeoVec.

Emanuel Sánchez Aimar, Gulnaz Zhambulova, Fahad Shahbaz Khan +2

Linköping University, Sweden

Multimodal Understanding Cross-Modal Learning Multimodal Embedding Cross-Modal Retrieval Vision-Language Grounding Remote Sensing Benchmarks+2

Ask PDF

Jun 9, 2026

Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Tasks

RS-MLLMs enable natural-language understanding and spatial reasoning over earth observation imagery. However, existing models support only a narrow range of sensor types and tasks, yielding a fragmented view of the earth and leaving cross-modal geoscientific knowledge largely unexploited. This work presents Earth-OneVision, a 2B RS-MLLM that unifies six sensor modalities (i.e., optical, SAR, infrared, multispectral, temporal, and video) and cross-sensor fusion across 9 task categories within a single autoregressive framework. Three dedicated mechanisms address three bottlenecks. Full-Granularity Vision-Language Alignment (FGVLA) aligns multi-level visual features with the multi-dimensional language space. Spatial-Linguistic Isomorphic Serialization (SLIS) unifies heterogeneous spatial outputs as autoregressive tokens. Progressive Cross-Modality Adaptation (PCMA) decomposes the compound domain gap into sequential stages, tackling the viewpoint and imaging physics gaps in turn. To support joint training, MMRS-OneVision is constructed with ~34M QA pairs spanning all six sensor modalities and cross-sensor fusion across 9 task categories, substantially exceeding existing RS multimodal instruction datasets. With only 2B parameters, Earth-OneVision achieves competitive or state-of-the-art results across extensive benchmarks, consistently matching or outperforming 4B-72B RS-MLLMs. It achieves 87.52% [email protected] on the OPT-RSVG testset for optical visual grounding and 80.68% on the SAR VQA benchmark SARLANG-Bench, exceeding 7B models by over 7%. It further achieves 75.74% recall on the BigEarthNet-MS testset for multispectral classification, and 81.94% MCQ accuracy on EarthMind-Bench for cross-modality reasoning.

Miaoxin Cai, Guanqun Wang, Wei Zhang +6

National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing (SBIIP), Beijing Institute of Technology, Beijing 100081, China · Advanced Research Institute of Multidisciplinary Sciences, Beijing Institute of Technology, Beijing 100081, China · School of Mechatronical Engineering, Beijing Institute of Technology, Beijing 100081, China +5

Multimodal Understanding Multimodal Grounding Remote Sensing Visual Question Answering Remote Sensing Image Understanding Multimodal Large Language Models Multimodal Reasoning+3

Ask PDF

Abstract

Remote Sensing Visual Question Answering

Explore similar work

GeoNLI - A Natural Language Interpreter for Satellite Imagery

VLM2GeoVec: Toward Universal Multimodal Embeddings for Remote Sensing

Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Tasks