EarthJev: Calibrated, Non-Generative Answers to Natural-Language Questions about Sentinel-2 Land Cover

Domenico Benfenati; Antonio Maria Rinaldi; Cristian Tommasino

cs.CVNov 1, 2026

EarthJev: Calibrated, Non-Generative Answers to Natural-Language Questions about Sentinel-2 Land Cover

Authors: Domenico Benfenati, Antonio Maria Rinaldi, Cristian Tommasino

Organizations: Department of Electrical Engineering and Information Technology (DIETI), University of Naples Federico II, Via Claudio, 21, Naples, 80125, Italy

Abstract

Vision-language models for remote sensing answer questions about satellite images by generating text: every answer costs a decoding loop, has to be parsed back to an option, and comes without a usable probability. We present EarthJev, a 0.8B-parameter model that answers typed questions about multispectral Sentinel-2 patches without generating any token. All 12 bands enter the vision encoder of Qwen3.5-0.8B through a widened input layer initialised from the RGB kernels, and, following the open OneJev recipe, the answer is read from the logits of the option letters at a single position and trained with cross-entropy plus Brier score. Under a pre-specified analysis plan, we compare it with the same backbone fine-tuned to generate the answer and with three multimodal language models fine-tuned with LoRA, all trained on the same 70,890 questions from 5% of BigEarthNet v2.0 and timed in one job on the same GPU; VQA models that classify over a fixed answer vocabulary are reported as a reference for the cost of answering only the questions seen in training. EarthJev answers a question in 19.9 ms, 4.3 to 8.7 times faster than the multimodal language models and 6.8 times faster than its own backbone used generatively; when all the questions about an image are asked together, it answers them in 42.8 ms, 3.7 to 10.1 times faster than the batched generative models. On 36,028 test questions it reaches 86.0% accuracy (three seeds), 0.6 and 1.2 points below the two 2B multimodal models and 3.5 points below the remote-sensing EarthDial-4B, with answers that are valid by construction and calibrated probabilities (ECE 0.006). Replacing the nine non-RGB bands with their mean costs 4.5 points, so the model uses the multispectral input, and in an ablation training the whole vision encoder raises accuracy to 88.4% at the same cost. Code and model: https://github.com/CosmoIknosLab/EarthJev

Figures & tables

Ours
		CORINE test (36,028 q.)		RSVQAxBEN	Latency per question
Model	Params	Acc. (%)	ECE	Acc. (%)	p50 (ms)	Slower
EarthJev (ours)	0.8B	86.0 ±0.1	0.006	81.0 ±0.6	19.9	–
Same backbone and input, generative fine-tuning
Qwen3.5-0.8B, generative	0.8B	86.8 ±0.2	0.005	80.5 ±0.6	134.7	6.8 $\times$
Multimodal LLMs, LoRA fine-tuning

Table 1: Speed against quality. CORINE: seed mean (standard deviation as subscript) of the accuracy on the 36,028 test questions, and mean of the per-seed ECE (15 bins, before temperature scaling). Latency per question at batch 1 and how many times slower than EarthJev each model is, from Table 2 . Fixed-vocabulary models answer only the questions and options seen in training.

			One question per request			All questions of an image
Model	Params	Input	p50 (ms)	p95 (ms)	Slower	Image (ms)	Per q. (ms)	Slower	Mem. (GB)
Ours
EarthJev (ours)	0.8B	12 bands	19.9	20.1	–	42.8	7.9	–	3.5
Same backbone and input, generative fine-tuning
Qwen3.5-0.8B, generative	0.8B	12 bands	134.7	144.4	6.8 $\times$	191.9	36.9	4.5 $\times$	2.2
Multimodal LLMs, LoRA fine-tuning

Table 2: Inference cost of every model, measured in one job on one H100 PCIe GPU (bfloat16; 100 test images after 10 warm-up images; each timer runs from the decoded image in memory to the answer, preprocessing and tokenisation included). Slower : p50 latency of the model divided by that of EarthJev in the same scenario (below 1: faster than EarthJev). Image : p50 latency of a request with all the questions of an image (6.1 on average); Per q. : mean latency per question in that scenario. In the second scenario EarthJev encodes the image once and answers all its questions as branches of a shared prefix; the generative models answer them in one batched generation and the fixed-vocabulary models in one batched forward pass.

Ours
			CORINE test (36,028 questions)							RSVQA-	BigEarthNet.txt
Model	Params	Input	All	Dom.	Pres.	Share	Loc.	Comp.	ECE	xBEN	Binary	MC
EarthJev (ours)	0.8B	12 bands	86.0 ±0.1	83.6	93.2	82.9	66.1	87.9	0.006	81.0	59.6	40.1
Same backbone and input, generative fine-tuning
Qwen3.5-0.8B, generative	0.8B	12 bands	86.8 ±0.2	85.0	93.6	83.7	66.6	88.5	0.005	80.5	59.3	17.1
Multimodal LLMs, LoRA fine-tuning

Table 3: All models trained on the same 70,890 CORINE questions (three seeds). CORINE: accuracy (%) overall (seed mean, standard deviation as subscript) and per question type, seed-averaged; ECE: mean of the per-seed ECEs, 15 bins, before temperature scaling. RSVQAxBEN: 29,501 presence questions. BigEarthNet.txt: 6,927 binary and 5,550 four-option questions; the fixed-vocabulary models cannot answer them (their answer vocabulary does not cover them). Transfer tests are never used for training.

	CORINE		RSVQAxBEN		BigEarthNet.txt
Variant	Acc.	$\Delta$ [95% CI]	Acc.	$\Delta$ [95% CI]	Acc.	$\Delta$ [95% CI]
EarthJev (seed 0)	85.9	–	80.3	–	50.8	–
whole vision encoder trained	88.4	+2.5 [+2.2, +2.8]	83.0	+2.7 [+2.2, +3.1]	51.1	+0.3 [-0.4, +1.1]
new channels start from the mean RGB kernel	84.3	-1.6 [-2.0, -1.3]	80.3	+0.0 [-0.4, +0.5]	47.5	-3.3 [-4.1, -2.5]
acquisition date and country in the state	86.4	+0.5 [+0.1, +0.8]	81.9	+1.5 [+1.2, +1.9]	50.7	-0.1 [-0.9, +0.6]
+ 81,000 EuroSAT-MS questions	86.5	+0.6 [+0.3, +0.9]	78.9	-1.5 [-1.9, -1.1]	49.2	-1.5 [-2.3, -0.8]

Table 4: One-factor ablations of EarthJev (seed 0).

\Delta

: accuracy of the variant minus that of EarthJev, in points, with the 95% paired bootstrap interval; positive values mean that the variant is more accurate. Pre-specified and descriptive: no correction for multiple comparisons and no claim rule.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Type	Template	Split
dominant	Which land-cover class covers most of this image?	seen
	What is the dominant land cover in this Sentinel-2 patch?	seen
	Which CORINE land-cover class occupies the largest area here?	seen
	Looking at this satellite image, which kind of land covers the biggest part of it?	unseen
presence	This image contains {c}.	seen
	Is there any {c} in this patch?	seen

Appendix

Table 5: Question templates. Seen templates are used for training, validation and the main test set; the unseen template of each type is used only for the second rendering of the test set.

Remote Sensing Visual Question Answering

Remote sensing visual question answering (VQA) studies how AI answers natural-language questions about satellite, aerial, and other Earth-observation imagery.

See all 41 papers

Sep 23, 2026

GeoNLI - A Natural Language Interpreter for Satellite Imagery

Multi-modal multitasking models have shown strong performance on remote sensing datasets. However, because these models are trained on heterogeneous data and vary across tasks, designing a unified model that performs well in captioning, visual question answering (VQA), and visual grounding remains challenging. In this work, we evaluate several models on the VRS Bench and NWPU-VHR-10 datasets. The EarthMind model demonstrates strong results in both captioning and VQA. For grounding, we propose multiple pipelines - RemoteSAM-SAM-v1, RemoteSAM-SAM-v2, and DiffuSAM - and ultimately adopt a majority-voting ensemble across EarthMind, RemoteSAM, SAM3, Falcon, RemoteSAM-SAM3-v1, RemoteSAM-SAM3-v2, and DiffuSAM predictions. Our unified, modular pipeline integrates advanced SAM variants with multimodal LLMs to jointly perform captioning, VQA, and grounding. It achieves 82% accuracy on captioning and 83.32% on VQA, with 90.94%, 52.04%, and 92.06% for binary, numeric, and semantic question types respectively. For grounding, it attains 64.94% accuracy. By combining diverse VLMs with our custom RemoteSAM-SAM3 models through ensemble majority voting, the system delivers more accurate and consistent remote-sensing understanding than task-specific approaches.

Ashutosh Gandhe, Anupam Rawat, Geet Sethi +5

Indian Institute of Technology, Bombay

Remote Sensing Visual Grounding Remote Sensing Visual Question Answering Remote Sensing Image Captioning Remote Sensing Image Understanding Unified Multimodal Models Visual Grounding+2

Ask PDF

Nov 4, 2026

EarthJev: Efficient Visual Evidence Acquisition and Decision for UHR Remote Sensing Understanding

Ultra-high-resolution (UHR) remote sensing visual question answering (VQA) requires locating fine-grained evidence in large scenes under limited visual input budgets. Active observation addresses this challenge through selective exploration, but repeated generative judgments can make region search costly. Its decisions also concern distinct prediction targets: spatial pointer validity, evidence presence, and reasoning correctness. We propose EarthJev, a framework that separates region scoring from answer generation and uses target-specific decision signals while keeping the answering model frozen. A spatial pointer verifier uses visual and geometric cues to guide exploration or bypass search. An evidence-aware decision mechanism scores compact candidate views in batches using a lightweight head on frozen representations, without generating textual region judgments. Format-specific confidence calibration then guides selection between candidate answers generated from memory and overview presentations of the acquired evidence. Across five benchmarks, EarthJev achieves a mean accuracy of 53.06% with Qwen3.5-9B, outperforming ZoomSearch with the same answering model by 2.33 to 14.73 percentage points per benchmark while reducing mean latency from 11.30 - 1.67 seconds per question. With four additional frozen answering models, model-specific fitting of the decision modules and calibration mappings yields mean accuracy gains of 3.41 - 6.53 percentage points over the corresponding single-pass baselines. Code is available at https://github.com/Da1daidaidai/EarthJev

Pengyu Dai, Yao Zhang, Jian Song +6

RIKEN Center for Advanced Intelligence Project (AIP), RIKEN, Tokyo, Japan. · The University of Tokyo, Japan. · the State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan, China.

Remote Sensing Visual Question Answering Visual Question Answering Efficient VLM Inference

Ask PDF

Dec 12, 2025

VLM2GeoVec: Toward Universal Multimodal Embeddings for Remote Sensing

Satellite imagery differs from natural images in viewpoint, resolution, scale variation, and the prevalence of small objects -- demanding both region-level spatial reasoning and holistic scene understanding. Existing remote-sensing approaches are fragmented: dual-encoder retrieval models scale well but cannot interleave modalities, whereas generative assistants support grounding, yet are inefficient for retrieval. Benchmarks mirror this split: interleaved evaluations mainly target generative assistants, while cross-modal retrieval benchmarks target dual encoders. To bridge this gap, we introduce RSMEB, a unified remote sensing benchmark that evaluates cross-modal and interleaved retrieval across 21 tasks under a single ranking protocol, enabling comprehensive comparison of retrieval models on region- and geo-aware capabilities as well as conventional retrieval. As a strong reference baseline, we present VLM2GeoVec, an instruction-conditioned, single-encoder interleaving formulation tailored to remote sensing that packs image, text, bounding-box, and geo-coordinate tokens into one sequence and learns a unified embedding via contrastive training. Across RSMEB, VLM2GeoVec achieves

\textbf{26.6\%}

P@1 in region-caption retrieval (

\textbf{+25}

percentage points),

\textbf{32.5\%}

in referring-expression retrieval (

\textbf{+19}

), and

\textbf{17.8\%}

in semantic geo-aware retrieval (

\textbf{>3}$$\times

prior best), while remaining competitive in conventional scene classification and text--image retrieval in zero-shot settings. Together, the proposed suite and reference baseline standardize evaluation and deliver a unified embedder for scalable retrieval and region-/geo-aware grounding. The code, the model checkpoints, and the data are available at https://github.com/emasa/VLM2GeoVec.

Emanuel Sánchez Aimar, Gulnaz Zhambulova, Fahad Shahbaz Khan +2

Linköping University, Sweden

Multimodal Understanding Cross-Modal Learning Multimodal Embedding Cross-Modal Retrieval Vision-Language Grounding Remote Sensing Benchmarks+2

Ask PDF

Abstract

Remote Sensing Visual Question Answering

Explore similar work

GeoNLI - A Natural Language Interpreter for Satellite Imagery

EarthJev: Efficient Visual Evidence Acquisition and Decision for UHR Remote Sensing Understanding

VLM2GeoVec: Toward Universal Multimodal Embeddings for Remote Sensing