cs.CVNov 2, 2026

EO-Jev: Decision Model for Earth Observation

Authors: Mohamed Atef Abouelhadid, Diaa Addeen Abuhani, Muhammad Akhtar Munir, Fahad Shahbaz Khan, Salman Khan

Organizations: Mohamed Bin Zayed University of Artificial Intelligence · Massachusetts Institute of Technology · Linköping University Sweden · Australian National University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Earth observation (EO) workflows require selecting among known outcomes, estimating confidence, and processing large volumes of imagery. Generative vision-language models can address these tasks, but autoregressive decoding is computationally costly and their confidence estimates are unreliable. Decision models offer an alternative, returning probabilities over candidate answers in a single forward pass. However, their accuracy, calibration, and transfer across EO tasks remain insufficiently understood. We investigate these properties through a diagnostic study of d3-flash (8.4B parameters) across 30 settings from 23 public EO benchmarks. We find that compositional reasoning transfers well and calibration improves over its Qwen3.5-9B backbone on 24 settings, while counting, change quantification, damage grading, and absent-object recognition remain challenging. To address these limitations, we adapt d3-flash through low-rank fine-tuning on public EO training splits, producing EO-Jev. EO-Jev improves significantly over d3-flash on 26 of 30 settings and matches or exceeds specialist references on 20 of 28. It also achieves an average gain of 8.0 points across 12 held-out settings and lower expected calibration error than specialists on 21 of 28 settings. These improvements come at a modest cost of 1.8 points in general decision performance. Our results demonstrate the potential of adapting general decision models into competitive EO generalists while highlighting persistent challenges in quantitative perception, option-order consistency, and calibration under distribution shift.

Figures & tables

Figure 1: EO-Jev is more accurate, better calibrated and faster. Centre: mean accuracy per capability (30 benchmarks). Corners: real questions with each model’s answer and confidence; footers give benchmark accuracy (specialist →\rightarrow EO-Jev) and throughput vs. the specialist. Bottom: per-capability medians of accuracy gain and throughput vs. the best specialist, and ECE drop vs. d3-flash; green = better, red = worse.
Figure 2: EO-Jev Architecture and Training. EO images (single images, pre/post pairs or time series) are paired with typed questions with predefined answers. General decision data is replayed in 20% of each batch. The vision encoder and text tower Φθ\Phi_{\theta} process the images and questions, respectively. A frozen readout WrW_{r} converts the final hidden state hh into option probabilities pp . Training: LoRA adapters in the vision encoder and text tower are optimized using KL divergence against target distributions qq , with an additional ranked probability score (RPS) on ordinal questions. Inference: merged adapters ( W+BAW+BA ) enable single-pass predictions at d3-flash cost. Answers are accepted when confidence exceeds a per-question threshold τq\tau_{q} and otherwise deferred to an analyst. Examples and probabilities are illustrative.
Table 1: Earth-observation results. All models answer identical questions (SAR inputs excluded). The metric is accuracy, except exact-set match on multi-select rows, strict accuracy on LHRS-Bench and correct abstention on VHM deceptive questions. Best EO specialist : the strongest open-weight EO model in our common question format. Domain : whether EO-Jev was trained on other images from the benchmark’s training split. Colours are scaled within each row. ∗ : the paired 95% bootstrap interval (over images) of EO-Jev −- d3-flash excludes zero. The specialist gap is a plain difference. Decision Index : general (non-EO) retention check.
Figure 3: Qualitative examples across capability groups. Each panel shows the input image(s), the question and its options (gold option highlighted in green), and the top answer of each model with its probability, or “written” for specialists whose best mode is a generated answer. Badges: I = in-domain, O = out-of-domain for EO-Jev. In (b)–(g), d3-flash sometimes answers incorrectly with high confidence, defined as assigning a probability of at least 0.8 to the selected answer (e.g., predicting the Americas for an Oceanian site in (f)), the color cannot be determined in (g)), while EO-Jev selects the correct option; in (d) and (f) the best specialist is wrong as well. This includes two out-of-domain settings, NWPU VHR-10 counting (c) and VHM deceptive questions (g), where no training image came from the benchmark’s dataset. Panel (a) is an out-of-domain scene that every model answers correctly, EO-Jev with p=1.00p=1.00 . All images, questions and probabilities are taken unchanged from the evaluation runs.
Figure 4: Calibration change from d3-flash to EO-Jev. Each arrow runs from d3-flash’s ECE (open circle) to EO-Jev’s (filled circle); lower is better. Rows are grouped by Δ\Delta ECE == EO-Jev −- d3-flash: much better calibrated ( Δ\Delta ECE ≤−0.05\leq-0.05 ), better calibrated, no significant change, and worse calibrated. ∗ : the paired 95% bootstrap interval (over images) excludes zero. Badges: I = in-domain, O = out-of-domain, G = general (non-EO) retention check. All EO rows in the bottom tier are over-confident; the Decision Index is slightly under-confident.
Figure 5: Inference cost against the specialists. (a) Specialist median latency per question as a multiple of EO-Jev’s, one question per call. (b) EO-Jev throughput as a multiple of the specialist’s, each at its best batch size. Bars are grouped by speed-up; red rows are those where the specialist is faster. Badges give the specialist’s best-accuracy mode (P: prompted, W: wrapped/generative). All models run on the same exclusive node (4 ×\times A100, BF16), on the same 160-question seeded sample per row, with image decoding inside the timed region. Medians: 3.2 ×\times in (a) and 6.4 ×\times in (b).
Figure 6: EO capability relative to specialist level. For each setting with a specialist reference, EO-Jev’s score as a percentage of the best open EO specialist’s. Settings are grouped into tiers: exceeds specialist level (above 110%), at specialist level (100–110%) and approaching (85–100%). No setting falls below 85%. Bars beyond 165% are clipped. Badges: I = in-domain, O = out-of-domain.
CapabilitySettingsSpecialist
Scene classificationUC Merced, AID, WHU-RS19, NWPU-RESISC45RSThinker ( Liu et al., 2025 )
 SIRI-WHU, EuroSAT, fMoW facility typeGeoZero ( Wang et al., 2025a )
Objects & attributesVRSBenchRSThinker ( Liu et al., 2025 )
 RSVQA-LREarthDial ( Soni et al., 2024 )
 RSVQA-HRGeoZero ( Wang et al., 2025a )
CountingNWPU VHR-10, DOTA-v2RSThinker ( Liu et al., 2025 )
Table 2: Specialist reference models. The strongest open-weight EO specialist per setting in our common question format.
More in this topic

Remote Sensing VLMs

See all 46 papers

Explore similar work

CardsList
Oct 13, 2026

The Last Remote Sensing Model Built by Humans? Casting Any Earth Observation Task as Recursive Self-Improvement

Earth observation (EO) models are still built by hand. For each new task, experts design a training recipe and tune it against a benchmark. We show that this loop can be handed to an agent. Any verifiable task can be cast as a harness and a verifier, where the harness defines what the agent may use and the verifier defines what counts as success. Every EO task can be made verifiable, through labeled benchmarks, independent sensors, physical models or the repeated observation of the same place. Given only these two components, a general-purpose coding agent builds the EO model and improves it recursively on its own outputs. Settings long treated as separate research problems, from learning with a handful of labels to adapting across domains, become choices of harness and verifier. On six EO benchmarks spanning optical, SAR and multispectral time-series imagery, with outputs that range from land-cover maps and ship instances to canopy heights and answers about large images, agent-built models surpass the state of the art on every one, and on land-cover mapping, they do so from 11% of the training labels. This establishes a new paradigm for EO, in which people design harnesses and verifiers and agents build the models, so EO models improve as agents do.

Sijie Zhao, Zeqi Zhou, Heli Qi +6

The University of Tokyo · RIKEN · Stanford University

Remote SensingRecursive Self-ImprovementAutonomous Research Agents
AskPDF
May 2, 2026

EO-Gym: A Multimodal, Interactive Environment for Earth Observation Agents

Earth Observation (EO) analysis is inherently interactive: resolving uncertainty often requires expanding the region of interest, retrieving historical observations, and switching across sensors such as optical and Synthetic Aperture Radar. However, most EO benchmarks collapse this process into fixed-input, single-turn tasks. To address this gap, we present EO-Gym, a controlled executable framework for multimodal, tool-using EO agents that formulates EO analysis as a Gymnasium-style local geospatial workspace backed by more than 660k multimodal files indexed by location, time, and sensor type, with 35 EO-specialized tools spanning six task families. Built on this environment, we construct EO-Gym-Data, a benchmark of 9,078 trajectories and 34,604 reasoning steps, and grounded in eight public EO datasets together with Landsat and Sentinel-2 imagery. Evaluating 1010 open and closed VLMs shows that strong general-purpose models still struggle with interactive EO reasoning, especially on temporal and cross-modal workflows. As a reference baseline, EO-Gym-4B, obtained by fine-tuning Qwen3-VL-4B-Instruct on EO-Gym-Data, improves overall Pass@3 from 0.490.49 to 0.740.74 under the main evaluation setting. O-Gym provides a reproducible environment for interactive EO agents, operationalizing EO as an evidence-gathering problem that requires planning across geospatial, temporal, and sensing modality.

Sai Ma, Zhuang Li, Sichao Li +4

The Australian National University · Royal Melbourne Institute of Technology · University of Sydney +1

Spatiotemporal ReasoningRemote SensingGeospatial ReasoningRemote Sensing BenchmarksAgent BenchmarksMultimodal Tool Use
AskPDF
Jul 29, 2026

OVEarth-Bench: Evaluating Category Breadth and Query Diversity for Open-Vocabulary Earth Observation

Open-vocabulary Earth observation (EO) aims to localize geospatial concepts specified in natural language rather than a fixed label set. Existing benchmarks, however, usually cover narrow category vocabularies or limited query forms. To fill this gap, we introduce OVEarth-Bench, which extends existing evaluation in two directions: category breadth, through broad hierarchical category coverage with positive and negative expressions, and query diversity, through vocabulary, referring, and reasoning queries. The benchmark supports mask and box localization under a unified zero-shot protocol. We evaluate a broad set of general and EO-specific methods. The evaluation reveals that: (1) the performance of current methods remains limited, while broader category coverage yields more stable model rankings; (2) MLLM-based methods achieve the strongest overall performance; and (3) EO-specific methods generally underperform general models and rarely match the strongest methods. These findings provide guidance for future open-vocabulary EO method design and highlight the importance of developing more realistic, diverse, high-quality, and large-scale benchmarks for reliable evaluation. Our data and evaluation package are released at https://earth-insights.github.io/OVEarth-bench.

Kaiyu Li, Zepeng Xin, Zixuan Jiang +4

Xi’an Jiaotong University

VLM EvaluationOpen-Vocabulary Object DetectionRemote SensingOpen-Vocabulary Semantic SegmentationRemote Sensing Benchmarks
AskPDF