EVE: A Domain-Specific LLM Framework for Earth Intelligence
Authors: Àlex R. Atrio, Antonio Lopez, Jino Rohit, Yassine El Ouahidi, Marcello Politi, Vijayasri Iyer, Umar Jamil, Sébastien Bratières, Nicolas Longépé
Organizations: Pi School · Mistral AI · Translated · ESA Φ-lab
We introduce Earth Virtual Expert (EVE), the first open-source, end-to-end initiative for developing and deploying domain-specialized LLMs for Earth Intelligence. At its core is EVE-Instruct, a domain-adapted 24B model built on Mistral Small 3.2 and optimized for reasoning and question answering. On newly constructed Earth Observation and Earth Sciences benchmarks, it outperforms comparable models while preserving general capabilities. We release curated training corpora and the first systematic domain-specific evaluation benchmarks, covering MCQA, open-ended QA, and factuality. EVE further integrates RAG and a hallucination-detection pipeline into a production system deployed via API and GUI, supporting 350 pilot users so far. All models, datasets, and code are ready to be released under open licenses as contributions to our field at huggingface.co/eve-esa and github.com/eve-esa.
Figures & tables
Figure 1: System architecture of EVE depicting component interactions.
Task Type
Samples
MCQA (Multiple Answer)
431
MCQA (Single Answer)
1261
Hallucination Detection
2326
Open-Ended QA (No context)
1257
Open-Ended QA (with Context)
418
Table 1: EO and Earth Sciences benchmark suite. Multiple Choice Question Answering (MCQA) differ in the number of correct options. Hallucination detection is a balanced binary QA classification task. Open-ended QA evaluate self-contained and context-grounded questions.
Task Type
Example
MCQA (Multiple Answer)
Q: Which statements are true about surging glaciers? A) Surging glaciers are subject to cyclical flow instabilities B) Surging glaciers flow at a constant rate down a mountain C) […] a glacier’s velocity moves up to 100 times faster than normal D) Surging glaciers suddenly increase in mass → Answers: A, C
MCQA (Single Answer)
Q: What is the main obstacle to using optical EO in the aftermath of a severe weather event? A) Sensor calibration issues B) Atmospheric interference C) Low light conditions D) Cloud cover → Answer: D
Hallucination Detection
Q: What advantages do radar systems offer for environmental monitoring compared to optical sensors? Hallucinated Answer: Radars are active sensors that create their own electromagnetic signals. Radars cannot operate at night and are ineffective in hazy or smoky environments. SAR […] uses the movement between the antenna and the target area […] to create high-resolution images for remote sensing. Correct Answer: Radars are active sensors […] Radars can operate day and night, and penetrate clouds, haze, and smoke. SAR […] → Label: Hallucinated ( underlined span)
Open-Ended QA
Q: What is the calving front of an ice stream? A: The point where icebergs are lost from a glacier.
Open-Ended QA w/ Context
Q: How does LiDAR map buildings? [3 source documents provided as context] A: LiDAR maps buildings by sending laser pulses that reflect off rooftops, walls, and other structural surfaces, recording precise 3D coordinates. These points are stored in point clouds […]
Table 2: Representative examples from each evaluation category in the EO and Earth Sciences benchmark suite (see Table 1 ). In hallucination detection, underlined spans are the annotated hallucinated content.
Long-form text: 30% (10B)
Instruction-formatted data: 70% (23.5B)
Long-Form Replay
50%
Instruct-formatted Replay
60%
EO and Earth Sciences
50%
EO and Earth Sciences
40%
Raw
2%
ContextQA (Best)
12%
Best Chunks
14%
ContextQA (Good)
21%
Active Reading (Agnostic)
28%
SelfQA
2.6%
Active Reading (Specific)
6%
MultiHop QA
2.1%
Table 3: Distribution of training data across long-form and instruction-formatted data.
MCQA Mult.
MCQA Sing.
Hallucin.
Open-Ended
Open-Ended w/ Context
Model
Size (B)
IoU
Acc.
Acc.
F1
Judge
EVE WR
Judge
EVE WR
Rank ↓
Llama4 Scout
109-A17
80.32
71.23
91.67
66.08
87.37
53.95
71.73
58.31
3.67
Qwen3
30-A3
78.40
66.36
93.02
81.30
94.92
50.70
81.81
52.12
2.67
Gemma3
27
73.60
57.54
87.31
75.07
94.41
50.92
78.31
51.58
3.83
Mistral Small 3.2
24
80.19
70.30
83.51
82.19
91.78
51.69
71.93
57.27
3.50
EVE-Instruct
24
86.12
77.73
96.35
84.70
96.40
—
78.28
—
1.33
Table 4: Model performance across EO and Earth Sciences benchmark tasks presented in Table 1 (0-shot). EVE WR (win rate) indicates percentage of pairwise comparisons where EVE-Instruct is preferred over the comparison model ( >50% means EVE is preferred). Rank ↓ (lower is better) reports the average per-metric rank across MCQA multiple (IoU and Accuracy), MCQA single (Accuracy), Hallucination (F1), Open-ended QA (Judge), and Open-Ended QA with Context (Judge).
Category
Small 3.2
EVE-Instruct
Δ
Math & Reasoning
50.8
54.9
+4.1
Coding
55.6
56.5
+0.9
Knowledge
67.7
69.0
+1.3
Tool Calling
87.9
90.9
+3.0
Instruction Following
80.1
81.2
+1.1
Chat Quality
90.8
91.7
+0.9
Table 5: General-domain performance after domain adaptation (category-level averages over several standard benchmarks, 0–100 scale).
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Source
Released Tokens
%
Licence
MDPI
1.3 B
46.6
CC-BY
Copernicus
723 M
25.9
CC-BY
NCBI
485 M
17.4
CC-BY
ISPRS
74.1 M
2.7
CC-BY
Wiley
71.6 M
2.6
CC-BY
Elsevier
43.7 M
1.6
CC-BY
Appendix
Table 6: Token distribution across data sources in the released subset of the open-access EO and Earth Sciences corpus (2.8B tokens released out of 4.2B total open-access tokens).
Tool
Text
Formulas
Tables
Avg.
Markitdown
0.81
0.00
0.00
0.27
Docling
0.79
0.00
0.40
0.40
Pymupdf4llm
0.81
0.00
0.26
0.36
Pypdf2
0.84
0.00
0.00
0.28
Marker
0.80
0.31
0.42
0.51
Unstructured
0.79
0.00
0.00
0.26
Appendix
Table 7: OCR benchmark results (Normalized Levenshtein Similarity) across tools, evaluating text, formula, and table extraction. The Avg. column reports the mean across the three categories.
Benchmark
Small 3.2
EVE
Δ
aime25@16
26.7
35.2
+8.5
aime24@16
37.1
41.2
+4.1
math
88.6
88.4
-0.2
Average (Math & Reasoning)
50.8
54.9
+4.1
livecodebench_p@1
36.4
39.1
+2.7
mbpp_p@1
74.7
73.9
-0.8
Appendix
Table 8: Evaluation results (0–100 scale) comparing Mistral Small 3.2 and EVE-Instruct across general-domain benchmarks. Category averages are shown for each task group. Tool Calling, Instruction Following, and Chat Quality correspond to private internal evaluation.
MCQA Mult.
MCQA Sing.
Hallucin.
Open-Ended
Open-Ended w/ Context
Model
Size (B)
IoU
Acc.
Acc.
F1
Judge
EVE WR
Judge
EVE WR
Rank ↓
GPT-4.1
1800 *
87.56
78.19
94.37
81.58
96.48
49.77
86.65
49.22
2.83
Qwen3
235-A22
87.40
80.97
95.16
84.40
97.05
48.24
86.10
50.09
2.17
MiniMax m2.5
230A10
84.82
77.72
94.95
83.77
91.00
51.10
81.57
51.20
5.17
GPT OSS
120A5
84.56
76.79
89.77
89.92
94.20
50.30
84.80
50.70
4.83
Mistral Medium 3.1
200 *
85.44
76.33
95.00
76.89
96.45
50.20
86.44
50.99
4.17
Appendix
Table 9: Extension of Table 4 to larger-scale models under the same evaluation setup. Rank ↓ (lower is better) reports the average per-metric rank across MCQA multiple (IoU and Accuracy), MCQA single (Accuracy), Hallucination (F1), Open-ended QA (Judge), and Open-Ended QA with Context (Judge). * Model size is reported when publicly available; otherwise estimated internally.
Open-Ended
Open-Ended w/ Context
Claude Judge
Gemini Judge
Claude Judge
Gemini Judge
Model
Size (B)
Score
EVE WR
Score
EVE WR
Score
EVE WR
Score
EVE WR
Rank prev↓
Rank ↓
Δ Rank ↓
Llama Scout 4
109-A17
69.22
52.7
78.09
54.9
55.64
59.83
62.44
59.45
3.67
3.50
− 0.17
Qwen3
30-A3
82.75
51.2
88.89
51.2
69.13
53.71
64.59
53.23
2.67
2.75
+ 0.08
Gemma 3
27
80.82
51.6
87.55
51.4
62.48
52.75
63.49
51.91
3.83
3.88
+ 0.05
Mistral 3.2 Small
24
73.76
51.0
84.04
52.5
57.21
57.19
60.28
56.69
3.50
3.75
+ 0.25
Appendix
Table 10: Comparison of models on Open-Ended and Open-Ended with Context tasks using two judges (Claude and Gemini - anthropic/claude-sonnet-4.6 and google/gemini-2.5-flash). Rank prev refers to the single-judge baseline ranking from Table 4 . Rank ↓ (lower is better) is the average per-metric rank across the four judge score columns. Δ Rank denotes the difference between the two rankings.
TelQNA
Satcom Open-Ended
Model
Size (B)
Acc.
Judge
Llama4 Scout
109-A17
74.57
75.00
Qwen3
30-A3
73.81
83.00
Gemma3
27
70.84
81.25
Mistral Small 3.2
24
72.63
76.50
EVE-Instruct
24
75.72
81.25
Appendix
Table 11: Model performance on TelQNA and Satcom Open-Ended benchmarks (0-shot).
Chunker
Chunk Size
Embedding
Doc recall
Passage recall
IoU
Precision
Recall
Two-pass
512
INDUS 512
91.60
57.70
2.77
2.78
77.00
Two-pass
512
INDUS 512*
91.60
50.80
2.64
2.65
73.20
Two-pass
512
INDUS 1024
83.24
38.60
2.34
2.35
59.90
Two-pass
512
INDUS 1024*
82.60
38.50
2.32
2.33
59.70
Two-pass
512
Qwen3-Embedding-4B
95.70
65.60
2.93
2.94
85.30
Two-pass
512
Qwen3-Embedding-4B*
95.60
63.60
2.94
2.94
85.00
Appendix
Table 12: Performance comparison of different embedding models across chunking strategies and chunk sizes. All metrics are expressed as percentages with two decimal precision. Precision and recall are per token, considering 10 as the number of passages retrieved. * indicates quantized embeddings. The Two-pass chunker refers to the approach presented in Section 8 . The Recursive chunker is based on LangChain’s RecursiveCharacterTextSplitter .
Collection
Chunk Size
Reranker
Ref Retrieved Ratio @10
MRR @10
Retrieval
Reranked
Retrieval
Reranked
INDUS 512*
512
INDUS
51.00
54.10
45.90
55.70
INDUS 512*
512
Qwen3-4B
51.00
54.70
45.90
61.70
Qwen3-Embedding-4B*
512
INDUS
63.30
62.80
62.60
60.10
Qwen3-Embedding-4B*
512
Qwen3-4B
63.30
65.10
62.60
69.30
Qwen3-Embedding-4B*
1024
Qwen3-4B
68.60
71.60
65.40
73.40
Appendix
Table 13: Performance comparison at top-10 with K=20 retrievals. Ref Retrieved Ratio @10 measures the percentage of queries for which at least one relevant chunk appears in the top-10 reranked results. MRR @10 (Mean Reciprocal Rank) is the average of the reciprocal rank of the first relevant chunk across queries, rewarding systems that place relevant chunks higher in the ranked list. All metrics are expressed as percentages with two decimal precision. * indicates quantized embeddings.
Figure 2: End-to-end architecture of the deployed EVE system.
Figure 3: LLM-as-a-judge evaluation prompt for Open-Ended with retrieval context.
Figure 4: Prompt used to filter corpus chunks before Active Reading synthesis. Only Best -rated chunks enter the Active Reading pipeline.
Figure 5: Prompt used for task-specific Active Reading. The model first generates questions from the source chunk, then devises two active learning strategies per question to guide synthesis.
Figure 6: Prompt used for predefined Active Reading. The model applies strict eligibility rules to select at most 2 strategies from a fixed predefined set.
Figure 7: Prompt used to generate synthetic training documents by applying a selected Active Reading strategy to a source chunk. The model is explicitly instructed to output only the resulting document.
Figure 8: Prompt used to generate SelfQA samples. Context-grounded QA pairs are reformulated into self-contained questions that can be answered from the model’s parametric knowledge alone.
Figure 9: Prompt used to filter ContextQA samples. A two-step assessment first applies hard filters, then evaluates quality on a five-point scale. Best and Good samples are retained for training as described in Table 3 .
Real-world Earth observation (EO) agents must translate high-level scientific questions into executable workflows to acquire observations, prepare data, perform domain computations, and derive conclusions from runtime evidence. Existing EO agents typically start from supplied observations, while benchmarks typically provide prepared inputs or candidate answers, leaving full-chain open-world EO execution largely untested. We present Earth-Agent-Pro, an execution-adaptive Plan-and-Execute framework using expert-authored skills to constrain planning and runtime tool use. Workflow-centered structured memory records planned steps, accepted evidence, and their dependencies, enabling repair of only the affected workflow suffix when runtime evidence invalidates a step. Separate large language model adapters use sequence-level supervised fine-tuning for planner workflow composition and node-level group relative policy optimization with locally verifiable rewards for executor tool-argument grounding. Earth-Bench-Pro instantiates 248 expert-curated task cores as 744 questions under three matched regimes. Its 248 Open-World Execution questions span RGB imagery, spectral observations, and remote sensing products, pairing high-level requests with runtime data requirements, executable trajectories, and open-ended answers grounded in execution evidence. With a shared GPT-5 backbone, Earth-Agent-Pro achieves 66.13% LLM-as-Judge accuracy, exceeding ReAct by 20.95 points in this metric and 24.44 points in Tools-In-Order. Joint adapter tuning raises Qwen3.5-9B LLM-as-Judge accuracy from 38.31% to 50.00%, an 11.69-point gain over the untuned configuration. Planning-only evaluation and execution with the reference workflow show that the adapters improve workflow composition and argument grounding, respectively. Code and datasets will be released soon.
Zhutao Lv, Chenhao Dang, Yi Feng +5
Tsinghua Shenzhen International Graduate School, Tsinghua University · Sun Yat-Sen University · Shanghai Jiao Tong University +3
Earth Observation (EO) provides critical planetary data for environmental monitoring, disaster management, climate science, and other scientific domains. In this work we ask: Are AI systems ready for reliable Earth Observation? To answer this, we introduce UnivEARTH, a coding benchmark of 408 yes/no questions from NASA Earth Observatory articles across 7 various topics and over 15 satellite instruments and sources. Using Google Earth Engine API as a tool in a zero-shot setup, LLM agents achieve an accuracy of 40.0% where the code fails to run over 44% of the time. To better understand LLM agent behavior, we also analyze the impact of using the JavaScript API versus Python and the effect of providing documentation. Furthermore, we find that using a Reflexion framework significantly reduces errors: Claude-4.5-Sonnet, Gemini-2.5-Pro, and GPT-5 accuracies rise to around 60%. However, these results remain only marginally above random chance. Taken together, our findings identify significant challenges to be solved before AI agents can automate earth observation, and suggest paths forward.
Open-vocabulary Earth observation (EO) aims to localize geospatial concepts specified in natural language rather than a fixed label set. Existing benchmarks, however, usually cover narrow category vocabularies or limited query forms. To fill this gap, we introduce OVEarth-Bench, which extends existing evaluation in two directions: category breadth, through broad hierarchical category coverage with positive and negative expressions, and query diversity, through vocabulary, referring, and reasoning queries. The benchmark supports mask and box localization under a unified zero-shot protocol. We evaluate a broad set of general and EO-specific methods. The evaluation reveals that: (1) the performance of current methods remains limited, while broader category coverage yields more stable model rankings; (2) MLLM-based methods achieve the strongest overall performance; and (3) EO-specific methods generally underperform general models and rarely match the strongest methods. These findings provide guidance for future open-vocabulary EO method design and highlight the importance of developing more realistic, diverse, high-quality, and large-scale benchmarks for reliable evaluation. Our data and evaluation package are released at https://earth-insights.github.io/OVEarth-bench.