EVE: A Domain-Specific LLM Framework for Earth Intelligence

Àlex R. Atrio; Antonio Lopez; Jino Rohit; Yassine El Ouahidi; Marcello Politi; Vijayasri Iyer; Umar Jamil; Sébastien Bratières

cs.CLMar 20, 2026

EVE: A Domain-Specific LLM Framework for Earth Intelligence

Authors: Àlex R. Atrio, Antonio Lopez, Jino Rohit, Yassine El Ouahidi, Marcello Politi, Vijayasri Iyer, Umar Jamil, Sébastien Bratières,

Organizations: Pi School · Mistral AI · Translated · ESA Φ-lab

Abstract

We introduce Earth Virtual Expert (EVE), the first open-source, end-to-end initiative for developing and deploying domain-specialized LLMs for Earth Intelligence. At its core is EVE-Instruct, a domain-adapted 24B model built on Mistral Small 3.2 and optimized for reasoning and question answering. On newly constructed Earth Observation and Earth Sciences benchmarks, it outperforms comparable models while preserving general capabilities. We release curated training corpora and the first systematic domain-specific evaluation benchmarks, covering MCQA, open-ended QA, and factuality. EVE further integrates RAG and a hallucination-detection pipeline into a production system deployed via API and GUI, supporting 350 pilot users so far. All models, datasets, and code are ready to be released under open licenses as contributions to our field at huggingface.co/eve-esa and github.com/eve-esa.

Figures & tables

Figure 1: System architecture of EVE depicting component interactions.

Task Type	Samples
MCQA (Multiple Answer)	431
MCQA (Single Answer)	1261
Hallucination Detection	2326
Open-Ended QA (No context)	1257
Open-Ended QA (with Context)	418

Table 1: EO and Earth Sciences benchmark suite. Multiple Choice Question Answering (MCQA) differ in the number of correct options. Hallucination detection is a balanced binary QA classification task. Open-ended QA evaluate self-contained and context-grounded questions.

Task Type	Example
MCQA (Multiple Answer)	Q: Which statements are true about surging glaciers? A) Surging glaciers are subject to cyclical flow instabilities B) Surging glaciers flow at a constant rate down a mountain C) […] a glacier’s velocity moves up to 100 times faster than normal D) Surging glaciers suddenly increase in mass $\rightarrow$ Answers: A, C
MCQA (Single Answer)	Q: What is the main obstacle to using optical EO in the aftermath of a severe weather event? A) Sensor calibration issues B) Atmospheric interference C) Low light conditions D) Cloud cover $\rightarrow$ Answer: D
Hallucination Detection	Q: What advantages do radar systems offer for environmental monitoring compared to optical sensors? Hallucinated Answer: Radars are active sensors that create their own electromagnetic signals. Radars cannot operate at night and are ineffective in hazy or smoky environments. SAR […] uses the movement between the antenna and the target area […] to create high-resolution images for remote sensing. Correct Answer: Radars are active sensors […] Radars can operate day and night, and penetrate clouds, haze, and smoke. SAR […] $\rightarrow$ Label: Hallucinated ( underlined span)
Open-Ended QA	Q: What is the calving front of an ice stream? A: The point where icebergs are lost from a glacier.
Open-Ended QA w/ Context	Q: How does LiDAR map buildings? [3 source documents provided as context] A: LiDAR maps buildings by sending laser pulses that reflect off rooftops, walls, and other structural surfaces, recording precise 3D coordinates. These points are stored in point clouds […]

Table 2: Representative examples from each evaluation category in the EO and Earth Sciences benchmark suite (see Table 1 ). In hallucination detection, underlined spans are the annotated hallucinated content.

Long-form text: 30% (10B)		Instruction-formatted data: 70% (23.5B)
Long-Form Replay	50%	Instruct-formatted Replay	60%
EO and Earth Sciences	50%	EO and Earth Sciences	40%
Raw	2%	ContextQA (Best)	12%
Best Chunks	14%	ContextQA (Good)	21%
Active Reading (Agnostic)	28%	SelfQA	2.6%
Active Reading (Specific)	6%	MultiHop QA	2.1%

Table 3: Distribution of training data across long-form and instruction-formatted data.

Model	Size (B)	IoU	Acc.	Acc.	F1	Judge	EVE WR	Judge	EVE WR	Rank $\downarrow$
		MCQA Mult.		MCQA Sing.	Hallucin.	Open-Ended		Open-Ended w/ Context
Llama4 Scout	109-A17	80.32	71.23	91.67	66.08	87.37	53.95	71.73	58.31	3.67
Qwen3	30-A3	78.40	66.36	93.02	81.30	94.92	50.70	81.81	52.12	2.67
Gemma3	27	73.60	57.54	87.31	75.07	94.41	50.92	78.31	51.58	3.83
Mistral Small 3.2	24	80.19	70.30	83.51	82.19	91.78	51.69	71.93	57.27	3.50
EVE-Instruct	24	86.12	77.73	96.35	84.70	96.40	—	78.28	—	1.33

Table 4: Model performance across EO and Earth Sciences benchmark tasks presented in Table 1 (0-shot). EVE WR (win rate) indicates percentage of pairwise comparisons where EVE-Instruct is preferred over the comparison model (

>50\%

means EVE is preferred). Rank

\downarrow

(lower is better) reports the average per-metric rank across MCQA multiple (IoU and Accuracy), MCQA single (Accuracy), Hallucination (F1), Open-ended QA (Judge), and Open-Ended QA with Context (Judge).

Category	Small 3.2	EVE-Instruct	$\Delta$
Math & Reasoning	50.8	54.9	+4.1
Coding	55.6	56.5	+0.9
Knowledge	67.7	69.0	+1.3
Tool Calling	87.9	90.9	+3.0
Instruction Following	80.1	81.2	+1.1
Chat Quality	90.8	91.7	+0.9

Table 5: General-domain performance after domain adaptation (category-level averages over several standard benchmarks, 0–100 scale).

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Released Tokens	%	Licence
MDPI	1.3 B	46.6	CC-BY
Copernicus	723 M	25.9	CC-BY
NCBI	485 M	17.4	CC-BY
ISPRS	74.1 M	2.7	CC-BY
Wiley	71.6 M	2.6	CC-BY
Elsevier	43.7 M	1.6	CC-BY

Appendix

Table 6: Token distribution across data sources in the released subset of the open-access EO and Earth Sciences corpus (2.8B tokens released out of 4.2B total open-access tokens).

Tool	Text	Formulas	Tables	Avg.
Markitdown	0.81	0.00	0.00	0.27
Docling	0.79	0.00	0.40	0.40
Pymupdf4llm	0.81	0.00	0.26	0.36
Pypdf2	0.84	0.00	0.00	0.28
Marker	0.80	0.31	0.42	0.51
Unstructured	0.79	0.00	0.00	0.26

Appendix

Table 7: OCR benchmark results (Normalized Levenshtein Similarity) across tools, evaluating text, formula, and table extraction. The Avg. column reports the mean across the three categories.

Benchmark	Small 3.2	EVE	$\Delta$
aime25@16	26.7	35.2	+8.5
aime24@16	37.1	41.2	+4.1
math	88.6	88.4	-0.2
Average (Math & Reasoning)	50.8	54.9	+4.1
livecodebench_p@1	36.4	39.1	+2.7
mbpp_p@1	74.7	73.9	-0.8

Appendix

Table 8: Evaluation results (0–100 scale) comparing Mistral Small 3.2 and EVE-Instruct across general-domain benchmarks. Category averages are shown for each task group. Tool Calling, Instruction Following, and Chat Quality correspond to private internal evaluation.

Model	Size (B)	IoU	Acc.	Acc.	F1	Judge	EVE WR	Judge	EVE WR	Rank $\downarrow$
		MCQA Mult.		MCQA Sing.	Hallucin.	Open-Ended		Open-Ended w/ Context
GPT-4.1	1800 *	87.56	78.19	94.37	81.58	96.48	49.77	86.65	49.22	2.83
Qwen3	235-A22	87.40	80.97	95.16	84.40	97.05	48.24	86.10	50.09	2.17
MiniMax m2.5	230A10	84.82	77.72	94.95	83.77	91.00	51.10	81.57	51.20	5.17
GPT OSS	120A5	84.56	76.79	89.77	89.92	94.20	50.30	84.80	50.70	4.83
Mistral Medium 3.1	200 *	85.44	76.33	95.00	76.89	96.45	50.20	86.44	50.99	4.17

Appendix

Table 9: Extension of Table 4 to larger-scale models under the same evaluation setup. Rank

\downarrow

Model	Size (B)	Score	EVE WR	Score	EVE WR	Score	EVE WR	Score	EVE WR	Rank ${}_{\text{prev}}$ $\downarrow$	Rank $\downarrow$	$\Delta$ Rank $\downarrow$
		Open-Ended				Open-Ended w/ Context
		Claude Judge		Gemini Judge		Claude Judge		Gemini Judge
Llama Scout 4	109-A17	69.22	52.7	78.09	54.9	55.64	59.83	62.44	59.45	3.67	3.50	$-$ 0.17
Qwen3	30-A3	82.75	51.2	88.89	51.2	69.13	53.71	64.59	53.23	2.67	2.75	$+$ 0.08
Gemma 3	27	80.82	51.6	87.55	51.4	62.48	52.75	63.49	51.91	3.83	3.88	$+$ 0.05
Mistral 3.2 Small	24	73.76	51.0	84.04	52.5	57.21	57.19	60.28	56.69	3.50	3.75	$+$ 0.25

Appendix

Table 10: Comparison of models on Open-Ended and Open-Ended with Context tasks using two judges (Claude and Gemini - anthropic/claude-sonnet-4.6 and google/gemini-2.5-flash). Rank

{}_{\text{prev}}

refers to the single-judge baseline ranking from Table 4 . Rank

\downarrow

(lower is better) is the average per-metric rank across the four judge score columns.

\Delta

Rank denotes the difference between the two rankings.

		TelQNA	Satcom Open-Ended
Model	Size (B)	Acc.	Judge
Llama4 Scout	109-A17	74.57	75.00
Qwen3	30-A3	73.81	83.00
Gemma3	27	70.84	81.25
Mistral Small 3.2	24	72.63	76.50
EVE-Instruct	24	75.72	81.25

Appendix

Table 11: Model performance on TelQNA and Satcom Open-Ended benchmarks (0-shot).

Chunker	Chunk Size	Embedding	Doc recall	Passage recall	IoU	Precision	Recall
Two-pass	512	INDUS 512	91.60	57.70	2.77	2.78	77.00
Two-pass	512	INDUS 512*	91.60	50.80	2.64	2.65	73.20
Two-pass	512	INDUS 1024	83.24	38.60	2.34	2.35	59.90
Two-pass	512	INDUS 1024*	82.60	38.50	2.32	2.33	59.70
Two-pass	512	Qwen3-Embedding-4B	95.70	65.60	2.93	2.94	85.30
Two-pass	512	Qwen3-Embedding-4B*	95.60	63.60	2.94	2.94	85.00

Appendix

Table 12: Performance comparison of different embedding models across chunking strategies and chunk sizes. All metrics are expressed as percentages with two decimal precision. Precision and recall are per token, considering 10 as the number of passages retrieved. * indicates quantized embeddings. The Two-pass chunker refers to the approach presented in Section 8 . The Recursive chunker is based on LangChain’s RecursiveCharacterTextSplitter .

Collection	Chunk Size	Reranker	Ref Retrieved Ratio @10		MRR @10
			Retrieval	Reranked	Retrieval	Reranked
INDUS 512*	512	INDUS	51.00	54.10	45.90	55.70
INDUS 512*	512	Qwen3-4B	51.00	54.70	45.90	61.70
Qwen3-Embedding-4B*	512	INDUS	63.30	62.80	62.60	60.10
Qwen3-Embedding-4B*	512	Qwen3-4B	63.30	65.10	62.60	69.30
Qwen3-Embedding-4B*	1024	Qwen3-4B	68.60	71.60	65.40	73.40

Appendix

Table 13: Performance comparison at top-10 with K=20 retrievals. Ref Retrieved Ratio @10 measures the percentage of queries for which at least one relevant chunk appears in the top-10 reranked results. MRR @10 (Mean Reciprocal Rank) is the average of the reciprocal rank of the first relevant chunk across queries, rewarding systems that place relevant chunks higher in the ranked list. All metrics are expressed as percentages with two decimal precision. * indicates quantized embeddings.

Figure 2: End-to-end architecture of the deployed EVE system.

Panel of Figure 3 — Figure 3: LLM-as-a-judge evaluation prompt for Open-Ended with retrieval context.

Panel of Figure 4 — Figure 4: Prompt used to filter corpus chunks before Active Reading synthesis. Only Best -rated chunks enter the Active Reading pipeline.

Panel of Figure 5 — Figure 5: Prompt used for task-specific Active Reading. The model first generates questions from the source chunk, then devises two active learning strategies per question to guide synthesis.

Panel of Figure 6 — Figure 6: Prompt used for predefined Active Reading. The model applies strict eligibility rules to select at most 2 strategies from a fixed predefined set.

Panel of Figure 7 — Figure 7: Prompt used to generate synthetic training documents by applying a selected Active Reading strategy to a source chunk. The model is explicitly instructed to output only the resulting document.

Panel of Figure 8 — Figure 8: Prompt used to generate SelfQA samples. Context-grounded QA pairs are reformulated into self-contained questions that can be answered from the model’s parametric knowledge alone.

Panel of Figure 9 — Figure 9: Prompt used to filter ContextQA samples. A two-step assessment first applies hard filters, then evaluates quality on a five-point scale. Best and Good samples are retained for training as described in Table 3 .

Earth Science

See all 23 papers

Earth-Agent-Pro: Towards Real-World Full-Chain Earth Observation with Agents
Sep 11, 2026Zhutao Lv, Chenhao Dang, Yi Feng +5Remote Sensing AI Agent Benchmarks
Tsinghua Shenzhen International Graduate School, Tsinghua University · Sun Yat-Sen University · Shanghai Jiao Tong University +3
Towards LLM Agents for Earth Observation
Apr 16, 2025Chia Hsiang Kao, Wenting Zhao, Cheryl Lam +10Code Generation LLM Agent Evaluation
Cornell University · Columbia University
OVEarth-Bench: Evaluating Category Breadth and Query Diversity for Open-Vocabulary Earth Observation
Jul 29, 2026Kaiyu Li, Zepeng Xin, Zixuan Jiang +4VLM Evaluation Open-Vocabulary Object Detection
Xi’an Jiaotong University

Sep 11, 2026J/K move · Enter open · S save

Earth-Agent-Pro: Towards Real-World Full-Chain Earth Observation with Agents

Zhutao Lv, Chenhao Dang, Yi Feng, Yanpei Gong+4

Tsinghua Shenzhen International Graduate School, Tsinghua University · Sun Yat-Sen University · Shanghai Jiao Tong University+3

Remote Sensing AI Agent Benchmarks Tool-Using Agents LLM Agent Planning

Open paper AskPDF arXiv

Real-world Earth observation (EO) agents must translate high-level scientific questions into executable workflows to acquire observations, prepare data, perform domain computations, and derive conclusions from runtime evidence. Existing EO agents typically start from supplied observations, while benchmarks typically provide prepared inputs or candidate answers, leaving full-chain open-world EO execution largely untested. We present Earth-Agent-Pro, an execution-adaptive Plan-and-Execute framework using expert-authored skills to constrain planning and runtime tool use. Workflow-centered structured memory records planned steps, accepted evidence, and their dependencies, enabling repair of only the affected workflow suffix when runtime evidence invalidates a step. Separate large language model adapters use sequence-level supervised fine-tuning for planner workflow composition and node-level group relative policy optimization with locally verifiable rewards for executor tool-argument grounding. Earth-Bench-Pro instantiates 248 expert-curated task cores as 744 questions under three matched regimes. Its 248 Open-World Execution questions span RGB imagery, spectral observations, and remote sensing products, pairing high-level requests with runtime data requirements, executable trajectories, and open-ended answers grounded in execution evidence. With a shared GPT-5 backbone, Earth-Agent-Pro achieves 66.13% LLM-as-Judge accuracy, exceeding ReAct by 20.95 points in this metric and 24.44 points in Tools-In-Order. Joint adapter tuning raises Qwen3.5-9B LLM-as-Judge accuracy from 38.31% to 50.00%, an 11.69-point gain over the untuned configuration. Planning-only evaluation and execution with the reference workflow show that the adapters improve workflow composition and argument grounding, respectively. Code and datasets will be released soon.

Abstract

Earth Science

Explore similar work

Earth-Agent-Pro: Towards Real-World Full-Chain Earth Observation with Agents

Towards LLM Agents for Earth Observation

OVEarth-Bench: Evaluating Category Breadth and Query Diversity for Open-Vocabulary Earth Observation