BigEarthNet.txt: A Large-Scale Multi-Sensor Image-Text Dataset and Benchmark for Earth Observation

Johann-Ludwig Herzog; Mathis Jürgen Adler; Leonard Hackel; Yan Shu; Angelos Zavras; Ioannis Papoutsis; Paolo Rota; Begüm Demir

cs.CVMar 31, 2026

BigEarthNet.txt: A Large-Scale Multi-Sensor Image-Text Dataset and Benchmark for Earth Observation

Authors: Johann-Ludwig Herzog, Mathis Jürgen Adler, Leonard Hackel, Yan Shu, Angelos Zavras, Ioannis Papoutsis, Paolo Rota, Begüm Demir

Organizations: BIFOLD and Technische Universität Berlin · University of Trento · National Technical University and National Observatory of Athens

Abstract

Vision-langugage models (VLMs) have shown strong performance in computer vision (CV), yet their performance on remote sensing (RS) data remains limited due to the lack of large-scale, multi-sensor RS image-text datasets with diverse textual annotations. Existing datasets predominantly include aerial Red-Green-Blue imagery, with short or weakly grounded captions, and provide limited diversity in annotation types. To address this limitation, we introduce BigEarthNet $.$ txt, a large-scale, multi-sensor image-text dataset designed to advance instruction-driven image-text learning in Earth observation across multiple tasks. BigEarthNet $.$ txt contains 464044 co-registered Sentinel-1 synthetic aperture radar and Sentinel-2 multispectral images with 9.6M text annotations, including: i) geographically anchored captions describing land-use/land-cover (LULC) classes, their spatial relations, and environmental context; ii) visual question answering pairs relevant for different tasks; and iii) referring expression detection instructions for bounding box prediction. Through a comparative statistical analysis, we demonstrate that BigEarthNet $.$ txt surpasses existing RS image-text datasets in textual richness and annotation type variety. We further establish a manually-verified benchmark split to evaluate VLMs in RS and CV. The results show the limitations of these models on tasks that involve complex LULC classes, whereas fine-tuning using BigEarthNet $.$ txt results in consistent performance gains across all considered tasks.

Figures & tables

Panel of Figure 1 — Figure 1: BigEarthNet.txt comprises $464\,044$ co-registered S1 and S2 images with diverse text annotations, resulting in a total of $\sim~$ 9.6$$ million S1 - S2 -text triplets. The dataset supports 15 tasks (Presence, Area, Counting, Adjacency, Relative Position, Country, Season, and Climate Zone, denoted as Pr, A, Cnt, Adj, RP, Loc, S, and Clt, respectively) across 4 broad categories.

Dataset

Multi-Sensor Images

Annotation Method

Supported Tasks

Geolocation data

#IT samples

Available

>3 bands per image

Co-registered

Manual

Template

Web scraped

LLM-based

Manual QC

Captioning

VQA

Ref. Exp. Detection

UCM-Captions [ 25 ]

✗

✓

✗

✓

✗

10.5k

Sydney-Captions [ 25 ]

✗

✓

✗

✓

✗

RSICD [ 20 ]

✗

✓

✗

✓

✗

54.6k

RSITMD [ 39 ]

✗

✓

✗

✓

✗

23.7k

NWPU-Captions [ 6 ]

✓

✗

✓

✗

✓

✗

157k

Table 1: Comparison of existing RS image-text datasets. Datasets are compared in terms of: 1) the presence of co-registered multi-sensor imagery exceeding three spectral bands; 2) methods used for text annotation, including the presence of an extensive manual quality check (QC); 3) supported tasks; 4) availability of geolocation data and 5) the number of image-text (IT) samples. (✓) indicates that only parts of the dataset fulfill this property.

Figure 2: Caption generation process for BigEarthNet.txt : i) Attribute extraction from the reference map and template filling based on the attributes and metadata such as location, season and climate zone information, followed by concatenation of the templates; ii) increasing linguistic variance and ensuring correctness via self-refinement.

Panel of Figure 3 — Figure 3: PDF of (a) caption lengths and (b) number of sentences in a caption in the BigEarthNet.txt dataset. (c) Comparison of existing image-text RS datasets in terms of size and semantic richness of the text data. The NLTK word tokenizer is used to divide the captions and questions into tokens. The color of each point indicates the total number of tokens in the dataset. Datasets that encompass images with more than three spectral bands are denoted with a star. The BigEarthNet.txt dataset is $1.7\times$ more diverse compared to the largest existing RS dataset encompassing more than three bands.

Model	# Parameters	Captioning	Binary VQA	MCQ	Ref. Exp. Detection
GC [ 15 ]	7B	$0.75$	$50.82$	$28.36$	$4.85$
LHRS [ 24 ]	7B	$0.43$	$48.23$	$22.99$	$4.79$
SE [ 42 ]	7B	$0.00$	$48.87$	$27.60$	$6.08$
ED ${}_{\text{rgb}}$ [ 30 ]	4B	$0.62$	$58.38$	$32.94$	$7.13$
ED ${}_{\text{s2}}$ [ 30 ]	4B	$0.00$	$44.06$	$8.43$	$0.49$
EM ${}_{\text{rgb}}$ [ 27 ]	4B	1.66	$57.90$	$34.25$	$12.12$

Table 2: Results of VLM s in RS and CV on the main categories of the BigEarthNet.txt benchmark split. Reported metrics for captioning: BLEU-4, binary VQA : accuracy, MCQ : accuracy, and referring expression detection: mIoU . All results in percent (%). *: Reported parameter count.

Model	Pr	A	Cnt	Adj	OA	IF
GC [ 15 ]	$51.65$	$46.86$	$43.57$	$55.51$	$50.82$	✓
LHRS [ 24 ]	$48.83$	$51.23$	$49.23$	$46.10$	$48.23$	✗
SE [ 42 ]	$52.19$	$51.61$	$46.63$	$46.94$	$48.87$	✓
ED ${}_{\text{rgb}}$ [ 30 ]	$64.47$	$53.06$	$51.99$	60.62	$58.38$	✗
ED ${}_{\text{s2}}$ [ 30 ]	$37.45$	$40.20$	$47.17$	$47.78$	$44.06$	✗
EM ${}_{\text{rgb}}$ [ 27 ]	69.34	$56.20$	$51.07$	$55.97$	$57.90$	✓

Table 3: Results on the binary VQA tasks for VLM s in RS and CV : Presence, Area, Counting, Adjacency, and Overall, denoted as Pr, A, Cnt, Adj, and OA, in terms of accuracy. All results are in percent (%). IF shows whether an unambiguous answer could be extracted all the time.

Model	Pr	A	Cnt	Adj	RP	Loc	S	Clt	OA	IF
GC [ 15 ]	$32.58$	$26.80$	$24.72$	$29.75$	$25.46$	$34.62$	$22.57$	$30.52$	$28.23$	✗
LHRS [ 24 ]	$23.84$	$16.54$	$21.19$	$27.23$	$17.48$	$28.53$	$23.71$	$23.56$	$22.99$	✗
SE [ 42 ]	$30.89$	$21.75$	$24.91$	$30.51$	$22.09$	$29.81$	$24.71$	$33.63$	$27.60$	✓
ED ${}_{\text{rgb}}$ [ 30 ]	46.40	$27.41$	36.43	$30.66$	$29.14$	54.17	$25.57$	$27.26$	$32.94$	✗
ED ${}_{\text{s2}}$ [ 30 ]	$21.16$	$5.51$	$4.65$	$8.62$	$0.00$	$15.06$	$6.86$	$7.26$	$8.43$	✗
EM ${}_{\text{rgb}}$ [ 27 ]	$41.75$	$24.81$	$27.32$	$42.87$	$29.91$	$40.06$	$23.43$	$37.04$	$34.25$	✗

Table 4: Results on the MCQ tasks for VLM s in RS and CV : Presence, Area, Counting, Adjacency, Relative Position, Country, Season, Climate Zone, and Overall, denoted as Pr, A, Cnt, Adj, RP, Loc, S, Clt, and OA, respectively in terms of accuracy. All results are in percent (%). IF shows whether an unambiguous answer could be extracted all the time.

Model	mIoU	Acc@25	Acc@50	Acc@75	Acc@90	IF
GC [ 15 ]	$7.51$	$11.34$	$1.51$	$0.00$	$0.00$	✓
LHRS [ 24 ]	$7.47$	$12.59$	$1.89$	$0.50$	$0.00$	✗
SE [ 42 ]	$12.12$	$20.03$	$6.55$	$2.27$	$0.13$	✓
ED ${}_{\text{rgb}}$ [ 30 ]	$8.99$	$12.85$	$2.02$	$0.00$	$0.00$	✓
ED ${}_{\text{s2}}$ [ 30 ]	$0.98$	$0.63$	$0.00$	$0.00$	$0.00$	✗
EM ${}_{\text{rgb}}$ [ 27 ]	$21.90$	$35.52$	17.88	5.67	1.39	✗

Table 5: Results on referring LULC detection for VLM s in RS and CV in terms of mIoU , and accuracy@

k

(where a prediction is counted as correct if the overlap with the reference is greater than

k\%

). All results are in percent (%). IF shows whether an unambiguous answer could be extracted all the time.

Model	mIoU	Acc@25	Acc@50	Acc@75	Acc@90	IF
GC [ 15 ]	$2.16$	$3.43$	$0.63$	$0.13$	$0.00$	✗
LHRS [ 24 ]	$3.09$	$0.25$	$0.00$	$0.00$	$0.00$	✗
SE [ 42 ]	$0.00$	$0.00$	$0.00$	$0.00$	$0.00$	✗
ED ${}_{\text{rgb}}$ [ 30 ]	$5.26$	$3.05$	$0.13$	$0.00$	$0.00$	✓
ED ${}_{\text{s2}}$ [ 30 ]	$0.00$	$0.00$	$0.00$	$0.00$	$0.00$	✗
EM ${}_{\text{rgb}}$ [ 27 ]	$9.35$	$14.21$	$4.57$	$1.27$	$0.13$	✗

Table 6: Results on referring point detection for VLM s in RS and CV in terms of mIoU , and accuracy@

k

(where a prediction is counted as correct if the overlap with the reference is greater than

k\%

). IF shows whether an unambiguous answer could be extracted all the time.

Model	BLEU-4	ROUGE	METEOR	CIDEr	BERTScore	SBERT- Cosine	CLAIR
GC [ 15 ]	$0.75$	$14.95$	$14.24$	$0.65$	$83.42$	$43.05$	$19.91$
LHRS [ 24 ]	$0.72$	$12.44$	$16.93$	$0.01$	$82.52$	54.61	$28.62$
SE [ 42 ]	$0.00$	$5.91$	$2.21$	$0.00$	$82.79$	$38.30$	$43.86$
ED ${}_{\text{rgb}}$ [ 30 ]	$0.48$	$13.65$	$11.36$	$0.51$	$83.45$	$50.49$	$46.51$
ED ${}_{\text{s2}}$ [ 30 ]	$0.03$	$8.64$	$3.51$	$0.00$	$83.34$	$35.00$	59.70
EM ${}_{\text{rgb}}$ [ 27 ]	1.66	16.54	$14.93$	0.96	84.34	$50.55$	$56.73$

Table 7: Results on the captioning task for VLM s in RS and CV in terms of n-gram-based (BLEU-4, ROUGE, METEOR, CIDEr), sentence-embedding-based (BERTScore, SBert-Cosine) and LLM -based (CLAIR) metrics. All results are in percent (%).

Model	Captioning	Binary VQA	MCQ	Ref. Exp. Detection
SOTA RS	$1.66$	$58.38$	$35.26$	$16.18$
SOTA CV	$0.96$	$61.96$	$37.55$	$31.73$
RS-InternVL	34.04	73.29	51.49	65.84

Table 8: Results on the main tasks of the BigEarthNet.txt benchmark split for the fine-tuned adapted RS-InternVL model as well as best results from VLM s in RS and CV , which are EM

{}_{\text{rgb}}

[ 27 ] and LLaVa [ 16 ] for captioning, ED

{}_{\text{rgb}}

[ 30 ] and Qwen [ 1 ] for binary VQA , EM

{}_{\text{s1s2}}

[ 27 ] and Qwen [ 1 ] for MCQ , and EM

{}_{\text{s1s2}}

[ 27 ] and GPT [ 29 ] for referring expression detection. Reported metric for captioning is BLEU-4, while that for binary VQA , and MCQ is accuracy, and that for referring expression detection is mIoU. All results are in percent (%).

Remote Sensing

Remote sensing uses satellite, aerial, and ground-based observations to measure and map Earth and other planetary surfaces.

See all 513 papers

OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Tasks
Mar 10, 2026Ronghao Fu, Haoran Liu, Weijie Zhang +4VLM Evaluation Remote Sensing Benchmarks
Jilin University, Changchun, China · Chang Guang Satellite Technology, Changchun, China
Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Tasks
Jun 9, 2026Miaoxin Cai, Guanqun Wang, Wei Zhang +6Remote Sensing Image Understanding Multimodal Grounding
National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing (SBIIP), Beijing Institute of Technology, Beijing 100081, China · Advanced Research Institute of Multidisciplinary Sciences, Beijing Institute of Technology, Beijing 100081, China · School of Mechatronical Engineering, Beijing Institute of Technology, Beijing 100081, China +5
More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe
Jul 17, 2026Stefan Maria Ailuro, Mario Markov, Mohammad Mahdi +2Remote Sensing Image Understanding Remote Sensing VQA
INSAIT, Sofia University "St. Kliment Ohridski"

Mar 10, 2026J/K move · Enter open · S save

OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Tasks

Ronghao Fu, Haoran Liu, Weijie Zhang, Zhiwen Lin+3

Jilin University, Changchun, China · Chang Guang Satellite Technology, Changchun, China

VLM Evaluation Remote Sensing Benchmarks Remote Sensing VQA Remote Sensing Visual Grounding

Open paper AskPDF arXiv

Vision-Language Models (VLMs) have demonstrated effective perception and reasoning capabilities on general-domain tasks, leading to growing interest in their application to Earth observation. However, a systematic benchmark for comprehensively evaluating remote sensing vision-language models (RSVLMs) remains lacking. To address this gap, we introduce OmniEarth, a benchmark for evaluating RSVLMs under realistic Earth observation scenarios. OmniEarth organizes tasks along three capability dimensions: perception, reasoning, and robustness. It defines 28 fine-grained tasks covering multi-source sensing data and diverse geospatial contexts. The benchmark supports two task formulations: multiple-choice VQA and open-ended VQA. The latter includes pure text outputs for captioning tasks, bounding box outputs for visual grounding tasks, and mask outputs for segmentation tasks. To reduce linguistic bias and examine whether model predictions rely on visual evidence, OmniEarth adopts a blind test protocol and a quintuple semantic consistency requirement. OmniEarth includes 9,275 carefully quality-controlled images, including proprietary satellite imagery from Jilin-1 (JL-1), along with 44,210 manually verified instructions. We conduct a systematic evaluation of contrastive learning-based models, general closed-source and open-source VLMs, as well as RSVLMs. Results show that existing VLMs still struggle with geospatially complex tasks, revealing clear gaps that need to be addressed for remote sensing applications. OmniEarth is publicly available at https://huggingface.co/datasets/sjeeudd/OmniEarth.

Abstract

Remote Sensing

Explore similar work

OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Tasks

Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Tasks

More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe