TerraMesh-Masks: Open-Vocabulary Segmentation for Earth Observation

Benedikt Blumenstiel; Hugues Devimeux; Johannes Jakubik; Konrad Schindler

cs.CVNov 5, 2026

TerraMesh-Masks: Open-Vocabulary Segmentation for Earth Observation

Authors: Benedikt Blumenstiel, Hugues Devimeux, Johannes Jakubik, Konrad Schindler

Organizations: ETH Zurich Zurich, Switzerland

Abstract

Open-vocabulary segmentation (OVS) enables models to segment a variety of concepts specified in natural language, removing the need for fixed label sets and fine-tuning. While recent OVS approaches have shown strong performance on natural images, they transfer poorly to Earth observation (EO) data due to differing semantics, scales, sensing modality, and geographic context. To date, large-scale EO-specific datasets and benchmarks for OVS remain scarce. We, therefore, introduce TerraMesh-Masks, a large-scale open-vocabulary segmentation dataset for EO, comprising 16 million binary masks aligned with nine million multimodal TerraMesh samples and annotated using OpenStreetMap and Overture-derived semantics. In addition, we provide an expert-reviewed evaluation set of 523 samples covering over 200 classes to enable robust evaluation. To demonstrate the impact of TerraMesh-Masks, we build an open-vocabulary version of TerraMind. Our results demonstrate significantly improved performance over general-purpose state-of-the-art models reaching 35.18% IoU versus 16.29% for DINOv3.txt and 11.33% for SAM-3. We release the dataset publicly under a permissive license at https://huggingface.co/datasets/ibm-esa-geospatial/TerraMesh-Masks.

Figures & tables

Figure 1: Average IoU pos results per group on TerraMesh-Masks-Eval. While other models struggle with most categories, TerraMind-OVS performs particularly well on tasks related to land cover. Building-related categories are generally challenging due to many small masks which are best identified by GeoPixel.

Dataset	Number of images	Vocabulary size	Modalities	Resolution	Image source	Annotation source	License
GeoPixelD [ 3 ]	18.7k	53k	RGB	$\sim$ 1m	iSAID	Source labels	CC-BY 4.0
MM-OVSeg [ 17 ]	25k	unknown	RGB, SAR	0.5–3 m	SpaceNet6; DFC2023	unknown	unknown
LandDiscover50K [ 15 ]	51k	40	RGB	high-res	9 datasets	Source labels	MIT
RemoteSAM-270K [ 4 ]	270k	1000	RGB	high-res	10 datasets	Source labels	CC-BY 4.0
EarthMind [ 18 ]	>500k	unknown	Optical, SAR	various	7+ datasets	Source labels	Apache 2.0
TerraMesh-Masks (ours)	9M	44k	S1, S2, RGB, DEM	10m	TerraMesh	Overture, OpenStreetMap	ODbL

Table 1: Overview of existing remote sensing datasets for OVS. Most datasets focus on high-resolution imagery. Some dataset licenses might not reflect source dataset restrictions.

Figure 2: TerraMesh-Masks samples drawn from the manually reviewed Eval set. Each patch is shown with a mask overlay and its open-vocabulary class label, country, and a synonym set.

Figure 3: Our dataset creation pipeline pairs TerraMesh samples with Overture and OpenStreetMap annotations. For each sample, we compute heuristics based on subtype, location, polygon size, and previous LLM quality scores. The sampled annotation is processed by Mistral-Large-3 which first scores the annotation quality and then generates the class label and synonyms. The annotations are further subsampled with an NDVI-based filtering for pretraining. We tasked experts to manually review a subset of the dataset to obtain a high-quality evaluation dataset.

Figure 4: Tokenizer architecture for discretizing binary segmentation masks into tokens. The tokens are later used as training targets of TerraMind-OVS, while the ViT decoder is used during inference.

Figure 5: Overview of our model architecture. Multimodal inputs are patchified and encoded with TerraMind. The text query is encoded using a frozen CLIP text encoder followed by an MLP. Both embeddings are used as context by the TerraMind decoder to predict the binary mask tokens. During inference, the predicted tokens are de-tokenized to generate binary segmentation masks.

Figure 6: Qualitative comparison on examples from TerraMesh-Masks-Eval. TerraMind performs well on larger land-cover related labels, while GeoPixel is better in spotting objects. Other models lean towards overprediction or no prediction.

Figure 7: Eval mIoU results per semantic group for TerraMind-OVS-B with different modality inputs. S2 and RGB perform compareable, while S1 and DEM significalty reduce the performance.

Model	Param.	Training	Input	IoU pos	mIoU	F1	Precision	Recall
SAM-3	841M	RGB	RGB	11.33 $\pm$ 2.30	10.97 $\pm$ 1.39	13.32 $\pm$ 2.54	12.86 $\pm$ 2.58	20.97 $\pm$ 3.33
RemoteSAM	242M	RGB	RGB	14.79 $\pm$ 2.13	30.42 $\pm$ 1.81	19.62 $\pm$ 2.48	20.60 $\pm$ 2.84	50.19 $\pm$ 3.49
DINOv3.txt	867M	RGB	RGB	16.29 $\pm$ 1.87	41.40 $\pm$ 1.42	23.14 $\pm$ 2.28	24.70 $\pm$ 2.72	51.53 $\pm$ 3.50
DINO Soars	869M	RGB	RGB	19.49 $\pm$ 2.55	44.29 $\pm$ 1.93	24.57 $\pm$ 2.83	23.65 $\pm$ 2.92	48.43 $\pm$ 3.98
SegEarth-OV	150M	RGB	RGB	21.59 $\pm$ 2.47	48.38 $\pm$ 1.53	28.01 $\pm$ 2.77	30.81 $\pm$ 3.10	45.34 $\pm$ 3.51
GeoPixel	11.3B	RGB	RGB	15.06 $\pm$ 1.99	49.25 $\pm$ 1.58	20.62 $\pm$ 2.41	40.27 $\pm$ 3.69	20.15 $\pm$ 2.57

Table 2: TerraMesh-Masks-Eval results with Student-t 95% confidence intervals, each metric averaged over the 523 per-sample results. Some TerraMind versions (TM-OVS) are trained on multimodal data that includes S1, S2, RGB, and DEM. Best results are in bold, second-best results are underlined.

Figure 8: These predictions from TerraMind-OVS-L (S2) show that the model does not have an understanding of clouds , snow , and other weather patterns as these are not included in the annotations. The model covers a range of concepts, but can still fail for related terms like dirt track or terminal .

Model	CaFFe (OOD)		reBEN-7k (Test split)		reBEN-7k (OOD split)
	mIoU (%)	F1 (%)	mIoU (%)	F1 (%)	mIoU (%)	F1 (%)
DINOv3.txt	15.32	22.64	3.52	5.20	3.07	4.60
SAM-3	1.00	1.24	7.30	1.00	8.76	1.03
TerraMind-OVS-S	14.57	22.82	10.93	18.40	11.26	17.59
TerraMind-OVS-S (S2)	14.35	22.29	11.46	19.25	14.06	21.23
TerraMind-OVS-B	9.10	14.14	13.75	22.23	16.54	24.32

Table 3: Open-vocabulary segmentation performance across two multi-label datasets: CaFFe [ 26 ] and reBEN-7k [ 25 , 27 ] . Thresholds for SAM-3 are optimized over the test set for maximum performance.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Modality examples in TerraMesh. We exclude NDVI and LULC from training and from the Eval subset as these modalities are related to specific semantic classes and might bias OVS models.

Panel of Figure 10 — Figure 10: Spatiotemporal distribution of the TerraMesh source dataset with (a) sample count visualized on a one degree map, and (b) monthly temporal distribution of its subsets using the S2 timestamps.

Figure 11: Geographic distribution of the 523 samples in TerraMesh-Masks-Eval, coloured by their source dataset.

Figure 12: Additional examples from the TerraMesh-Masks-Eval set. Each patch is shown with a polygon mask overlay and its open-vocabulary class label, country, and a synonym set.

Id	Group name	Sample	Unique	Examples
Id	Group name	count	labels	Examples
1	Inland water	63	14	reservoir, river, stream, canal, lake
2	Coastal water	45	12	coastal bay, ocean, bay, sea, coral reef
3	Coastal land	39	16	islet, coastal island, island, peninsula
4	Forest & wetland	30	9	forest, mangrove forest, wetland, mangrove
5	Grass & shrubland	12	8	scrubland, meadow, shrubland, hedge

Appendix

Table 4: Overview of the assigned label groups based on manual review and grouping of 200+ class labels in TerraMesh-Masks-Eval. We display the most common class names of each group.

Figure 13: Wrong or challenging annotations. The dataset contains several noisy labels, while water-related classes are often very precise.

Figure 14: Example of the provided information to the reviewers when creating the TerraMesh-Masks-Eval dataset. Based on this information, reviewers rated the image-mask-text combination as valid , invalid , or unsure .

Panel of Figure 15 — Figure 15: Label-wise mIoU results of all 207 labels in the eval set, sorted per model.

Figure 16: Qualitative comparison on examples from TerraMesh-Masks-Eval. SAM- and DINOv3-based models often lead towards overprediction or no prediction. TerraMind-OVS performs well on larger land-cover related labels, while GeoPixel is much better in spotting objects.

	Model	Inland water	Coastal water	Coastal land	Forest & wet.	Grass & shrub.	Agriculture	Barren terrain	Airport	Transport	Utilities	Industry	Settlements	Civic & inst.	Recreation	Mean
IoU pos	SAM-3	20.49	43.58	22.77	3.48	5.84	5.26	12.91	4.66	0.00	0.14	10.90	11.97	1.08	0.77	10.28
	RemoteSAM	6.24	19.98	27.29	32.51	28.95	17.89	36.96	13.92	5.04	4.51	9.71	13.76	2.48	9.50	16.34
	DINOv3.txt	11.22	42.05	23.41	26.12	8.32	17.03	21.80	12.58	4.57	5.09	19.76	26.55	7.90	4.96	16.53
	DINO Soars	11.79	58.89	21.89	39.29	16.10	13.99	42.61	16.97	1.52	5.90	20.56	26.47	3.54	2.96	20.18
	SegEarth-OV	16.82	57.85	23.32	39.14	18.62	18.89	46.84	10.99	3.89	5.76	18.36	33.13	7.20	5.87	21.91
	GeoPixel	9.60	19.65	19.51	20.01	7.26	16.59	13.76	19.13	16.56	4.47	25.32	20.26	12.46	11.26	15.42

Appendix

Table 5: IoU results on TerraMesh-Masks-Eval per group. All models are using RGB inputs, except for the last two rows in each block, which show TerraMind-OVS-L with S2 L2A inputs.

Figure 17: IoU pos against ground-truth mask area on TerraMesh-Masks-Eval for the EO-specific models, binned into log-spaced quintiles. Error bars show 95% confidence intervals.

Figure 18: Predictions on CaFFe with TerraMind-OVS-S.

Open-Vocabulary Semantic Segmentation

Open-vocabulary semantic segmentation labels image pixels using text-defined categories beyond a fixed training vocabulary.

See all 88 papers

OpenEarth-OVS44: A Multimodal Remote Sensing Dataset for Open-Vocabulary Segmentation
Oct 21, 2026Yimin Wei, Hongruixuan Chen, Hao Zheng +5Remote Sensing Image Segmentation Open-Vocabulary Semantic Segmentation
The University of Tokyo · RIKEN AIP · East China Normal University +1
OVEarth-Bench: Evaluating Category Breadth and Query Diversity for Open-Vocabulary Earth Observation
Jul 29, 2026Kaiyu Li, Zepeng Xin, Zixuan Jiang +4VLM Evaluation Open-Vocabulary Object Detection
Xi’an Jiaotong University
Perturbation-Regularized Open-Vocabulary Remote Sensing Segmentation with Unified Multi-Domain Evaluation
Apr 17, 2026Bingyu Li, Tao Huo, Haocheng Dong +4Image Segmentation Remote Sensing Image Segmentation
Department of Electronic Engineering and Information Science, University of Science and Technology of China, China · Institute of Artificial Intelligence (TeleAI), China Telecom, P. R. China · School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University, China

Oct 21, 2026J/K move · Enter open · S save

OpenEarth-OVS44: A Multimodal Remote Sensing Dataset for Open-Vocabulary Segmentation

Yimin Wei, Hongruixuan Chen, Hao Zheng, Hao Liu+4

The University of Tokyo · RIKEN AIP · East China Normal University+1

Remote Sensing Image Segmentation Open-Vocabulary Semantic Segmentation Remote Sensing Benchmarks Multimodal Semantic Segmentation Fine-Grained Semantic Segmentation

Open paper AskPDF arXiv

In remote sensing, open-vocabulary segmentation is particularly important for flexible and scalable land-cover understanding across diverse regions, as it improves generalization to novel categories without requiring exhaustive pixel-level annotations. Despite its great promise, progress in this area remains largely constrained by existing datasets with limited category coverage and overly coarse taxonomies, and therefore struggles to support fine-grained semantic understanding required in real-world applications such as urban functional zoning, carbon accounting, and post-disaster assessment. To address these challenges, we present OpenEarth-OVS44, a multimodal benchmark with 290,587 expert-verified segments from 2,832 RGB images and 1,341 strictly registered RGB-SAR pairs at sub-meter spatial resolution, covering 40 cities across France, Japan, and the United States under a unified 44-class taxonomy organized into 9 base categories. Three evaluation tracks are provided: an RGB-only fine-grained benchmark, a multimodal track with diverse synthetic cloudy conditions for weather-robust evaluation, and a held-out cross-region track. Extensive experiments on representative open-vocabulary segmentation models demonstrate that training on OpenEarth-OVS44 substantially improves fine-grained segmentation performance over the strongest training-free baseline, with improvements of up to 7.1 mIoU points on unseen categories and 27.6 mIoU points on seen categories. The dataset and code will be released.

Abstract

Open-Vocabulary Semantic Segmentation

Explore similar work

OpenEarth-OVS44: A Multimodal Remote Sensing Dataset for Open-Vocabulary Segmentation

OVEarth-Bench: Evaluating Category Breadth and Query Diversity for Open-Vocabulary Earth Observation

Perturbation-Regularized Open-Vocabulary Remote Sensing Segmentation with Unified Multi-Domain Evaluation