Enabling Training-Free Text-Based Remote Sensing Segmentation

Jose Sosa; Danila Rukhovich; Anis Kacem; Djamila Aouada

cs.CVFeb 19, 2026

Enabling Training-Free Text-Based Remote Sensing Segmentation

Authors: Jose Sosa, Danila Rukhovich, Anis Kacem, Djamila Aouada

Organizations: SnT, University of Luxembourg

Abstract

Recent advances in Vision Language Models (VLMs) and Vision Foundation Models (VFMs) have opened new opportunities for zero-shot text-guided segmentation of remote sensing imagery. However, most existing approaches still rely on additional trainable components, limiting their generalisation and practical applicability. In this work, we investigate to what extent text-based remote sensing segmentation can be achieved without additional training, by relying solely on existing foundation models. We propose a simple yet effective approach that integrates contrastive and generative VLMs with the Segment Anything Model (SAM), enabling a fully training-free or lightweight LoRA-tuned pipeline. Our contrastive approach employs CLIP as mask selector for SAM's grid-based proposals, achieving state-of-the-art open-vocabulary semantic segmentation (OVSS) in a completely zero-shot setting. In parallel, our generative approach enables reasoning and referring segmentation by generating click prompts for SAM using GPT-5 in a zero-shot setting and a LoRA-tuned Qwen-VL model, with the latter yielding the best results. Extensive experiments across 19 remote sensing benchmarks, including open-vocabulary, referring, and reasoning-based tasks, demonstrate the strong capabilities of our approach. Code will be released at https://github.com/josesosajs/trainfree-rs-segmentation.

Figures & tables

Figure 2: Inference schemes of our segmentation approaches with (a) contrastive and (b) generative VLMs.

	Method	OEM	LoveDA	iSAID	Potsdam	Vaihingen	UAVid	UDD5	VDD	Avg.
Trained on remote sensing data
	SegEarth-OV [ 33 ]	40.3	36.9	21.7	48.5	40.0	42.5	50.6	45.3	39.2
	Oracle	64.4	50.0	36.2	74.3	61.2	59.7	56.5	62.9	58.2
Zero-shot methods
	CLIP [ 50 ]	12.0	12.4	7.5	15.6	10.8	10.9	9.5	14.2	11.4
	MaskCLIP [ 87 ]	25.1	27.8	14.5	33.9	29.9	28.6	32.4	32.9	27.2

Table 1: Results of our contrastive VLM-based approach for text-based remote sensing segmentation on OVSS task. We evaluate 8 remote sensing multi-class datasets. Avg. is for average across all datasets. Oracle represents the upper bound, achieved by a fully supervised model [ 74 ] .

	Method	Building extraction				Road Extraction				Flood Detection	Avg.
	Method	WHU-A	WHU-S	Inria	xBD-pre	CHN6	DG	MA	SpaceNet	WBS-SI	Avg.
Trained on remote sensing data
	SegEarth-OV [ 33 ]	49.2	28.4	44.6	37.0	35.4	17.8	11.5	23.8	60.2	34.2
Zero-shot methods
	CLIP [ 50 ]	17.7	3.5	19.6	16.0	7.7	3.9	4.9	7.1	18.6	11.0
	MaskCLIP [ 87 ]	29.8	14.0	33.4	29.2	28.1	13.2	10.6	20.8	39.8	24.3

Table 2: Results of our contrastive VLM-based approach for text-based remote sensing segmentation on OVSS task. We evaluate 9 remote sensing single-class datasets across building extraction, road extraction, and flood detection. Avg. is for average across all datasets.

Panel of Figure 3 — Figure 3: Qualitative results of the training-free contrastive VLM pipeline on multi-class (first and second rows) and single-class (third row) OVSS tasks using remote sensing datasets.

Panel of Figure 4 — Figure 4: Qualitative results of the LoRA-tuned generative VLM pipeline on reasoning (first and second rows) and referring (third row) tasks using remote sensing datasets.

	Method	LLM	Trained on RS data			RRSIS-D		EarthReason
	Method	LLM	LLM	Mask Decoder	Extra	Val	Test	Val	Test
Zero-shot methods
	GPT-Image-1	GPT-5	✗	✗	✗	20.1	17.2	38.4	41.0
	Ours	GPT-5	✗	✗	✗	25.8	24.9	46.0	47.4
Classical methods
	RRSIS [ 80 ]	BERT-base	✓	✓	✓	60.2	59.4	–	–

Table 3: Results of our generative VLM-based approach for text-based remote sensing segmentation on reasoning and referring tasks. We evaluate on test and validation sets from RRSIS-D [ 40 ] (referring) and EarthReason [ 34 ] (reasoning) datasets.

SAM	# clicks	OEM	LoveDA	UAVid	CHN6
SAM-Tiny	29 $\times$ 29	33.9	37.7	43.8	35.2
SAM-Base	29 $\times$ 29	34.2	38.1	44.2	36.2
SAM-Large	10 $\times$ 10	29.7	36.2	40.8	31.0
SAM-Large	20 $\times$ 20	33.1	38.1	44.1	35.6
SAM-Large	29 $\times$ 29	34.2	38.2	44.3	36.4

Table 4: Ablation of SAM scale and grid density on contrastive VLM-based approach.

Method	# clicks	EarthReason		RRSIS-D
Method	# clicks	Val	Test	Val	Test
Qwen2.5-VL-7B	6	67.8	69.3	61.4	60.9
Qwen3-VL-4B	6	70.4	71.7	67.3	67.2
Qwen3-VL-2B	2	63.8	64.9	61.1	61.0
Qwen3-VL-2B	4	67.9	69.8	66.4	66.4
Qwen3-VL-2B	6	70.6	72.7	68.1	67.6

Table 5: Ablation of generative VLM scale and click configuration.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Visualisations from click generation procedure. Masks (red) are produced by SAM prompted by clicks (green). Reported IoU is compared to groung truth mask.

	Method	Building extraction				Road Extraction				Flood Detection	Avg.
	Method	WHU-A	WHU-S	Inria	xBD-pre	CHN6	DG	MA	SpaceNet	WBS-SI	Avg.
Trained on remote sensing data
	SegEarth-OV [ 33 ]	49.9	–	48.9	43.1	32.8	20.1	17.2	29.1	57.9	37.4
Zero-shot methods
	Ours	58.7	–	53.8	39.1	33.5	15.0	20.3	29.5	56.3	38.3

Appendix

Table 6: Additional results of our contrastive VLM-based approach for text-based remote sensing segmentation on the OVSS task using images of size

896\times 896

. Avg. denotes the average across all datasets. Best results are highlighted in bold .

Contrastive VLM; LoveDA test set
	Method	VLM	Decoder	Time↓	IoU↑
	SegEarth-OV [ 33 ]	0.15B	SimFeatUp $\star$	4.1	36.9
	Ours	0.15B	SAM (10 $\times$ 10)	3.6	36.2
	Ours	0.15B	SAM (29 $\times$ 29)	14.0	38.2
Generative VLM; EarthReason test set
	SegEarth-R1 [ 34 ]	1.3B	Mask2Former $\star$	0.44	70.7

Appendix

Table 7: Inference time comparison.

\star

indicates that the component is trained on remote sensing data.

Panel of Figure 6 — Figure 6: Qualitative comparison with baseline SegEarth-OV on OVSS datasets.

Panel of Figure 7 — Figure 7: Top row: input image and corresponding prompts for the different generative VLM settings. Bottom row: outputs produced by each model.

Panel of Figure 8 — Figure 8: Qualitative comparison with baseline methods on EarthReason dataset.

Panel of Figure 9 — Figure 9: Qualitative results for reasoning segmentation on EarthReason dataset.

Panel of Figure 10 — Figure 10: Qualitative results for referring segmentation on RRSIS-D dataset.

Panel of Figure 11 — Figure 11: Failure cases on the EarthReason and RRSIS-D datasets for reasoning and referring segmentation, respectively.

	Dataset	Source / Sensor	Resolution	Classes (FG+BG)	Split Used
Multi-class Open-Vocabulary Semantic Segmentation
	OpenEarthMap [ 73 ]	Satellite & aerial imagery (global)	0.25–0.5 m	8 + 1	Val
	LoveDA [ 69 ]	Google Earth (urban/rural)	0.3 m	6 + 1	Val
	iSAID [ 70 ]	Google Earth, JL-1, GF-2	-	15 + 1	Val
	Potsdam [ 22 ]	Aerial (urban)	5 cm	5 + 1	Val
	Vaihingen [ 22 ]	Aerial (urban)	9 cm	5 + 1	Val

Appendix

Table 8: Summary of datasets used in our paper. Rows highlighted in brown correspond to building extraction datasets, green to road extraction, and blue to flood detection datasets. FG and BG are for foreground and background classes respectively.

Dataset	Class names
OpenEarthMap [ 73 ]	Bareland, Rangeland, Developed Space, Road, Tree, Water, Agriculture Land, and Building
LoveDA [ 69 ]	Building, Road, Water, Barren, Forest, and Agriculture
iSAID [ 70 ]	Plane, Ship, Storage Tank, Baseball Diamond, Tennis Court, Basketball Court, Ground Track Field,
	Harbor, Bridge, Large Vehicle, Small Vehicle, Helicopter, Roundabout, Soccer Ball Field and Swimming Pool
Potsdam [ 22 ]	Impervious Surfaces, Buildings, Low Vegetation, Tree, and Car
Vaihingen [ 22 ]	Impervious Surfaces, Buildings, Low Vegetation, Tree, and Car

Appendix

Table 9: Class names of OVSS datasets.

Text-Guided Image Segmentation

Text-guided image segmentation maps natural-language descriptions or concepts to pixel-level regions in images.

See all 90 papers

Few-Shot Open-Vocabulary Remote Sensing Segmentation via Textual Inversion
Jul 28, 2026Junhyuk Heo, Junghwan ParkRemote Sensing Image Segmentation Few-Shot Semantic Segmentation
TelePIX, Seoul, Republic of Korea
ConInfer: Context-Aware Inference for Training-Free Open-Vocabulary Remote Sensing Segmentation
Mar 31, 2026Wenyang Chen, Zhanxuan Hu, Yaping Zhang +2Image Segmentation Remote Sensing Image Segmentation
Yunnan Normal University · Xi’an University of Posts & Telecommunications
Open-Vocabulary Semantic Segmentation in Remote Sensing via Hierarchical Attention Masking and Model Composition
Feb 27, 2026Mohammadreza Heidarianbaei, Mareike Dorozynski, Hubert Kanyamahanga +2Open-Vocabulary Semantic Segmentation Remote Sensing Image Segmentation
Institute of Photogrammetry and GeoInformation Leibniz University Hannover Germany

Jul 28, 2026J/K move · Enter open · S save

Few-Shot Open-Vocabulary Remote Sensing Segmentation via Textual Inversion

Junhyuk Heo, Junghwan Park

TelePIX, Seoul, Republic of Korea

Remote Sensing Image Segmentation Few-Shot Semantic Segmentation Open-Vocabulary Semantic Segmentation Semantic Segmentation Remote Sensing VLMs

Open paper AskPDF arXiv

Open-vocabulary segmentation labels arbitrary categories from a text query without per-class training, yet on remote sensing imagery it underperforms on categories it handles reliably elsewhere. We find that much of this gap traces to the text query rather than to the segmentation model. Because these models are not specialized for overhead imagery, the class name that serves as the query is often a weak address into the vision-language embedding space. We show that a better name repairs part of the gap, while the remaining failures call for an address that the tested natural-language rephrasings do not provide. We recover that address from a few examples through textual inversion on a frozen model, keeping inference text only. On a representative benchmark this raises the mean intersection over union on the affected categories from 3.9 to 39.4, and across eight remote sensing datasets it improves over few-shot methods that instead inject visual prompts at inference.

Abstract

Text-Guided Image Segmentation

Explore similar work

Few-Shot Open-Vocabulary Remote Sensing Segmentation via Textual Inversion

ConInfer: Context-Aware Inference for Training-Free Open-Vocabulary Remote Sensing Segmentation

Open-Vocabulary Semantic Segmentation in Remote Sensing via Hierarchical Attention Masking and Model Composition