Open-Vocabulary Semantic Segmentation in Remote Sensing via Hierarchical Attention Masking and Model Composition

Mohammadreza Heidarianbaei; Mareike Dorozynski; Hubert Kanyamahanga; Max Mehltretter; Franz Rottensteiner

cs.CVFeb 27, 2026

Open-Vocabulary Semantic Segmentation in Remote Sensing via Hierarchical Attention Masking and Model Composition

Authors: Mohammadreza Heidarianbaei, Mareike Dorozynski, Hubert Kanyamahanga, Max Mehltretter, Franz Rottensteiner

Organizations: Institute of Photogrammetry and GeoInformation Leibniz University Hannover Germany

Abstract

In this paper, we propose ReSeg-CLIP, a new training-free Open-Vocabulary Semantic Segmentation method for remote sensing data. To compensate for the problems of vision language models, such as CLIP in semantic segmentation caused by inappropriate interactions within the self-attention layers, we introduce a hierarchical scheme utilizing masks generated by SAM to constrain the interactions at multiple scales. We also present a model composition approach that averages the parameters of multiple RS-specific CLIP variants, taking advantage of a new weighting scheme that evaluates representational quality using varying text prompts. Our method achieves state-of-the-art results across three RS benchmarks without additional training.

Figures & tables

Figure 1: Example of distorted inter-patch attention for a selected patch (white squares). Left to right: input image, attention map obtained by the original CLIP vision encoder [ 30 ] , and the attention map obtained by our method. Blue corresponds to low attention, red to high attention in relation to the selected patch. CLIP often assigns high attention to arbitrary patches without any relevance for the selected patch. Applying our method results in high attention concentrated on patches associated with the same object as the selected patch.

Figure 2: ReSeg-CLIP consists of CLIP-based vision and text encoders and SAM. The input image $\mathbf{X}$ is processed by both the vision encoder and SAM, resulting in features $\mathbf{F}$ . SAM produces hierarchical masks $\mathcal{M}$ , converted into attention masks $\mathcal{A}$ for the final vision encoder layers (red blocks). Text prompts for each class are encoded into embeddings $\mathbf{T}$ , which are compared with the features $\mathbf{F}$ via cosine similarity. The results are upsampled to score map Sim , and the segmentation $\hat{\mathbf{Y}}$ assigns to each pixel the class with highest similarity.

Figure 3: Results on the UDD5 dataset. SegEarth-OV yields more homogeneous masks, our method offers better class distinction in adjacent regions (red circles). Despite some interpolation-induced noise, our model effectively detects mislabeled areas (orange square).

Method	Potsdam	UDD5	OEM	Training
Cao et al. [ 2 ]	30.3	-	-	✓
Ye et al. [ 42 ]	45.7	-	-	✓
SegEarth-OV	47.1	50.6	40.3	✠
ReSeg-CLIP	38.3	43.2	32.4	✗

Table 1: Comparison of mIoU [%] across OVSS RS methods. Last column:

\checkmark

: full-network training; ✠: training of the upsampling module;

\times

: no training.

Dataset	CLIP	MaskCLIP	SCLIP	GEM	ClearCLIP	C+P	Ours
Potsdam	14.5	31.7	36.6	36.5	\textcolor red40.9	18.8	\textcolor blue38.3
UDD5	9.5	32.4	38.7	41.2	\textcolor blue41.8	15.0	\textcolor red43.2
OEM	12.0	25.1	29.3	\textcolor red33.9	31.0	15.3	\textcolor blue32.4

Table 2: Comparison of mIoU [%] (best in red, second best in blue) across training-free general-purpose methods. C+P refers to ReSeg-CLIP without using any SAM-based attention masks in the vision encoder, i.e.,

|\Theta|=0

Parametrization	Weighting	0	1	2	3	4	5	mIoU
CLIP [ 30 ]	-	25.9	44.4	34.6	41.5	6.1	3.4	24.5
GeoRSCLIP [ 50 ]	-	34.8	54.4	\textcolor blue50.1	51.0	\textcolor red14.7	\textcolor red5.6	33.0
RemoteCLIP [ 25 ]	-	36.5	\textcolor blue61.4	39.1	48.1	2.8	2.5	30.4
[ 50 ] + [ 25 ]	PVSM	\textcolor red41.7	60.2	\textcolor red53.3	\textcolor red59.3	11.3	3.7	\textcolor red38.3
[ 50 ] + [ 25 ]	equal	\textcolor blue39.1	56.5	49.9	\textcolor blue55.6	10.6	3.5	\textcolor blue35.9
[ 50 ] + [ 25 ] + [ 30 ]	PVSM	29.0	48.9	34.9	44.1	8.1	2.8	28.9

Table 3: Per-class IoU [%] and mean IoU (mIoU) [%] (best in red, second best in blue) on the Potsdam dataset. All variants with

|\Theta|=6

layers with SAM-based attention masks in the vision encoder (cf. Sec. 3.1 ). Classes: 0: Artificial Surface, 1: Building, 2: Natural Surface, 3: Vegetation, 4: Vehicle, 5: Background.

$\mathbf{\|\Theta\|}$	0	1	3	6	12	18
mIoU [%]	18.8	25.5	\textcolor blue33.7	\textcolor red38.3	32.1	19.4

Table 4: Effect of number of layers in ReSeg-CLIP’s vision encoder with SAM-based attention masks

|\Theta|

on Potsdam dataset (best in red, second best in blue).

Open-Vocabulary Semantic Segmentation

Open-vocabulary semantic segmentation labels image pixels using text-defined categories beyond a fixed training vocabulary.

See all 84 papers

SatOV: Restoring Spatial Priors for Training-Free Open-Vocabulary Segmentation in Remote Sensing Imagery
Sep 19, 2026Changhao Zhao, Linglin Zeng, Hai LiuOpen-Vocabulary Semantic Segmentation Remote Sensing Image Segmentation
College of Resources and Environment, Huazhong Agricultural University, Wuhan 430070, China
Decouple and Rectify: Semantics-Preserving Structural Enhancement for Open-Vocabulary Remote Sensing Segmentation
Apr 2, 2026Jie Feng, Fengze Li, Junpeng Zhang +4Remote Sensing Image Segmentation Open-Vocabulary Semantic Segmentation
Xidian University China · Jimei University China · South China University of Technology China
Enabling Training-Free Text-Based Remote Sensing Segmentation
Feb 19, 2026Jose Sosa, Danila Rukhovich, Anis Kacem +1Text-Guided Image Segmentation Remote Sensing Image Segmentation
SnT, University of Luxembourg

Sep 19, 2026J/K move · Enter open · S save

SatOV: Restoring Spatial Priors for Training-Free Open-Vocabulary Segmentation in Remote Sensing Imagery

Changhao Zhao, Linglin Zeng, Hai Liu

College of Resources and Environment, Huazhong Agricultural University, Wuhan 430070, China

Open-Vocabulary Semantic Segmentation Remote Sensing Image Segmentation Semantic Segmentation

Open paper AskPDF arXiv

Open-vocabulary semantic segmentation (OVS) of remote sensing imagery is a challenging pixel-level task requiring strong generalization and adaptation to the spatial characteristics of remote sensing data. Although existing vision-language foundation models perform well in general domains, their image-level classification design weakens the spatial priors needed for high-resolution remote sensing segmentation: structural spatial relations are degraded during deep feature transformation, and fine-grained spatial details are lost during downsampling. To address these complementary deficiencies, we propose SatOV, a training-free framework for open-vocabulary remote sensing segmentation that restores spatial priors at two stages of the representation pipeline. Specifically, Residual QQ Attention (ResQQ) extracts Query-Key self-attention from an intermediate CLIP layer and fuses it with final-layer Query-Query attention via a residual combination, restoring structural spatial priors suppressed by the final-layer representation. Spatially Modulated Upsampling (SatUp) uses the original high-resolution RGB image as spatial guidance, combining spatial feature modulation with guided cross-attention to reconstruct pixel-level textures and boundaries. Extensive experiments on DOTA, UDD, LoveDA, and Vaihingen show that SatOV consistently improves training-free OVS and achieves competitive quantitative and qualitative results against state-of-the-art methods. These results validate the effectiveness of restoring spatial priors at both the representation and spatial-resolution stages for remote sensing open-vocabulary segmentation.

Abstract

Open-Vocabulary Semantic Segmentation

Explore similar work

SatOV: Restoring Spatial Priors for Training-Free Open-Vocabulary Segmentation in Remote Sensing Imagery

Decouple and Rectify: Semantics-Preserving Structural Enhancement for Open-Vocabulary Remote Sensing Segmentation

Enabling Training-Free Text-Based Remote Sensing Segmentation