Open-vocabulary remote sensing image segmentation (OVRSIS) enables the segmentation of unseen classes without additional annotation or retraining. By analyzing the predictions of existing OVRSIS methods, we find that many misclassified patches coexist with visually similar, correctly classified patches of the same class within the same image. Unfortunately, most of these methods stop after a single inference rather than using their predictions to guide further inference. Motivated by this observation, we propose Puzzle, a plug-and-play iterative re-inference framework that combines prediction history with a visual-similarity prior to progressively refine patch-level attention. Puzzle employs a lightweight history-conditioned controller to regulate each patch's contribution as an information source and its reliance on visually similar patches, while an adaptive stopping rule determines when to stop. It keeps the segmentation model frozen and reuses the same controller across different models and datasets without retraining. Extensive experiments demonstrate consistent performance improvements across diverse OVRSIS models and datasets, highlighting the strong transferability of Puzzle. Moreover, Puzzle can further improve the performance of SOTA methods.
Figures & tables
Figure 1: Inference paradigms for open-vocabulary segmentation. Each jigsaw piece denotes an image patch, and red pieces denote misclassified patches. (a)–(c) Existing paradigms predict once, refine the output, or update parameters or prompts with a test-time loss. (d) Puzzle re-infers the frozen model and lets reliable pieces guide uncertain ones through attention.
Figure 2: Correctly classified look-alikes of misclassified patches. (a,b) A misclassified car patch (dark square) and its top-10 look-alikes (green: correct; red: misclassified) in a Potsdam image and the single-pass prediction of SegEarth-OV. (c) Share of misclassified patches with a correctly classified look-alike of the same class before and after applying Puzzle.
Figure 3: Overall framework of Puzzle. Starting from the single-pass prediction, Puzzle combines prediction history with a visual-similarity prior to adjust patch-level attention and re-infer the frozen OVRSIS model. The history is updated across rounds, while adaptive stopping determines when to return the last accepted prediction. (a) History-conditioned controller; (b) attention remix; (c) adaptive stopping.
Figure 4: Attention of a misclassified patch ( ★ ) before and after the remix (Potsdam, SegEarth-OV).
Figure 5: Qualitative comparison.
Remixed layers
SegEarth-OV
ClearCLIP
Avg.
UDD5
Last layer
52.16
44.41
48.28
Last 2 layers
48.51
24.23
36.37
Last 3 layers
46.06
24.93
35.50
VDD
Last layer
48.05
43.70
45.88
Table 3: Sensitivity to the number of remixed layers (mIoU).
Figure 6: (a) mIoU with six forced rounds; stars: adaptive stopping at its average returned round. (b) Correction rate of misclassified patches versus reference support, the share of correctly classified, same-class patches among their ten look-alikes.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Images
Misclassified patches
Coverage (%)
95% CI
Same (%)
Correct (%)
n
LoveDA
1,669
577,735
44.7
[43.8, 45.6]
80.7
21.3
1.41
Potsdam
2,016
462,003
45.5
[43.2, 47.7]
80.6
20.6
1.62
UDD5
40
5,019
66.1
[61.4, 70.9]
70.9
30.0
2.58
All
3,725
1,044,757
45.2
[44.0, 46.2]
80.6
21.0
1.51
Appendix
Table S1 : Correctly classified look-alikes of the same class of misclassified patches (single pass of SegEarth-OV). Coverage: share of misclassified patches with at least one such look-alike (95% bootstrap interval, resampled by image for LoveDA and UDD5, and by original scene for Potsdam); Same: share of all look-alikes with the same true class as the target; Correct: share of all look-alikes with the same true class and a correct prediction; n : average number of the latter per misclassified patch.
Class
Single
SE
CC
SC
Avg.
Background
38.8
+0.8
+4.4
+6.3
+3.8
Bareland
14.7
0.0
− 0.8
+0.5
− 0.1
Grass
22.3
+0.4
+5.2
+8.6
+4.7
Pavement
16.3
− 0.2
− 0.5
+0.2
− 0.2
Road
21.5
+0.4
+2.7
+4.3
+2.5
Tree
47.5
+0.3
+2.4
+2.5
+1.7
Appendix
Table S2 : Per-class IoU on OEM (full evaluation set). Single: single-pass IoU averaged over the three models; SE, CC, SC: change brought by Puzzle on SegEarth-OV, ClearCLIP and SCLIP, respectively; Avg.: average of the three changes.
Setting
Potsdam
Vaihingen
Single pass
47.13
29.15
Run 1
47.88
30.12
Run 2
47.94
30.15
Run 3
47.83
30.10
Average ± Std
47.89±0.06
30.12±0.03
Δ
+0.76
+0.97
Appendix
Table S3 : Repeated runs (SegEarth-OV, LoveDA-trained controller, full evaluation sets, mIoU). Run 1 is the controller of the main paper; Average ± Std: average and sample standard deviation over the three runs ( n=3 ); Δ : gain of the average over the single pass.
Controller
Dataset
Single
Puzzle
Δ
95% CI
LoveDA
Potsdam
40.871
43.190
+2.319
[1.834, 2.855]
Vaihingen
27.302
29.556
+2.254
[1.609, 2.985]
UAVid
35.035
37.903
+2.868
[2.262, 3.596]
OEM
Potsdam
40.871
42.462
+1.591
[1.406, 1.793]
Vaihingen
27.303
29.889
+2.586
[2.068, 3.039]
UAVid
35.038
36.707
+1.669
[1.347, 1.973]
Appendix
Table S4 : Paired cluster bootstrap of the gains (ClearCLIP, full evaluation sets, mIoU). Controller: training dataset of the controller; Single: single pass; Δ : gain of Puzzle over the single pass; 95% CI: percentile interval of Δ over 5,000 replicates, resampled by original scene for Potsdam and Vaihingen and by video sequence for UAVid.
Figure S1 : More qualitative results (SegEarth-OV): one example each from OEM, iSAID, Vaihingen, UDD5 and VDD. Boxes mark the regions that the single pass misclassifies (terracotta) and Puzzle classifies correctly (teal).
Temporal model
Three fixed rounds
Adaptive stopping
Selective state-space model (ours)
46.58
47.88
GRU
46.28
47.80
Causal Transformer
46.34
47.49
Appendix
Table S5 : Temporal model (SegEarth-OV, Potsdam, mIoU).
Model
Dataset
Single pass
DINOv3
SAM 3
SegEarth-OV
Potsdam
47.13
47.88
47.27
Vaihingen
29.15
30.12
30.22
ClearCLIP
Potsdam
40.87
43.19
42.01
Vaihingen
27.30
29.56
29.26
SCLIP
Potsdam
36.62
39.07
38.42
Vaihingen
28.37
30.78
30.65
Appendix
Table S6 : Source of the look-alikes (mIoU). Each VFM uses its own controller.
Host
Native components retained
CLIP
qk⊤ attention, residual and FFN
MaskCLIP
identity attention, residual and FFN
GEM
iterative self-self attention and native decoder
SCLIP
sum of qq⊤ and kk⊤ softmaxes
ClearCLIP
qq⊤ softmax without final residual or FFN
ResCLIP
native RCS and SFR attention/refinement
Appendix
Table S8 : Host-specific components retained by Puzzle. All listed hosts use ViT-B/16 CLIP encoders. Softmax normalization and decoding follow each native implementation.
Dataset
Data type
Images
OpenEarthMap [ 44 ]
satellite / aerial, global
384
LoveDA [ 42 ]
satellite, urban and rural
1,669
iSAID [ 49 ]
aerial, instance-level
11,644
Potsdam [ 30 ]
aerial, urban
2,016
Vaihingen [ 30 ]
aerial, urban
398
UAVid [ 23 ]
UAV, oblique
1,020
Appendix
Table S10 : Evaluation sets of the SegEarth-OV protocol.
Training-free open-vocabulary remote sensing segmentation (OVRSS), empowered by vision-language models, has emerged as a promising paradigm for achieving category-agnostic semantic understanding in remote sensing imagery. Existing approaches mainly focus on enhancing feature representations or mitigating modality discrepancies to improve patch-level prediction accuracy. However, such independent prediction schemes are fundamentally misaligned with the intrinsic characteristics of remote sensing data. In real-world applications, remote sensing scenes are typically large-scale and exhibit strong spatial as well as semantic correlations, making isolated patch-wise predictions insufficient for accurate segmentation. To address this limitation, we propose ConInfer, a context-aware inference framework for OVRSS that performs joint prediction across multiple spatial units while explicitly modeling their inter-unit semantic dependencies. By incorporating global contextual cues, our method significantly enhances segmentation consistency, robustness, and generalization in complex remote sensing environments. Extensive experiments on multiple benchmark datasets demonstrate that our approach consistently surpasses state-of-the-art per-pixel VLM-based baselines such as SegEarth-OV, achieving average improvements of 2.80% and 6.13% on open-vocabulary semantic segmentation and object extraction tasks, respectively. The implementation code is available at: https://github.com/Dog-Yang/ConInfer
Wenyang Chen, Zhanxuan Hu, Yaping Zhang +2
Yunnan Normal University · Xi’an University of Posts & Telecommunications
Open-vocabulary remote sensing image segmentation (OVRSIS) aims to segment text-specified categories beyond a fixed label space. Its key challenge is to maintain reliable pixel--text correspondence under substantial appearance variation in remote-sensing imagery. Existing methods often rely on specialized vision--language representations or complex adaptation mechanisms, yet still perform deterministic matching between fixed text prototypes and individual visual features. We instead formulate OVRSIS as a distributional pixel--text alignment problem and propose Pi-Seg, a lightweight perturbation-based framework. Pi-Seg introduces learnable variations into textual prototypes and dense visual features before alignment. Segmentation supervision encourages perturbations that enlarge the target-to-distractor margin while suppressing harmful feature shifts. Pi-Seg consistently improves performance under the existing single-source OVRSISBench protocols. To evaluate OVRSIS beyond source-specific training settings, we further introduce a unified multi-source protocol and construct GlobalRSOV95K, containing approximately 95K densely annotated images and 35 semantic categories. Models are trained on this shared dataset and evaluated on an image-disjoint suite of 10 downstream datasets. The complete benchmark covers more than 170K images and 122 categories. Experiments under both single-source and multi-source protocols show that Pi-Seg improves cross-dataset transfer and generalizes effectively to practical geospatial segmentation tasks, while GlobalRSOV95K provides a common foundation for transferable OVRSIS research. \footnote{https://github.com/LiBingyu01/Pi-Seg}
Bingyu Li, Tao Huo, Haocheng Dong +4
Department of Electronic Engineering and Information Science, University of Science and Technology of China, China · Institute of Artificial Intelligence (TeleAI), China Telecom, P. R. China · School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University, China
Open-vocabulary semantic segmentation (OVS) of remote sensing imagery is a challenging pixel-level task requiring strong generalization and adaptation to the spatial characteristics of remote sensing data. Although existing vision-language foundation models perform well in general domains, their image-level classification design weakens the spatial priors needed for high-resolution remote sensing segmentation: structural spatial relations are degraded during deep feature transformation, and fine-grained spatial details are lost during downsampling. To address these complementary deficiencies, we propose SatOV, a training-free framework for open-vocabulary remote sensing segmentation that restores spatial priors at two stages of the representation pipeline. Specifically, Residual QQ Attention (ResQQ) extracts Query-Key self-attention from an intermediate CLIP layer and fuses it with final-layer Query-Query attention via a residual combination, restoring structural spatial priors suppressed by the final-layer representation. Spatially Modulated Upsampling (SatUp) uses the original high-resolution RGB image as spatial guidance, combining spatial feature modulation with guided cross-attention to reconstruct pixel-level textures and boundaries. Extensive experiments on DOTA, UDD, LoveDA, and Vaihingen show that SatOV consistently improves training-free OVS and achieves competitive quantitative and qualitative results against state-of-the-art methods. These results validate the effectiveness of restoring spatial priors at both the representation and spatial-resolution stages for remote sensing open-vocabulary segmentation.
Changhao Zhao, Linglin Zeng, Hai Liu
College of Resources and Environment, Huazhong Agricultural University, Wuhan 430070, China