Puzzle: One Pass Is Not Enough for Open-Vocabulary Remote Sensing Segmentation
Organizations: 1National University of Defense Technology
Abstract
Open-vocabulary remote sensing image segmentation (OVRSIS) enables the segmentation of unseen classes without additional annotation or retraining. By analyzing the predictions of existing OVRSIS methods, we find that many misclassified patches coexist with visually similar, correctly classified patches of the same class within the same image. Unfortunately, most of these methods stop after a single inference rather than using their predictions to guide further inference. Motivated by this observation, we propose Puzzle, a plug-and-play iterative re-inference framework that combines prediction history with a visual-similarity prior to progressively refine patch-level attention. Puzzle employs a lightweight history-conditioned controller to regulate each patch's contribution as an information source and its reliance on visually similar patches, while an adaptive stopping rule determines when to stop. It keeps the segmentation model frozen and reuses the same controller across different models and datasets without retraining. Extensive experiments demonstrate consistent performance improvements across diverse OVRSIS models and datasets, highlighting the strong transferability of Puzzle. Moreover, Puzzle can further improve the performance of SOTA methods.
Figures & tables
| Remixed layers | SegEarth-OV | ClearCLIP | Avg. |
|---|---|---|---|
| UDD5 | |||
| Last layer | 52.16 | 44.41 | 48.28 |
| Last 2 layers | 48.51 | 24.23 | 36.37 |
| Last 3 layers | 46.06 | 24.93 | 35.50 |
| VDD | |||
| Last layer | 48.05 | 43.70 | 45.88 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Images | Misclassified patches | Coverage (%) | 95% CI | Same (%) | Correct (%) | |
|---|---|---|---|---|---|---|---|
| LoveDA | 1,669 | 577,735 | 44.7 | [43.8, 45.6] | 80.7 | 21.3 | 1.41 |
| Potsdam | 2,016 | 462,003 | 45.5 | [43.2, 47.7] | 80.6 | 20.6 | 1.62 |
| UDD5 | 40 | 5,019 | 66.1 | [61.4, 70.9] | 70.9 | 30.0 | 2.58 |
| All | 3,725 | 1,044,757 | 45.2 | [44.0, 46.2] | 80.6 | 21.0 | 1.51 |
| Class | Single | SE | CC | SC | Avg. |
|---|---|---|---|---|---|
| Background | 38.8 | +0.8 | +4.4 | +6.3 | +3.8 |
| Bareland | 14.7 | 0.0 | 0.8 | +0.5 | 0.1 |
| Grass | 22.3 | +0.4 | +5.2 | +8.6 | +4.7 |
| Pavement | 16.3 | 0.2 | 0.5 | +0.2 | 0.2 |
| Road | 21.5 | +0.4 | +2.7 | +4.3 | +2.5 |
| Tree | 47.5 | +0.3 | +2.4 | +2.5 | +1.7 |
| Setting | Potsdam | Vaihingen |
|---|---|---|
| Single pass | 47.13 | 29.15 |
| Run 1 | 47.88 | 30.12 |
| Run 2 | 47.94 | 30.15 |
| Run 3 | 47.83 | 30.10 |
| Average Std | ||
| +0.76 | +0.97 |
| Controller | Dataset | Single | Puzzle | 95% CI | |
|---|---|---|---|---|---|
| LoveDA | Potsdam | 40.871 | 43.190 | +2.319 | [1.834, 2.855] |
| Vaihingen | 27.302 | 29.556 | +2.254 | [1.609, 2.985] | |
| UAVid | 35.035 | 37.903 | +2.868 | [2.262, 3.596] | |
| OEM | Potsdam | 40.871 | 42.462 | +1.591 | [1.406, 1.793] |
| Vaihingen | 27.303 | 29.889 | +2.586 | [2.068, 3.039] | |
| UAVid | 35.038 | 36.707 | +1.669 | [1.347, 1.973] |
| Temporal model | Three fixed rounds | Adaptive stopping |
|---|---|---|
| Selective state-space model (ours) | 46.58 | 47.88 |
| GRU | 46.28 | 47.80 |
| Causal Transformer | 46.34 | 47.49 |
| Model | Dataset | Single pass | DINOv3 | SAM 3 |
|---|---|---|---|---|
| SegEarth-OV | Potsdam | 47.13 | 47.88 | 47.27 |
| Vaihingen | 29.15 | 30.12 | 30.22 | |
| ClearCLIP | Potsdam | 40.87 | 43.19 | 42.01 |
| Vaihingen | 27.30 | 29.56 | 29.26 | |
| SCLIP | Potsdam | 36.62 | 39.07 | 38.42 |
| Vaihingen | 28.37 | 30.78 | 30.65 |
| Host | Native components retained |
|---|---|
| CLIP | attention, residual and FFN |
| MaskCLIP | identity attention, residual and FFN |
| GEM | iterative self-self attention and native decoder |
| SCLIP | sum of and softmaxes |
| ClearCLIP | softmax without final residual or FFN |
| ResCLIP | native RCS and SFR attention/refinement |
| Dataset | Data type | Images |
|---|---|---|
| OpenEarthMap [ 44 ] | satellite / aerial, global | 384 |
| LoveDA [ 42 ] | satellite, urban and rural | 1,669 |
| iSAID [ 49 ] | aerial, instance-level | 11,644 |
| Potsdam [ 30 ] | aerial, urban | 2,016 |
| Vaihingen [ 30 ] | aerial, urban | 398 |
| UAVid [ 23 ] | UAV, oblique | 1,020 |