Semantic segmentation of remote sensing imagery is fundamental to Earth observation. Achieving accurate results requires integrating not only optical images but also physical variables such as the Digital Elevation Model (DEM), Synthetic Aperture Radar (SAR) and Normalized Difference Vegetation Index (NDVI). Recent foundation models (FMs) leverage pre-training to exploit these variables but still depend on spatially aligned data and costly retraining when involving new sensors. To overcome these limitations, we introduce a novel paradigm for integrating domain-specific physical priors into segmentation models. We first construct a Physical-Centric Knowledge Graph (PCKG) by prompting large language models to extract physical priors from 1,763 vocabularies, and use it to build a heterogeneous, spatial-aligned dataset, Phy-Sky-SA. Building on this foundation, we develop PriorSeg, a physics-aware residual refinement model trained with a joint visual-physical strategy that incorporates a novel physics-consistency loss. Experiments on heterogeneous settings demonstrate that PriorSeg improves segmentation accuracy and physical plausibility without retraining the FMs. Ablation studies verify the effectiveness of the Phy-Sky-SA dataset, the PCKG, and the physics-consistency loss.
Figures & tables
Figure 1: Visual and visual-physical segmentation: adding the physical prior (simulated SAR) preserves the true object while preventing the shadow from being segmented.
Figure 2: Schematic of PCKG construction with GPT-4o. Category labels are prompted to the LLM, which returns NDVI, DEM, and SAR intervals [a,b] plus a brief reasoning trace. The results are stored as JSON records and aggregated into a lightweight Physics-Centric Knowledge Graph that later guides synthetic-modal generation and loss design.
Figure 3: Example sample from the Phy-Sky-SA dataset. From left to right: Image, GT, and the three simulated physical variables-NDVI(S), DEM(S), and SAR(S)-generated according to the PCKG, where the “(S)” denotes simulated values.
Figure 4: Overview of the PriorSeg training framework. A frozen backbone extracts visual features and an initial mask, which are fused with optional physical variables. A residual head refines the segmentation, and three losses jointly enforce visual quality and physical consistency.
Model
Publication
RGB–Only OEM
RGB–Only iSAID
RGB–Only LoveDA
RGB–Phy WHU-OPT-SAR
RGB–Phy SEN12MS
SegGPT
ICCV 2023
25.19
29.17
31.40
23.42
21.84
SegGPT+PriorSeg
27.33(+2.14)
32.18(+3.01)
33.55(+2.15)
29.33(+5.91)
27.56(+5.72)
DINOv
CVPR 2023
23.05
25.35
25.17
22.74
20.09
DINOv+PriorSeg
24.32(+1.27)
27.39(+2.04)
27.95(+2.78)
28.67(+5.93)
25.44(+5.35)
SAN
CVPR 2023
24.87
11.77
27.83
25.97
21.33
SAN+PriorSeg
27.22(+2.35)
15.24(+3.47)
30.51(+2.68)
32.71(+6.74)
26.76(+5.43)
Table 1: Performance comparison of six backbone models with and without the proposed PriorSeg refinement module. Results are reported as mIoU(%) on four datasets: RGB-only of OEM, iSAID, LoveDA, and RGB-Physics WHU-OPT-SAR, SEN12MS.
Figure 5: Three representative visualization examples on SegEarth-OV backbone, comparing GT, SegEarth-OV predictions, and our PriorSeg-enhanced results.
Figure 6: Comparison between real and simulated physical variables on the WHU-OPT-SAR dataset.
+Phy-Sky-SA
+PCKG
+ Lphys
mIoU (%)
25.97
✓
27.69 (+1.72)
✓
✓
29.93 (+2.24)
✓
✓
✓
32.71 (+2.78)
Table 2: The ablation results of the SAN model on the WHU-OPT-SAR dataset, evaluating Phy-Sky-SA, PCKG, and Lphys .
Training-free open-vocabulary remote sensing segmentation (OVRSS), empowered by vision-language models, has emerged as a promising paradigm for achieving category-agnostic semantic understanding in remote sensing imagery. Existing approaches mainly focus on enhancing feature representations or mitigating modality discrepancies to improve patch-level prediction accuracy. However, such independent prediction schemes are fundamentally misaligned with the intrinsic characteristics of remote sensing data. In real-world applications, remote sensing scenes are typically large-scale and exhibit strong spatial as well as semantic correlations, making isolated patch-wise predictions insufficient for accurate segmentation. To address this limitation, we propose ConInfer, a context-aware inference framework for OVRSS that performs joint prediction across multiple spatial units while explicitly modeling their inter-unit semantic dependencies. By incorporating global contextual cues, our method significantly enhances segmentation consistency, robustness, and generalization in complex remote sensing environments. Extensive experiments on multiple benchmark datasets demonstrate that our approach consistently surpasses state-of-the-art per-pixel VLM-based baselines such as SegEarth-OV, achieving average improvements of 2.80% and 6.13% on open-vocabulary semantic segmentation and object extraction tasks, respectively. The implementation code is available at: https://github.com/Dog-Yang/ConInfer
Wenyang Chen, Zhanxuan Hu, Yaping Zhang +2
Yunnan Normal University · Xi’an University of Posts & Telecommunications
Recent advances in MLLMs are reframing segmentation from fixed-category prediction to instruction-grounded localization. While reasoning based segmentation has progressed rapidly in natural scenes, remote sensing lacks a generalizable solution due to the prohibitive cost of reasoning-oriented data and domain-specific challenges like overhead viewpoints. We present GeoSeg, a zero-shot, training-free framework that bypasses the supervision bottleneck for reasoning-driven remote sensing segmentation. GeoSeg couples MLLM reasoning with precise localization via: (i) bias-aware coordinate refinement to correct systematic grounding shifts and (ii) a dual-route prompting mechanism to fuse semantic intent with fine-grained spatial cues. We also introduce GeoSeg-Bench, a diagnostic benchmark of 810 image--query pairs with hierarchical difficulty levels. Experiments show that GeoSeg consistently outperforms all baselines, with extensive ablations confirming the effectiveness and necessity of each component.
Lifan Jiang, Yuhang Pei, oxi Wu +5
State Key lab of CAD&CG, Zhejiang University · UniTTEC Co. Ltd. · Wuchan Zhongda Chengtou (Ningbo) Holdings. Ltd.
Open-vocabulary semantic segmentation (OVS) of remote sensing imagery is a challenging pixel-level task requiring strong generalization and adaptation to the spatial characteristics of remote sensing data. Although existing vision-language foundation models perform well in general domains, their image-level classification design weakens the spatial priors needed for high-resolution remote sensing segmentation: structural spatial relations are degraded during deep feature transformation, and fine-grained spatial details are lost during downsampling. To address these complementary deficiencies, we propose SatOV, a training-free framework for open-vocabulary remote sensing segmentation that restores spatial priors at two stages of the representation pipeline. Specifically, Residual QQ Attention (ResQQ) extracts Query-Key self-attention from an intermediate CLIP layer and fuses it with final-layer Query-Query attention via a residual combination, restoring structural spatial priors suppressed by the final-layer representation. Spatially Modulated Upsampling (SatUp) uses the original high-resolution RGB image as spatial guidance, combining spatial feature modulation with guided cross-attention to reconstruct pixel-level textures and boundaries. Extensive experiments on DOTA, UDD, LoveDA, and Vaihingen show that SatOV consistently improves training-free OVS and achieves competitive quantitative and qualitative results against state-of-the-art methods. These results validate the effectiveness of restoring spatial priors at both the representation and spatial-resolution stages for remote sensing open-vocabulary segmentation.
Changhao Zhao, Linglin Zeng, Hai Liu
College of Resources and Environment, Huazhong Agricultural University, Wuhan 430070, China