A Geometric-Transformation Feature-Adaptive Manifold Restoration Method for Open-Vocabulary Semantic Segmentation of Remote Sensing Images

Jianzheng Wang; Huan Ni; Xiaonan Niu; Danfeng Hong; Haiyan Guan

cs.CVOct 3, 2026

A Geometric-Transformation Feature-Adaptive Manifold Restoration Method for Open-Vocabulary Semantic Segmentation of Remote Sensing Images

Authors: Jianzheng Wang, Huan Ni, Xiaonan Niu, Danfeng Hong, Haiyan Guan

Organizations: School of Remote Sensing and Geomatics Engineering, Nanjing University of Information Science and Technology, China, Nanjing, 210044, China · Nanjing Center, China Geological Survey, Nanjing, 210016, China · School of Automation, Southeast University, Nanjing, 210096, China

Abstract

The semantic information of objects in remote sensing images is typically invariant to geometric transformations from the dihedral group D4. However, SAM3-based open-vocabulary semantic segmentation (OVSS) methods often exhibit inconsistent responses to different geometric transformations. To exploit this property and improve the stability of OVSS for remote sensing images, we propose a feature-adaptive manifold repair method based on dihedral-group geometric transformations. First, we introduce multi-scale harmonic-guided D4 view selection (MH-D4VS) to select complementary candidate views from a set of geometrically transformed views. Next, we propose original-view-anchored adaptive manifold repair (OAMR), which uses the original view as an anchor and reliable cross-view information to selectively repair locally unreliable visual features. Finally, we develop pixel decoder test-time adaptation (PD-TTA) for SAM3, which fine-tunes only the parameters of the GroupNorm layers online during inference, thereby enhancing the model's ability to adapt to sample-level distribution shifts. Experimental results show that the proposed method achieves an average mIoU of 55.6% across eight remote sensing semantic segmentation benchmarks and delivers consistent performance improvements under different SAM3-based inference frameworks.

Figures & tables

Figure 1: Overview of the proposed Geometry-Transformation-Aware Feature-Adaptive Manifold Restoration framework (GFMR). GFMR consists of three components including MH-D4VS, OAMR and PD-TTA.

Figure 2: Overview of MH-D4VS. Multi-scale oriented edges are first used to construct and aggregate first- to fourth-order complex harmonic moments, while chirality is computed to capture handedness information. The action of the $\mathrm{D}_{4}$ group is then applied to derive the structural preferences of different geometric views. Finally, the optimal three-view combination $C^{\ast}$ is selected by jointly considering structural quality $Q(C)$ , geometric diversity $D(C)$ , and transformation-family balance $F(C)$ . The selected views are combined with the original view $O$ to form the multi-view representation used in subsequent processing.

Figure 3: Overview of OAMR. Features from the three candidate views are first aligned to the original coordinate system through inverse $\mathrm{D}_{4}$ transformations, and reliable cross-view observations are used to construct adaptive local subspaces. Refined cross-view metrics and candidate support then identify reliable replacement sources, enabling selective repair only of original-view features that deviate significantly from the local manifold. Finally, candidate-source consistency at a high-resolution feature level guides cross-scale extension at the target repair level.

Figure 4: The PD-TTA in PixelDecoder of SAM3

Figure 5: Qualitative comparison of the proposed method with existing open-vocabulary semantic segmentation methods.

Method	OpenEarthMap	LoveDA	iSAID	Potsdam	Vaihingen	UAVid	UDD5	VDD	Avg.
CLIP [ 1 ]	12.0	12.4	7.5	15.6	10.8	10.9	9.5	14.2	11.6
MaskCLIP [ 2 ]	25.1	27.8	14.5	33.9	29.9	28.6	32.4	32.9	28.1
SCLIP [ 3 ]	29.3	30.4	16.1	39.6	35.9	31.4	38.7	37.9	32.4
GEM [ 60 ]	33.9	31.6	17.7	39.1	36.4	33.4	41.2	39.5	34.1
ClearCLIP [ 4 ]	31.0	32.4	18.2	42.0	36.2	36.2	41.8	39.3	34.6
ProxyCLIP [ 11 ]	38.9	34.3	21.8	49.0	47.5	35.8	40.8	47.8	39.5

Table 1: Performance comparison of open-vocabulary semantic segmentation methods on eight remote sensing benchmark datasets (mIoU, %). The best results are highlighted in bold.

Method	OpenEarthMap	LoveDA	iSAID	Potsdam	Vaihingen	UAVid	UDD5	VDD	Avg.
ProC-SAM3	41.0	41.6	20.7	57.1	60.4	54.3	71.6	64.1	51.4
ProC-SAM3 + MH-D4VS + OAMR	47.7	42.0	25.8	57.3	60.6	54.6	72.4	65.4	53.2

Table 2: Evaluation of the proposed training-free visual adaptation modules on ProC-SAM3, another SAM 3-based framework (mIoU, %).

Figure 6: Qualitative results of the proposed method within the ProC-SAM3 framework. (a) Input images; (b) predictions from the original ProC-SAM3; (c) predictions after integrating the proposed MH-D4VS and OAMR modules; (d) ground truth.

Ablation Setting	MH-D4VS	OAMR	PD-TTA	OpenEarthMap	Vaihingen	VDD
A. (SegEarth-OV3 baseline)	–	–	–	42.90	60.80	64.50
B.	$\checkmark$	$\times$	–	44.16	60.77	53.05
C.	$\times$	$\checkmark$	–	46.27	62.74	64.79
D.	$\circ$	$\checkmark$	–	47.21	63.17	65.06
E.	$\checkmark$	$\checkmark$	–	48.78	63.79	65.77
F.	$\checkmark$	$\checkmark$	$\checkmark$	49.45	64.30	66.21

Table 3: Overall ablation results for MH-D4VS, OAMR, and PD-TTA (mIoU, %). A dash (–) indicates that the original SegEarth-OV3 baseline configuration is retained without introducing the corresponding module or replacement strategy. A check mark (

\checkmark

) indicates the proposed module. For MH-D4VS, a cross (

\times

) denotes Random-3 view selection, while a circle (

\circ

) denotes the use of all candidate views. For OAMR, a cross (

\times

) denotes Mean Fusion. PD-TTA denotes Pixel Decoder Test-Time Adaptation.

Figure 7: Qualitative comparison of different ablation configurations. From left to right: representative samples from the OpenEarthMap, Vaihingen, and VDD datasets.

Multi-Scale Harmonic Structural Encoding	Cross-Order Harmonic Chirality Representation	Diversity-Aware Triplet Selection	OpenEarthMap	Vaihingen	VDD
$\times$	$\checkmark$	$\checkmark$	48.17	63.56	65.71
$\checkmark$	$\times$	$\checkmark$	48.07	63.58	65.62
$\checkmark$	$\checkmark$	$\times$	48.19	63.64	65.72
$\checkmark$	$\checkmark$	$\checkmark$	49.45	64.30	66.21

Table 4: Ablation study of different components in MH-D4VS (mIoU, %).

Adaptive Local Subspace Modeling	Refined Cross-View Metric-Guided Refit	Cross-Scale Guidance	OpenEarthMap	Vaihingen	VDD
$\times$	$\checkmark$	$\checkmark$	44.13	60.88	64.35
$\checkmark$	$\times$	$\checkmark$	44.24	60.90	64.36
$\checkmark$	$\checkmark$	$\times$	46.26	63.74	65.64
$\checkmark$	$\checkmark$	$\checkmark$	49.45	64.30	66.21

Table 5: Ablation study of different components in OAMR (mIoU, %).

Figure 8: Visualization of multi-view feature repair with OAMR. On the left, the top row shows the Original View and three transformed views (Principal, Secondary, and Third), while the bottom row visualizes their features in a shared PCA space after alignment to the original coordinate system. The right panel shows the features repaired by OAMR. All features use the same PCA projection and color mapping. Colors indicate only relative feature projections in the low-dimensional PCA space and do not correspond to specific semantic classes.

Figure 9: Spatial distribution, replacement sources, and changes in local manifold consistency for selective feature repair with OAMR. (a) Final feature repair regions, with red locations indicating feature replacement. (b) Distribution of final replacement sources. $O$ indicates retention of the original-view features, while $P$ , $S$ , and $T$ indicate replacement with aligned features from the Principal, Secondary, and Third candidate views, respectively. (c) Spatial distribution of improvements in local manifold anomaly scores. Colors represent the minimum local subspace anomaly score before repair minus the anomaly score after repair. Positive values indicate improved consistency with the local manifold, whereas negative values indicate increased deviation. Black contours delineate the final repair locations, and the displayed values report the mean improvement within the final repair regions. (d) Changes in the feature distribution at the final repair locations in a shared PCA space. Gray crosses denote the means of reliable anchor features. Red and green points represent the original-view features before repair and the candidate features used for replacement, respectively. Connecting lines indicate paired feature changes at the same spatial location, while red and green crosses denote the feature means before and after repair, respectively.

Remote Sensing Image Segmentation

Remote sensing image segmentation labels land cover and objects in satellite, aerial, radar, and multispectral imagery.

See all 220 papers

Prompt-Calibrated SAM 3 for Open-Vocabulary Remote Sensing Semantic Segmentation
Jun 20, 2026Yanghui Song, Nanqing Liu, Haonan Yin +3Remote Sensing Image Segmentation Promptable Image Segmentation
School of Information Science and Technology, Yunnan Normal University, Kunming, China · School of Computer Science and Engineering, Beihang University, Beijing, China · College of Computer Science, Beijing University of Technology, Beijing, China
Seeking Consensus: Geometric-Semantic On-the-Fly Recalibration for Open-Vocabulary Remote Sensing Semantic Segmentation
Apr 29, 2026Guanchun Wang, Chenxiao Wu, Xiangrong Zhang +4Remote Sensing Image Segmentation Test-Time Adaptation
School of Artificial Intelligence, Xidian University · MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University
RSGPNet: Geometric Prompting for Remote Sensing Open-Vocabulary Semantic Segmentation
Jun 25, 2026Shanwen Wang, Xin Sun, Sirui Wang +1Remote Sensing Image Segmentation Text-Guided Image Segmentation
Faculty of Data Science, City University of Macau, 999078, SAR Macao, China · Chair of Data Science in Earth Observation, Technical University of Munich (TUM), Germany · Munich Center for Machine Learning, Munich, Germany

Jun 20, 2026J/K move · Enter open · S save

Prompt-Calibrated SAM 3 for Open-Vocabulary Remote Sensing Semantic Segmentation

Yanghui Song, Nanqing Liu, Haonan Yin, Yingjie Gao+2

School of Information Science and Technology, Yunnan Normal University, Kunming, China · School of Computer Science and Engineering, Beihang University, Beijing, China · College of Computer Science, Beijing University of Technology, Beijing, China

Remote Sensing Image Segmentation Promptable Image Segmentation Remote Sensing Open-Vocabulary Semantic Segmentation Prompt Engineering

Open paper AskPDF arXiv

Open-vocabulary semantic segmentation (OVSS) in remote sensing images aims to segment categories beyond a fixed label space. Recent SAM 3-based methods provide a promising training-free foundation, yet three key issues remain: (1) a single class-name prompt lacks sufficient semantic coverage for complex remote sensing categories; (2) expanding each category into multiple prompts introduces redundant online text encoding; and (3) directly aggregating multiple prompt responses propagates noisy activations into the final prediction. To address these issues, we propose ProC-SAM3, which calibrates SAM 3's prompt interface for remote sensing OVSS from three complementary aspects. First, we construct an offline prompt pool where a Category Matcher groups MLLM-generated candidates into per-category sets, and Expansion Constraints further refine each set using category-specific prior knowledge. Second, the resulting text embeddings are cached and reused across all test images, eliminating repeated text encoding. Third, we introduce Presence-Guided Residual Fusion to gate unreliable decoder outputs by prompt presence and confidence, followed by peak-preserving class aggregation that retains fine-grained activations for small and sparse objects. Experiments on eight benchmarks show that ProC-SAM3 achieves an average mIoU of 56.1%, outperforming the previous best training-free method by 3.9 percentage points. Code will be available at https://github.com/YanghuiSong/ProC-SAM3.

Abstract

Remote Sensing Image Segmentation

Explore similar work

Prompt-Calibrated SAM 3 for Open-Vocabulary Remote Sensing Semantic Segmentation

Seeking Consensus: Geometric-Semantic On-the-Fly Recalibration for Open-Vocabulary Remote Sensing Semantic Segmentation

RSGPNet: Geometric Prompting for Remote Sensing Open-Vocabulary Semantic Segmentation