Frequency and Edge-Guided Segment Anything Model for Remote Sensing Image Semantic Segmentation

Feng Gao; Zizhe Pan; Haoting Wang; Ruzhuang Hua; Jingchao Cao; Junyu Dong; Qian Du

cs.CVAug 15, 2026

Frequency and Edge-Guided Segment Anything Model for Remote Sensing Image Semantic Segmentation

Authors: Feng Gao, Zizhe Pan, Haoting Wang, Ruzhuang Hua, Jingchao Cao, Junyu Dong, Qian Du

Organizations: State Key Laboratory of Physical Oceanography, Ocean University of China, Qingdao 266100, China · Department of Electrical and Computer Engineering, Mississippi State University, Starkville, MS 39762 USA

Abstract

Remote sensing image semantic segmentation (RSISS) has attracted significant attention due to the growing demand for fine-grained land cover information. The Segment Anything Model (SAM), proposed as a foundation vision model, offers strong segmentation performance and generalization capabilities for RSISS tasks. However, existing SAM-based approaches face two limitations: (1) Insufficient adaptation of SAM's features to the diverse characteristics of land cover types. (2) Semantic ambiguity at object boundaries, which hinders accurate delineation. To address these limitations, we propose Frequency and Edge-guided SAM (FE-SAM), a scalable and efficient framework for RSISS. Specifically, we introduce a Frequency-Modulated Adapter (FMA) that adaptively decomposes and modulates frequency-domain features based on the input data. It selectively enhances informative high- and low-frequency components corresponding to different land cover types. Furthermore, to improve SAM's ability to capture fine-grained details, we design EGRefiner, which integrates multi-scale edge-enhanced information extracted from the input image. Extensive experiments on three benchmark datasets demonstrate that FE-SAM outperforms state-of-the-art methods. The source codes are available at: https://github.com/oucailab/FE-SAM.

Figures & tables

Fig. 1: Framework of our Frequency and Edge-guided Segment Anything Model ( FE-SAM ) for remote sensing image semantic segmentation. It reuses the pre-trained weights of SAM with two novel modules: Frequency-Modulated Adapter (FMA) and Edge-Guided Refiner (EGRefiner).

Fig. 2: Framework of our Frequency and Edge-guided Segment Anything Model ( FE-SAM ) for remote sensing image semantic segmentation. It reuses the pre-trained weights of SAM with two novel modules: Frequency-Modulated Adapter (FMA) and Edge-Guided Refiner (EGRefiner).

Fig. 3: Cross-dataset spectral analysis. Top rows: representative patches and their 2D log-magnitude spectra from SA-1B and three remote sensing datasets. Bottom: averaged radial PSD curves.

Fig. 4: Illustration of the Edge-Guided Refiner (EGRefiner).

	Class	Pixels( $\times 10^{6}$ )	Proportion(%)
Vaihingen	Imp. surf.	121.5	29.2
	Building	113.6	27.3
	Low veg.	79.1	19.0
	Tree	80.7	19.4
	Car	5.0	1.2
	Clutter	16.2	3.9

TABLE I: Pixel-level class distribution on three datasets

Fig. 5: Statistical significance analysis of mIoU on the ISPRS Vaihingen and Potsdam datasets.

Method	Type	Vaihingen	Potsdam	LoveDA
FCN8s	CNN- based	78.25	81.03	48.02
U-Net		80.96	83.96	50.46
DeepLabv3+		81.78	84.02	51.45
MA-Net		82.76	84.91	51.64
A2-FPN		82.40	85.16	52.20
UNetFormer	Transformer- based	82.86	85.24	52.44

TABLE II: Semantic segmentation results (measured by mIoU) of various methods on three benchmark datasets. The bold and underline denote the best and second results.

Method	Type	Backbone	Per-class F1-score					mIoU(%)	m-F1(%)	OA(%)
Method	Type	Backbone	Imp.surf.	Building	Low veg.	Tree	Car	mIoU(%)	m-F1(%)	OA(%)
FCN8s	CNN- based	VGG-16	95.22	91.89	81.76	88.52	79.68	78.25	87.41	88.96
U-Net		ResNet-18	96.03	94.51	82.58	88.73	83.76	80.96	89.12	89.94
DeepLabv3+		ResNet-18	96.38	95.13	82.88	88.86	84.84	81.78	89.62	90.50
MA-Net		ResNet-18	96.39	95.17	83.86	89.51	86.33	82.76	90.25	90.69
A2-FPN		ResNet-18	96.31	95.04	83.01	89.08	86.67	82.40	90.02	90.52

TABLE III: Quantitative comparisons with the state-of-the-art methods on the ISPRS Vaihingen dataset. The bold and underline denote the best and second results.

Fig. 6: Additional qualitative results on the ISPRS Vaihingen dataset. From top to bottom, each row showcases a different challenging scene. Our model, FE-SAM (last column), consistently produces segmentation masks that are more detailed and accurate than those from competing methods.

Method	Type	Backbone	Per-class F1-score					mIoU(%)	m-F1(%)	OA(%)
Method	Type	Backbone	Imp. surf.	Building	Low veg.	Tree	Car	mIoU(%)	m-F1(%)	OA(%)
FCN8s	CNN- based	VGG-16	90.62	92.55	85.02	85.95	92.48	81.03	89.32	88.00
U-Net		ResNet-18	92.31	94.56	85.82	87.37	95.01	83.96	91.01	89.60
DeepLabv3+		ResNet-18	92.78	95.07	86.10	87.41	94.86	84.02	91.24	89.70
MA-Net		ResNet-18	92.86	95.30	86.47	87.77	95.45	84.91	91.57	90.41
A2-FPN		ResNet-18	93.08	95.79	86.53	88.00	94.84	85.16	91.65	90.44

TABLE IV: Quantitative comparisons with the state-of-the-art methods on the ISPRS Potsdam dataset. The bold and underline denote the best and second results.

Fig. 7: Additional qualitative results on the ISPRS Potsdam dataset. Our model, FE-SAM (last column), consistently generates more accurate and complete segmentation masks compared to other state-of-the-art methods, especially in areas with complex object layouts.

Method	Type	Backbone	Per-class IoU							mIoU(%)
Method	Type	Backbone	Background	Building	Road	Water	Barren	Forest	Agriculture	mIoU(%)
FCN8s	CNN- based	VGG-16	54.46	63.95	56.88	62.66	11.83	32.30	47.03	48.02
U-Net		ResNet-18	54.16	58.56	57.19	57.93	27.31	41.54	56.53	50.46
DeepLabv3+		ResNet-18	54.39	60.15	58.09	63.30	26.52	41.94	55.38	51.45
MA-Net		ResNet-18	54.76	60.94	57.10	62.18	34.82	37.26	54.42	51.64
A2-FPN		ResNet-18	55.64	62.00	58.42	65.56	28.08	42.19	53.54	52.20

TABLE V: Quantitative comparisons with the state-of-the-art methods on the LoveDA dataset. The bold and underline denote the best and second results.

Fig. 8: Additional qualitative results on the LoveDA dataset. The selected scenes showcase diverse urban and rural landscapes. Our model, FE-SAM (last column), consistently provides more complete and semantically accurate segmentation masks than competing methods.

Method	Backbone	mIoU (%)	B-IoU	B-IoU
Method	Backbone	mIoU (%)	( $d=8$ )	( $d=16$ )
U-Net[42]	ResNet-18	80.96	73.12	75.48
DeepLabv3+[43]	ResNet-18	81.78	74.56	76.82
UNetFormer[46]	ResNet-18	82.86	76.28	78.53
SAM[8]	ViT-B	83.04	75.82	77.96
SAM-Adapter[50]	SAM-B	83.35	77.14	79.21

TABLE VI: Quantitative comparison of boundary segmentation quality on the ISPRS Vaihingen dataset. B-IoU (

d=8

) and B-IoU (

d=16

) measure the accuracy within 8 and 16 pixels from the boundary

Method	Total Params(M)	FLOPs(M)	FPS
SAM [8]	86.94	103.17	15.6
SAM-Adapter [50]	90.78	109.61	14.1
RSPrompter [18]	93.45	113.66	11.8
Semantic-SAM [52]	101.12	115.83	10.2
FE-SAM (Ours)	94.52	114.11	12.4

TABLE VII: Model complexity comparison of SAM-based methods.

Fig. 9: Comparison of mIoU, total parameters, and inference speed among SAM-based methods.

FMA	EGRefiner	Vaihingen	Potsdam	LoveDA
		83.04	85.94	52.54
✓		84.06	86.50	54.13
	✓	83.57	86.25	53.64
✓	✓	85.24	86.73	55.66

TABLE VIII: Ablation studies on different modules of FE-SAM.

Fig. 10: Visual comparisons of feature representations before and after the FMA module. From left to right: original remote sensing images, feature heatmaps before FMA, and feature heatmaps after FMA.

Method	Trainable Params (M)	Vaihingen	Potsdam	LoveDA
Full tuning	83.02	83.04	85.94	52.54
Adapter	7.83	83.12	86.09	53.14
LoRA	7.09	83.20	86.15	53.07
FMA (Ours)	7.56	84.06	86.50	54.13

TABLE IX: Comparison of different parameter-efficient fine-tuning strategies. Performance is measured in mIoU(%).

FMA Configuration	Vaihingen	Potsdam	LoveDA
Baseline (w/o FMA)	83.57	86.25	53.64
Fixed LPF/HPF (Generic)	84.12	86.41	54.02
Fixed LPF/HPF (Learned)	84.45	86.50	54.31
FMA (Ours)	85.24	86.73	55.66

TABLE X: Ablation on the necessity of FMA’s adaptive partitioning, evaluated across all three benchmark datasets. Performance is measured in mIoU(%).

Fig. 11: Visual validation of FMA’s adaptive frequency partitioning on the ISPRS Vaihingen dataset.

Fig. 12: Visual validation of FMA’s adaptive frequency partitioning on the ISPRS Potsdam dataset.

Configuration	Vaihingen	Potsdam	LoveDA
Single-Scale DConv 3x3	84.72	86.55	54.71
Single-Scale DConv 5x5	84.85	86.60	55.06
Single-Scale DConv 7x7	84.81	86.58	54.89
Multi-Scale DConv (Ours)	85.24	86.73	55.66

TABLE XI: Ablation study on the multi-scale architecture of EGRefiner on three benchmark datasets (mIoU%).

Fig. 13: Qualitative comparison visualizing the efficacy of the Edge-Guided Refiner (EGRefiner) for boundary refinement on challenging scenes from the ISPRS Vaihingen dataset.

EGRefiner Configuration	Vaihingen	Potsdam	LoveDA
w/o Element-wise Subtraction	84.65	86.48	55.03
w/o Channel-wise Concat	84.81	86.55	55.21
w/o $F_{s}$ Feature Fusion	79.32	81.05	49.58
EGRefiner (Full, Ours)	85.24	86.73	55.66

TABLE XII: Ablation study on the architectural components of the EGRefiner, evaluated across all three benchmark datasets.

Remote Sensing Image Segmentation

Remote sensing image segmentation labels land cover and objects in satellite, aerial, radar, and multispectral imagery.

See all 220 papers

CROSS: Cascaded Distillation and Dual-Constraint Grounding for Remote Sensing Referring Segmentation
Aug 4, 2026Tingzhang Luo, Ruizhong Liu, Yichao Liu +3Referring Image Segmentation
City University of Hong Kong · The Hong Kong University of Science and Technology (Guangzhou) · Nankai University +1
Remote SAMsing: From Segment Anything to Segment Everything
Apr 30, 2026Osmar Luiz Ferreira de Carvalho, Osmar Abílio de Carvalho Júnior, Anesmar Olino de Albuquerque +1Image Segmentation Remote Sensing Image Segmentation
Department of Electrical Engineering, University of Brasília, Brasília, Brazil · Department of Geography, University of Brasília, Brasília, Brazil
GeoSAM-Lite: A Lightweight Foundation Model for Onboard Remote Sensing Segmentation
Jul 4, 2026Yongcong Wang, Jie Zhang, Rui Jiang +3Remote Sensing Image Segmentation Geospatial Foundation Models
College of Telecommunications and Information Engineering, Nanjing University of Posts and Telecommunications, Nanjing 210003, China

Aug 4, 2026J/K move · Enter open · S save

CROSS: Cascaded Distillation and Dual-Constraint Grounding for Remote Sensing Referring Segmentation

Tingzhang Luo, Ruizhong Liu, Yichao Liu, Cheng Fan+2

City University of Hong Kong · The Hong Kong University of Science and Technology (Guangzhou) · Nankai University+1

Referring Image Segmentation

Open paper AskPDF arXiv

Referring Remote Sensing Image Segmentation (RRSIS) has achieved significant progress through the integration of VLMs and the Segment Anything Model (SAM). However, this progress largely relies on strong pre-trained capabilities, while leaving two fundamental limitations insufficiently addressed: (1) Architectural Weak-Coupling, where the unidirectional flow forces reliance on coarse VLM prompts and wastes SAM's pixel-level structural guidance, causing localization drift; and (2) Object-Centric Semantic Bias, where models overemphasize dominant object semantics while remaining insensitive to spatial reasoning crucial for RRSIS. Motivated by these observations, we propose CROSS, a tightly integrated paradigm for RRSIS. First, we introduce Linguistic-Guided Cascaded Distillation (LGCD) to bridge the architectural gap, which distills SAM's geometric affinities as soft regularizers into VLM intermediate layers, injecting dense structural priors to refine localization. Second, Perspective-Spatial Contrastive Learning (PSCL) imposes cross-anchored constraints by mining mask-filtered deceptive distractors and spatial-linguistic counterfactuals as hard negatives, explicitly shattering semantic shortcuts to enforce genuine logical consistency. Extensive experiments on RRSIS benchmarks demonstrate that CROSS achieves state-of-the-art performance and maintains precise localization even under severe spatial description perturbations, standing as a robust new paradigm for RRSIS.

Abstract

Remote Sensing Image Segmentation

Explore similar work

CROSS: Cascaded Distillation and Dual-Constraint Grounding for Remote Sensing Referring Segmentation

Remote SAMsing: From Segment Anything to Segment Everything

GeoSAM-Lite: A Lightweight Foundation Model for Onboard Remote Sensing Segmentation