BASeg: Boundary-Aware Remote Sensing Segmentation with Structural Penalties
Authors: Yuexi Song, Kailai Sun, Zhuoyu Wang, Mingyi He, Paul Pu Liang, Shenhao Wang, Jinhua Zhao
Organizations: National University of Singapore · Singapore-MIT Alliance for Research and Technology · Massachusetts Institute of Technology · University of Florida
Semantic segmentation is a core computer vision task in the remote sensing field, accelerating advancements in ur- ban development, agriculture, ecology, water resources, and environmental monitoring. However, recent methods usually struggle to capture fine-grained object features and bound- ary details. Besides, current widely used datasets often lack city morphology diversity and segmentation on generative im- ages remains largely unexplored. To address these issues, we propose a Mahalanobis-Angle Boundary Loss (MABL) that explicitly enhances boundary and shape consistency. MABL jointly models structural importance and boundary orientation through Mahalanobis distance-based weighting and angle- aware penalty. It can be readily integrated into diverse seg- mentation architectures and consistently improves their accu- racy. Built upon MABL, we introduce BASeg, a boundary- aware remote sensing segmentation framework with Struc- tural Penalties. BASeg integrates a Global Visual State Space module (GSM) with a Cross-Feature Fusion module (CFM) to capture both long-range contextual dependencies and fine- grained local details. Additionally, we establish a global 10- city benchmark dataset (GCD-25k) to facilitate accurate build- ing and road segmentation. Extensive experiments on four remote-sensing benchmarks demonstrate that BASeg consis- tently outperforms existing methods, achieving up to a 2.8% improvement in mIoU while producing more accurate object boundary segmentation across diverse scenes. Moreover, integrating MABL into multiple existing segmentation archi- tectures consistently improves performance across datasets, demonstrating its robustness and broad applicability.
Figures & tables
Figure 1: ((a) Qualitative comparison of semantic segmentation results between BASeg and a representative baseline. Red rectangles highlight regions where BASeg produces more accurate boundaries and better separates adjacent objects. (b) Radar chart comparing mIoU and mAcc across the four benchmark datasets.
Figure 2: (a) Ground truth segmentation, (b) Euclidean distance map and (c) Mahalanobis distance map. The Mahalanobis prior better adapts to each instance’s orientation and geometry.
Figure 3: Overall Architecture of BASeg.
Figure 4: Qualitative comparison and analysis on four datasets: (a) Original images, (b) Ground truth, (c) ABCNet, (d) UNetFormer, (e) CMTFNet, (f) RS3Mamba, and (g) the proposed BASeg.
Method
Backbone
GCD-25k
Vaihingen
HSRS
LoveDA
Bkg
Bld
Rd
mIoU
Imp
LV
Bld
Tree
Car
mIoU
mIoU
mAcc
mIoU
mAcc
ABCNet
ResNet
73.08
52.57
41.48
55.71
73.27
59.85
80.59
70.92
51.45
67.22
62.48
76.46
44.35
62.54
UNetFormer
ResNet
72.90
52.70
50.56
58.72
73.79
60.23
81.16
71.93
54.65
68.35
63.76
76.62
41.90
64.32
RS3Mamba
Mamba+ResNet
77.69
62.68
57.89
66.09
77.90
62.28
81.00
77.37
51.32
70.77
68.19
80.31
45.70
64.39
CMTFNet
ResNet
73.21
54.76
45.97
57.98
78.24
59.80
85.25
72.54
50.35
69.24
65.53
74.11
40.53
58.22
CMTFNet+MABL
ResNet
76.55
61.02
56.97
64.85
77.08
64.76
85.99
73.65
54.86
71.27
66.04
78.03
44.97
63.46
Table 1: Quantitative comparison on GCD-25k, Vaihingen, HSRS, and LoveDA. Best results are shown in bold. The △ rows report the performance gain over the corresponding baseline.
Dataset
Class
Only AngL
Only MaL
Combined
IoU
Acc
IoU
Acc
IoU
Acc
GCD-25k
Building
63.66
77.41
63.87
77.26
63.87
78.00
Road
60.40
74.70
60.20
76.34
61.48
77.84
Background
81.21
89.87
81.36
89.88
81.24
89.36
mIoU / mAcc
68.42
80.66
68.48
81.16
68.86
81.73
Vaihingen
Impervious
75.40
85.49
73.66
83.81
79.72
87.00
Table 2: Ablation study of the proposed loss functions on the GCD-25k and Vaihingen datasets.
GSM
No RCM
No Coord. Attn.
✓
✓
CFM
✓
✓
CCM
✓
Class
IoU
Acc
IoU
Acc
IoU
Acc
IoU
Acc
Impervious
72.65
80.86
74.65
84.19
74.93
83.81
79.12
87.00
Low Vegetation
57.66
73.94
62.14
77.06
62.37
76.25
66.46
77.28
Building
82.66
93.30
85.47
93.29
85.66
94.41
88.25
94.42
Tree
71.49
81.45
72.18
82.66
72.36
83.49
73.92
84.82
Table 3: Ablation study of GSM and CFM architectures on the Vaihingen dataset. (CCM refers to the architecture used in RS3Mamba ( Ma and Zhang 2024 ) ).
Loss
Time (ms/image)
mIoU
mAcc
Cross-entropy
9.36
69.24
80.47
SAM loss
11.29
70.15
83.72
ABL
18.91
70.72
83.25
MABL
18.27
71.27
86.34
Table 4: Comparison of boundary losses on Vaihingen with CMTFNet. Runtime is measured per image.
Model
GFLOPs
Parameters (M)
mIoU
CMTFNet
8.57
30.07
69.24
RS3Mamba
15.83
43.32
70.77
PyramidMamba
64.70
125.09
69.63
Samba
118.11
51.86
65.99
Ours
17.13
41.79
73.08
Table 5: Model complexity and segmentation performance on Vaihingen at 256×256 resolution.
City
Building (%)
Road (%)
Abuja
16.27
7.33
Barcelona
23.04
9.89
Boston
11.76
8.26
Kinshasa
22.83
8.72
Paris
22.69
9.19
Perth
20.79
8.87
Table 6: City-wise class coverage (% of pixels).
Figure 5: Representative samples from the GCD-25k dataset across 10 cities, showing the original satellite image and the corresponding ground truth semantic mask.
Method
Bkg
Road
Bld
Veg
Water
Car
Field
Court
Square
mIoU
ABCNet
76.23
56.13
58.16
69.38
74.50
69.38
50.08
77.70
30.73
62.48
UNetformer
76.16
60.26
59.16
65.70
73.95
73.22
54.20
80.09
31.10
63.76
CMTFNet
71.38
61.32
58.93
63.84
71.34
82.83
54.07
88.70
37.38
65.53
RS3Mamba
78.24
67.25
64.01
69.53
77.53
78.73
56.14
84.54
37.73
68.19
Samba
75.21
60.21
61.61
71.03
72.47
64.96
46.72
85.38
32.55
63.35
PyramidMamba
77.40
67.24
62.73
69.87
76.38
47.26
57.36
87.30
43.11
65.41
Table 7: Per-class IoU comparison on the HSRS dataset. Bkg: Background; Bld: Building; Veg: Vegetation.
Method
Bkg
Bld
Road
Water
Forest
Agriculture
mIoU
ABCNet
37.43
56.04
53.32
54.67
35.65
28.98
44.35
UNetformer
37.61
52.78
51.89
63.47
34.23
11.44
41.90
CMTFNet
30.71
46.29
53.24
46.61
38.65
28.17
40.61
Samba
34.20
38.74
43.11
62.70
35.87
17.11
38.62
PyramidMamba
35.02
58.06
59.82
63.74
31.31
18.48
44.44
Ours
36.93
59.19
56.87
62.72
39.82
29.62
47.53
Table 8: Per-class IoU comparison of segmentation models on the LoveDA dataset. Bkg: Background; Bld: Building.
Dataset
Class
No Mahalanobis Loss
No Angle Loss
Ours
IoU
Acc
IoU
Acc
IoU
Acc
LoveDA
Background
36.03
58.95
35.51
61.15
36.93
61.45
Building
57.63
89.68
57.55
89.48
59.19
85.97
Road
54.69
71.28
54.37
71.69
56.87
69.51
Water
58.08
67.89
56.60
62.86
62.72
73.84
Forest
42.52
85.41
34.99
84.01
39.82
87.05
Table 9: Ablation study of loss functions on the LoveDA and HSRS datasets.
Class
No RCM
No Coord. Attn.
RCM-2
RCM-3
GSM
IoU
Acc
IoU
Acc
IoU
Acc
IoU
Acc
IoU
Acc
Building
63.38
76.43
61.32
75.84
63.97
77.89
64.22
77.01
63.87
78.00
Road
60.56
77.08
57.17
73.39
60.66
76.06
60.49
73.88
61.48
77.84
Background
81.10
89.74
76.62
86.73
78.46
87.90
79.07
89.09
81.24
89.36
mIoU / mAcc
68.34
81.08
65.04
78.65
67.70
80.62
67.93
79.99
68.86
81.73
Table 10: Ablation study of GSM architecture on the GCD-25k dataset. RCM: Rectangular Convolution Module. (CFM is kept active in all configurations).
Class
No SA
No CSA
CFM
IoU
Acc
IoU
Acc
IoU
Acc
Building
54.81
68.66
59.12
72.75
63.87
78.00
Road
45.50
50.17
53.84
67.96
61.48
77.84
Background
76.13
87.76
79.05
89.26
81.24
89.36
mIoU / mAcc
58.82
72.20
64.00
76.66
68.86
81.73
Table 11: Ablation study of CFM architecture on the GCD-25k dataset. SA: Spatial Attention; CSA: Cross Attention. (GSM is kept active in all configurations).
Open-vocabulary remote sensing image segmentation (OVRSIS) aims to segment text-specified categories beyond a fixed label space. Its key challenge is to maintain reliable pixel--text correspondence under substantial appearance variation in remote-sensing imagery. Existing methods often rely on specialized vision--language representations or complex adaptation mechanisms, yet still perform deterministic matching between fixed text prototypes and individual visual features. We instead formulate OVRSIS as a distributional pixel--text alignment problem and propose Pi-Seg, a lightweight perturbation-based framework. Pi-Seg introduces learnable variations into textual prototypes and dense visual features before alignment. Segmentation supervision encourages perturbations that enlarge the target-to-distractor margin while suppressing harmful feature shifts. Pi-Seg consistently improves performance under the existing single-source OVRSISBench protocols. To evaluate OVRSIS beyond source-specific training settings, we further introduce a unified multi-source protocol and construct GlobalRSOV95K, containing approximately 95K densely annotated images and 35 semantic categories. Models are trained on this shared dataset and evaluated on an image-disjoint suite of 10 downstream datasets. The complete benchmark covers more than 170K images and 122 categories. Experiments under both single-source and multi-source protocols show that Pi-Seg improves cross-dataset transfer and generalizes effectively to practical geospatial segmentation tasks, while GlobalRSOV95K provides a common foundation for transferable OVRSIS research. \footnote{https://github.com/LiBingyu01/Pi-Seg}
Bingyu Li, Tao Huo, Haocheng Dong +4
Department of Electronic Engineering and Information Science, University of Science and Technology of China, China · Institute of Artificial Intelligence (TeleAI), China Telecom, P. R. China · School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University, China
Recent advances in MLLMs are reframing segmentation from fixed-category prediction to instruction-grounded localization. While reasoning based segmentation has progressed rapidly in natural scenes, remote sensing lacks a generalizable solution due to the prohibitive cost of reasoning-oriented data and domain-specific challenges like overhead viewpoints. We present GeoSeg, a zero-shot, training-free framework that bypasses the supervision bottleneck for reasoning-driven remote sensing segmentation. GeoSeg couples MLLM reasoning with precise localization via: (i) bias-aware coordinate refinement to correct systematic grounding shifts and (ii) a dual-route prompting mechanism to fuse semantic intent with fine-grained spatial cues. We also introduce GeoSeg-Bench, a diagnostic benchmark of 810 image--query pairs with hierarchical difficulty levels. Experiments show that GeoSeg consistently outperforms all baselines, with extensive ablations confirming the effectiveness and necessity of each component.
Lifan Jiang, Yuhang Pei, oxi Wu +5
State Key lab of CAD&CG, Zhejiang University · UniTTEC Co. Ltd. · Wuchan Zhongda Chengtou (Ningbo) Holdings. Ltd.
Open-vocabulary semantic segmentation in the remote sensing (RS) field requires both language-aligned recognition and fine-grained spatial delineation. Although CLIP offers robust semantic generalization, its global-aligned visual representations inherently struggle to capture structural details. Recent methods attempt to compensate for this by introducing RS-pretrained DINO features. However, these methods treat CLIP representations as a monolithic semantic space and cannot localize where structural enhancement is required, failing to effectively delineate boundaries while risking the disruption of CLIP's semantic integrity. To address this limitation, we propose DR-Seg, a novel decouple-and-rectify framework in this paper. Our method is motivated by the key observation that CLIP feature channels exhibit distinct functional heterogeneity rather than forming a uniform semantic space. Building on this insight, DR-Seg decouples CLIP features into semantics-dominated and structure-dominated subspaces, enabling targeted structural enhancement by DINO without distorting language-aligned semantics. Subsequently, a prior-driven graph rectification module injects high-fidelity structural priors under DINO guidance to form a refined branch, while an uncertainty-guided adaptive fusion module dynamically integrates this refined branch with the original CLIP branch for final prediction. Comprehensive experiments across eight benchmarks demonstrate that DR-Seg establishes a new state-of-the-art.
Jie Feng, Fengze Li, Junpeng Zhang +4
Xidian University China · Jimei University China · South China University of Technology China